[OpenVMS documentation]
[Site home] [Send comments] [Help with this site] [How to order documentation] [OpenVMS site] [Compaq site]
Updated: 11 December 1998

OpenVMS Guide to System Security


Previous Contents Index


Chapter 10
System Security Breaches

Along with developing a security policy and selecting appropriate security measures to implement that policy, a site needs to establish and test procedures for handling system, site, or network compromises. The procedure should address two areas:

This chapter describes how to recognize when an attack on the system is in progress or has taken place and what countermeasures can be taken.

Note

1 Compaq support groups include the Software Security Response Team (SSRT) in the United States and the European Security Program Office (ESPO).

10.1 Forms of System Attacks

As security administrator, you must monitor the system on a regular basis for possible security breaches. Following are the most common forms of system attacks:

10.2 Indications of Trouble

When your system is vulnerable and possibly under attack, your first indications may come from the following sources:

10.2.1 Reports from Users

User observations frequently point to system security problems. A user may contact you with the following situations:

Follow up promptly when one of these items is reported to you. You must confirm or deny that the condition exists. If you find the complaint is valid, seek a cause and solution.

10.2.2 Monitoring the System

Section 6.7 lists those tasks that can help you detect potential security breaches on your system. The following list details possible warning signs you may uncover while performing the recommended tasks:

All these conditions warrant further investigation. Some indicate that you already have a problem, and some may have simple explanations, while others may indicate serious potential problems.

10.3 Routine System Surveillance

The operating system provides a number of mechanisms that allow systematic surveillance of the activity in your system. There are many mechanisms available for monitoring the system either manually or by user-written command procedures, for example:

Proper use of such mechanisms should help you verify settings, alert you to problems, and allow you to intervene. This section describes the most important system surveillance mechanisms--ACCOUNTING and ANALYZE/AUDIT.

10.3.1 System Accounting

You can learn what the normal pattern of resource use is by studying reports of the Accounting utility (ACCOUNTING). To obtain a report, you run the utility image SYS$SYSTEM:ACC.EXE. The resulting data file is SYS$MANAGER:ACCOUNTNG.DAT. Review ACCOUNTING reports because they can provide early indications of problems. Check for the following:

10.3.2 Security Auditing

As the security administrator, you can have the operating system report on security-related activity by enabling categories of events for auditing using the DCL command SET AUDIT. Using the Audit Analysis utility (ANALYZE/AUDIT), you can periodically review event messages collected in the security audit log file. (See Chapter 9 for a full description of the process.)

The operating system can send event messages to an audit log file or to an operator terminal. You define whether events are reported as audits or alarms in the following way:

Because security auditing affects system performance, enable auditing only for the most important events. The following security-auditing actions are presented in order of decreasing priority and increasing system cost:

  1. Enable security auditing for login failures and break-ins. This is the best way to detect probing by outsiders (and insiders looking for accounts). All sites needing security should enable alarms for these events.
  2. Enable security auditing for logins. Auditing successful logins from the more suspicious sources like remote and dialup users provides the best way to track which accounts are being used. An audit record is written before users logging in to a privileged account can disguise their identity.
  3. Enable security auditing for unsuccessful file access (ACCESS=FAILURE). This technique audits all file-protection violations and is an excellent method of catching probers.
  4. Apply ACL-based file access auditing to detect write access to critical system files. The most important files to audit are shown in Table 10-1. (Table 9-2 presents an example of how to establish security entries in ACLs.) You may want to audit only successful access to these files to detect penetration, or you may want to audit access failures to detect probing as well.
    Note that some of the files in Table 10-1 are written during normal system operation. For example, SYSUAF.DAT is written during each login, and SYSMGR.DIR is written when the system boots.

    Table 10-1 System Files Benefiting from ACL-Based Auditing
    Device and Directory File Name
    SYS$SYSTEM AUTHORIZE.EXE
      F11BXQP.EXE
      LOGINOUT.EXE
      DCL.EXE
      JOBCTL.EXE
      SYSUAF.DAT
      NETPROXY.DAT
      RIGHTSLIST.DAT
      STARTUP.COM
      VMS$OBJECTS.DAT
    SYS$LIBRARY SECURESHR.EXE
      SECURESHRP.EXE
    SYS$MANAGER VMS$AUDIT_SERVER.DAT
      SY*.COM
      VMSIMAGES.DAT
    SYS$SYSROOT [000000]SYSEXE.DIR
      [000000]SYSLIB.DIR
      [000000]SYS$LDR.DIR
      [000000]SYSMGR.DIR

  5. Enable security auditing for modifications to system parameters or the known file list (/ENABLE=(SYSGEN,INSTALL)).
  6. Audit use of privilege to access files (either write access or all forms of access). Implement the security audit with the keywords ACCESS=(SYSPRV,BYPASS,READALL,GRPPRV). Note that this class of auditing can produce a large volume of output because privileges are often used in normal system operation for such tasks as mail delivery and operator backups.

Section 9.3 provides further discussion of recommended sets of security events to audit.

10.4 Handling a Security Breach

There are four phases that security administrators experience while handling a security breach, whether the breach actually occurred or was attempted:

  1. Detection of a problem
  2. Identification of the perpetrator
  3. Prevention of further security violations
  4. Repair of damage

The following sections describe these phases for both attempted and successful break-ins.

In all phases, train personnel to retain information and data as evidence, should there be a need to apprehend and prosecute the perpetrator.

10.4.1 Unsuccessful Intrusion Attempts

Unsuccessful intrusion attempts include situations where someone has attempted to guess passwords or browse through files.

10.4.1.1 Detecting Intrusion Attempts

You usually detect intrusion attempts through the following sources:

10.4.1.2 Identifying the Perpetrator

Enabling file auditing simplifies identification of file browsers. If, however, browsing is being initiated from another node in the network, you must inspect the network server log file (NETSERVER.LOG) that corresponds to the times of the protection violations. Coordinate your investigation with the security administrator at the remote node.

Identifying a perpetrator who is guessing passwords is considerably more difficult, especially when the source is anonymous, as from a dialup line. Usually, you must trade identification for prevention. Often the only way to positively identify an outsider attempting to enter the system requires that you permit further attempts while establishing the perpetrator's identity.

10.4.1.3 Preventing Intrusion Attempts

The prevention phase for this kind of attack involves preventing the would-be intruder from actually gaining access to the system and making future attempts more difficult.

Password Guessing

To reduce the opportunities for successful password guessing:

File Browsing

To reduce the opportunities for successful file browsing:

10.4.2 Successful Intrusions

A successful security breach can include a successful password guessing scheme, theft or modification of information or system resources, and placement of damaging software on the system. An intrusion may require a considerable amount of time to repair, depending upon the skill and intent of the perpetrator.

10.4.2.1 Identifying the Successful Perpetrator

Identification is often the most difficult part of handling an intrusion. First, you must establish whether the perpetrator is an authorized user or not. This determines the nature of the preventive measures that you will take. However, the distinction between insiders and outsiders may be difficult to achieve.

Tradeoff Between Identification and Prevention

You may have to make a tradeoff between a positive identification of the intruder and preventing future attacks. Often, the data available initially does not allow complete identification. If it is important to identify the perpetrator, you will often find it necessary to permit continued intrusions while you analyze the intrusion activity. Increase your auditing. Consider planting traps in system procedures that are under your control (such as SYLOGIN.COM) to obtain additional information. Increase your system backup efforts to permit easier recovery if files become damaged.

Identification of Outsiders

Identifying external intruders is particularly difficult, especially if they use any switched forms of communication (such as dialup lines or public data networks). DECnet for OpenVMS software provides many features to help you trace the activity through the network back to the source node. If a local terminal is involved, physical surveillance may be appropriate.

When a switched connection is involved, one of the major computer security problems is the telephone system itself. Tracing a telephone or public data network connection is time-consuming. Chasing an intruder through the telephone system is likely to take months and will require the assistance of law enforcement authorities. The existence of multiple long-distance telephone services compounds the problem by increasing the number of organizations with whom you must deal.

As a result, identifying an outside intruder is usually worthwhile only when you have sustained substantial financial damage. In many cases, it may be more useful if you concentrate on preventing recurrences of the problem.

10.4.2.2 Securing the System

The actions you must take to secure your system after an intrusion depend on the nature and source of that intrusion. This section describes these actions in order of priority.

  1. Restore SYSUAF.DAT, NETPROXY.DAT, NET$PROXY.DAT and RIGHTSLIST.DAT (if damaged) from backups. Alternatively, generate listings of the files and inspect them closely, looking for improper entries, additional privileges, and changed UICs. If you are unsure of when SYSUAF.DAT might first have been modified, inspect it carefully regardless of whether you are using a backup copy or proceeding with the existing one. Be sure all authorization files are secure.
  2. The perpetrator may have discovered passwords by browsing through files or from other nodes in the network and may be using seldom accessed accounts for personal use. Change passwords for accounts, and have your users appear in person to learn their new passwords. At a minimum, change passwords on all privileged accounts. Do not use the same new password for all accounts.
  3. A sophisticated penetrator may have planted ways to provide future access to the system even though you have taken the obvious steps of securing your system. Therefore, you may have to restore selected components of the OpenVMS software from backups or from your OpenVMS distribution kit. If the intruder was an outsider, the two critical components are LOGINOUT.EXE and NETACP.EXE, which validate all entries to the system.
    However, if the intruder was an authorized user, restore all system files from backup copies. Authorized users can make use of a wide variety of illicit software patches (called trap doors) that they insert in the executive (SYS.EXE), the file system (F11BXQP.EXE), DCL, and other system files. The penetrator may have planted damaging software in any piece of software or command procedure likely to be used by a privileged user. Thus, complete assurance of a secure system requires a wholesale restoration of files from backups. Also reinstall any image (even from layered products) installed with privileges because it can also be used for a trap door. An alternate strategy is to restore trustworthy copies of the obvious targets of attack and to rely on increased auditing for a period of time to catch suspicious events.
  4. Consider implementing additional security features, such as system passwords, password generation, increased auditing, and more stringent file protection to prevent a recurrence.

10.4.2.3 Repair After a Successful Intrusion

After an intrusion, restore corrupted files. Decide whether it is appropriate to do a wholesale restoration of your system's data or to repair problems as they are discovered. Look for modifications to file protection that would have created paths for viruses and for Trojan horses that were introduced into the system and may still reside there.


Previous Next Contents Index

[Site home] [Send comments] [Help with this site] [How to order documentation] [OpenVMS site] [Compaq site]
[OpenVMS documentation]

Copyright © Compaq Computer Corporation 1998. All rights reserved.

Legal
6346PRO_025.HTML