Updated: 11 December 1998

OpenVMS Guide to System Security

Along with developing a security policy and selecting appropriate security measures to implement that policy, a site needs to establish and test procedures for handling system, site, or network compromises. The procedure should address two areas:

• Appropriate responses once a breach is suspected or confirmed. Site guidelines should help determine whether to increase site security (eliminating all possibility of further compromise), put proactive measures in place to apprehend the offender, or collect evidence to initiate a criminal or civil suit. Each decision has its own set of rules and guidelines.
• Appropriate contacts and resources outside of the site that may be needed should such an event occur. For example, a company might want to become familiar with local, state, and federal authorities (as applicable), local phone carriers (security division), and the Compaq support groups.¹

This chapter describes how to recognize when an attack on the system is in progress or has taken place and what countermeasures can be taken.

10.1 Forms of System Attacks

As security administrator, you must monitor the system on a regular basis for possible security breaches. Following are the most common forms of system attacks:

• Hunting for access lines
• Hunting for passwords
• Attempting a break-in
• Changing or creating user authorization file (UAF) records
• Granting/stealing extra privileges
• Introducing apparently innocent software (Trojan horse software) that is intended to steal user passwords or do other damage to the system
• Introducing viruses in command procedures and programs to gain access to privileged accounts
• Scavenging disks
• Using a node as a gateway to other nodes

When your system is vulnerable and possibly under attack, your first indications may come from the following sources:

• Reports from users
• Monitoring the system, for example:
  • Unexplained changes or behavior in applications or normal processes
  • Unexplained messages from OPCOM or the audit server
  • Unexplained changes to user accounts in the system authorization database (privilege changes, protections, priorities, quotas)

User observations frequently point to system security problems. A user may contact you with the following situations:

• Files are missing.
• There are unexplained forms of last login messages, such as successful logins the user did not perform or unexplained login failures.
• A user cannot log in, suggesting the user password might have been changed since the last successful login or some other form of tampering has occurred.
• Break-in evasion appears to be in effect, and the user cannot log in.
• Reports from the SHOW USERS command indicate that the user is logged in on another terminal when the user did not do so.
• A disconnected job message appears during a login for a process the user never initiated.
• Files exist in the user's directories that the user did not create.
• Unexplained changes have been found in the protection or ownership of user files.
• Listings appear that are generated under the user name without the user requesting the listing.
• A sudden reduction occurs in the availability of resources, such as dialup lines.

Follow up promptly when one of these items is reported to you. You must confirm or deny that the condition exists. If you find the complaint is valid, seek a cause and solution.

10.2.2 Monitoring the System

Section 6.7 lists those tasks that can help you detect potential security breaches on your system. The following list details possible warning signs you may uncover while performing the recommended tasks:

• A user appears on the SHOW USERS report that you know could not be currently logged in.
• You observe an unexplained change in the system load or performance.
• You discover media or program listings are missing or notice other indications that physical security has degraded.
• Your locked file cabinet has been tampered with, and the list of authorized users has disappeared.
• You find unfamiliar software in the system executable image library [SYSEXE] or in [SYSLIB].
• You observe unfamiliar images running when you examine the MONITOR SYSTEM report.
• You observe unauthorized user names when you enter the DCL command SHOW USER. When you examine the listing that the Authorize utility (AUTHORIZE) produces with the SHOW command, you find that those users have been given system access.
• You discover proxy users that you never authorized.
• The accounting report reveals unusual amounts of processing time expended recently, suggesting outside access.
• You observe unexplained batch jobs on the batch queues.
• You observe unexpected device allocations when you enter the SHOW DEVICE command.
• You observe a high level of processing activity at unusual hours.
• The protection codes or the access control lists (ACLs) change on critical files. Identifiers are added, or holders of identifiers are added to the rights list.
• There is high personnel turnover or low morale.

All these conditions warrant further investigation. Some indicate that you already have a problem, and some may have simple explanations, while others may indicate serious potential problems.

10.3 Routine System Surveillance

The operating system provides a number of mechanisms that allow systematic surveillance of the activity in your system. There are many mechanisms available for monitoring the system either manually or by user-written command procedures, for example:

• Accounting utility (ACCOUNTING)
• Authorize utility (AUTHORIZE)
• Install utility (INSTALL)
• System Management utility (SYSMAN)

Proper use of such mechanisms should help you verify settings, alert you to problems, and allow you to intervene. This section describes the most important system surveillance mechanisms--ACCOUNTING and ANALYZE/AUDIT.

10.3.1 System Accounting

You can learn what the normal pattern of resource use is by studying reports of the Accounting utility (ACCOUNTING). To obtain a report, you run the utility image SYS$SYSTEM:ACC.EXE. The resulting data file is SYS$MANAGER:ACCOUNTNG.DAT. Review ACCOUNTING reports because they can provide early indications of problems. Check for the following:

• Unfamiliar user names
• Unfamiliar patterns of use, such as unusual activity for a particular time of day or day of week
• Use of an unusual amount of resources
• Unfamiliar sources of login, such as network nodes or remote terminals

As the security administrator, you can have the operating system report on security-related activity by enabling categories of events for auditing using the DCL command SET AUDIT. Using the Audit Analysis utility (ANALYZE/AUDIT), you can periodically review event messages collected in the security audit log file. (See Chapter 9 for a full description of the process.)

The operating system can send event messages to an audit log file or to an operator terminal. You define whether events are reported as audits or alarms in the following way:

• Ordinarily, enable audits rather than alarms for security-related events because the audit records are written to the system security audit log where you can study them in volume and archive log files for future reference. While an isolated auditing message may offer little insight, numerous audit records produce a pattern of security violations. For example, with auditing of object access, you can see a pattern of time, types of objects being accessed, and other system information that, in total, paint a picture of how the system is being used at different times of day.
  To enable audits for unsuccessful access to files, devices, and volumes, enter the following command:

  $ SET AUDIT/AUDIT/ENABLE=ACCESS=FAILURE/CLASS=(FILE,DEVICE,VOLUME)

  
  This command records unsuccessful access events in the security audit log file but sends no alarms to the operator terminal.

• Enable security alarms for real-time events or events that should be reviewed immediately, for example, intrusion attempts or changes to the system user authorization file (SYSUAF.DAT). For example, to enable alarms for modification to the known file list and changes to system time, enter the following command:

  $ SET AUDIT/ALARM/ENABLE=(INSTALL,TIME)

  
  This command sends event messages to the operator terminal. To keep a hardcopy record of these alarms, use a hardcopy operator terminal, or enable the events as both alarms and audits.

Because security auditing affects system performance, enable auditing only for the most important events. The following security-auditing actions are presented in order of decreasing priority and increasing system cost:

1. Enable security auditing for login failures and break-ins. This is the best way to detect probing by outsiders (and insiders looking for accounts). All sites needing security should enable alarms for these events.
2. Enable security auditing for logins. Auditing successful logins from the more suspicious sources like remote and dialup users provides the best way to track which accounts are being used. An audit record is written before users logging in to a privileged account can disguise their identity.
3. Enable security auditing for unsuccessful file access (ACCESS=FAILURE). This technique audits all file-protection violations and is an excellent method of catching probers.
4. Apply ACL-based file access auditing to detect write access to critical system files. The most important files to audit are shown in Table 10-1. (Table 9-2 presents an example of how to establish security entries in ACLs.) You may want to audit only successful access to these files to detect penetration, or you may want to audit access failures to detect probing as well.
   Note that some of the files in Table 10-1 are written during normal system operation. For example, SYSUAF.DAT is written during each login, and SYSMGR.DIR is written when the system boots.

   Table 10-1 System Files Benefiting from ACL-Based Auditing

   Device and Directory	File Name
   SYS$SYSTEM	AUTHORIZE.EXE
   	F11BXQP.EXE
   	LOGINOUT.EXE
   	DCL.EXE
   	JOBCTL.EXE
   	SYSUAF.DAT
   	NETPROXY.DAT
   	RIGHTSLIST.DAT
   	STARTUP.COM
   	VMS$OBJECTS.DAT
   SYS$LIBRARY	SECURESHR.EXE
   	SECURESHRP.EXE
   SYS$MANAGER	VMS$AUDIT_SERVER.DAT
   	SY*.COM
   	VMSIMAGES.DAT
   SYS$SYSROOT	[000000]SYSEXE.DIR
   	[000000]SYSLIB.DIR
   	[000000]SYS$LDR.DIR
   	[000000]SYSMGR.DIR

5. Enable security auditing for modifications to system parameters or the known file list (/ENABLE=(SYSGEN,INSTALL)).
6. Audit use of privilege to access files (either write access or all forms of access). Implement the security audit with the keywords ACCESS=(SYSPRV,BYPASS,READALL,GRPPRV). Note that this class of auditing can produce a large volume of output because privileges are often used in normal system operation for such tasks as mail delivery and operator backups.

Section 9.3 provides further discussion of recommended sets of security events to audit.

10.4 Handling a Security Breach

There are four phases that security administrators experience while handling a security breach, whether the breach actually occurred or was attempted:

1. Detection of a problem
2. Identification of the perpetrator
3. Prevention of further security violations
4. Repair of damage

The following sections describe these phases for both attempted and successful break-ins.

In all phases, train personnel to retain information and data as evidence, should there be a need to apprehend and prosecute the perpetrator.

10.4.1 Unsuccessful Intrusion Attempts

Unsuccessful intrusion attempts include situations where someone has attempted to guess passwords or browse through files.

10.4.1.1 Detecting Intrusion Attempts

You usually detect intrusion attempts through the following sources:

• Reports from users about unexplained login failures
• Unusual system activity or unavailability of dialup lines
• Security alarms for login failures, break-in attempts, and file-protection violations
• Examination of the intrusion database

Enabling file auditing simplifies identification of file browsers. If, however, browsing is being initiated from another node in the network, you must inspect the network server log file (NETSERVER.LOG) that corresponds to the times of the protection violations. Coordinate your investigation with the security administrator at the remote node.

Identifying a perpetrator who is guessing passwords is considerably more difficult, especially when the source is anonymous, as from a dialup line. Usually, you must trade identification for prevention. Often the only way to positively identify an outsider attempting to enter the system requires that you permit further attempts while establishing the perpetrator's identity.

10.4.1.3 Preventing Intrusion Attempts

The prevention phase for this kind of attack involves preventing the would-be intruder from actually gaining access to the system and making future attempts more difficult.

Password Guessing

To reduce the opportunities for successful password guessing:

• Make certain your users choose appropriate passwords. Consider use of the password generator (see Section 7.3.2.4).
• Enable system passwords at the points of entry. While a minor inconvenience to your users, system passwords are the best protection against further probing. If you already had a system password enabled, change it (see Section 7.3.1.2).
• Enable auditing of successful logins to catch the event if the intruder succeeds in getting in (see Section 10.3.2).

File Browsing

To reduce the opportunities for successful file browsing:

• If you can identify the perpetrator, take action as established at your site.
• Warn your users about the importance of adequate protection of their files, and consider inspecting the protection of user files.
• If file browsing from other nodes in the network becomes a persistent problem, eliminate the default FAL account and authorize individual users through proxy login accounts (see Section 12.3.2).

A successful security breach can include a successful password guessing scheme, theft or modification of information or system resources, and placement of damaging software on the system. An intrusion may require a considerable amount of time to repair, depending upon the skill and intent of the perpetrator.

10.4.2.1 Identifying the Successful Perpetrator

Identification is often the most difficult part of handling an intrusion. First, you must establish whether the perpetrator is an authorized user or not. This determines the nature of the preventive measures that you will take. However, the distinction between insiders and outsiders may be difficult to achieve.

Tradeoff Between Identification and Prevention

You may have to make a tradeoff between a positive identification of the intruder and preventing future attacks. Often, the data available initially does not allow complete identification. If it is important to identify the perpetrator, you will often find it necessary to permit continued intrusions while you analyze the intrusion activity. Increase your auditing. Consider planting traps in system procedures that are under your control (such as SYLOGIN.COM) to obtain additional information. Increase your system backup efforts to permit easier recovery if files become damaged.

Identification of Outsiders

Identifying external intruders is particularly difficult, especially if they use any switched forms of communication (such as dialup lines or public data networks). DECnet for OpenVMS software provides many features to help you trace the activity through the network back to the source node. If a local terminal is involved, physical surveillance may be appropriate.

When a switched connection is involved, one of the major computer security problems is the telephone system itself. Tracing a telephone or public data network connection is time-consuming. Chasing an intruder through the telephone system is likely to take months and will require the assistance of law enforcement authorities. The existence of multiple long-distance telephone services compounds the problem by increasing the number of organizations with whom you must deal.

As a result, identifying an outside intruder is usually worthwhile only when you have sustained substantial financial damage. In many cases, it may be more useful if you concentrate on preventing recurrences of the problem.

10.4.2.2 Securing the System

The actions you must take to secure your system after an intrusion depend on the nature and source of that intrusion. This section describes these actions in order of priority.

1. Restore SYSUAF.DAT, NETPROXY.DAT, NET$PROXY.DAT and RIGHTSLIST.DAT (if damaged) from backups. Alternatively, generate listings of the files and inspect them closely, looking for improper entries, additional privileges, and changed UICs. If you are unsure of when SYSUAF.DAT might first have been modified, inspect it carefully regardless of whether you are using a backup copy or proceeding with the existing one. Be sure all authorization files are secure.
2. The perpetrator may have discovered passwords by browsing through files or from other nodes in the network and may be using seldom accessed accounts for personal use. Change passwords for accounts, and have your users appear in person to learn their new passwords. At a minimum, change passwords on all privileged accounts. Do not use the same new password for all accounts.
3. A sophisticated penetrator may have planted ways to provide future access to the system even though you have taken the obvious steps of securing your system. Therefore, you may have to restore selected components of the OpenVMS software from backups or from your OpenVMS distribution kit. If the intruder was an outsider, the two critical components are LOGINOUT.EXE and NETACP.EXE, which validate all entries to the system.
   However, if the intruder was an authorized user, restore all system files from backup copies. Authorized users can make use of a wide variety of illicit software patches (called trap doors) that they insert in the executive (SYS.EXE), the file system (F11BXQP.EXE), DCL, and other system files. The penetrator may have planted damaging software in any piece of software or command procedure likely to be used by a privileged user. Thus, complete assurance of a secure system requires a wholesale restoration of files from backups. Also reinstall any image (even from layered products) installed with privileges because it can also be used for a trap door. An alternate strategy is to restore trustworthy copies of the obvious targets of attack and to rely on increased auditing for a period of time to catch suspicious events.
4. Consider implementing additional security features, such as system passwords, password generation, increased auditing, and more stringent file protection to prevent a recurrence.

After an intrusion, restore corrupted files. Decide whether it is appropriate to do a wholesale restoration of your system's data or to repair problems as they are discovered. Look for modifications to file protection that would have created paths for viruses and for Trojan horses that were introduced into the system and may still reside there.

Copyright © Compaq Computer Corporation 1998. All rights reserved. Legal

6346PRO_025.HTML