Updated: 11 December 1998

OpenVMS Guide to System Security

This chapter presents the concepts that guided the design and implementation of the security features and mechanisms incorporated into the operating system. The intent is to provide a framework for thinking about your total system security picture. Subsequent chapters present details about the security features and their use.

2.1 Structure of a Secure Operating System

In the late 1960s, a great deal of research and development was dedicated to the problem of achieving security in multiuser computer systems. Much of the development work involved attempts to find all the things that could go wrong with a system's security and then to correct those flaws one by one. It became apparent to the researchers that this process was ineffective; effective system security could result only from a basic model of the structure of a secure computer system. The reference monitor concept was proposed as such a model and gained wide acceptance.

2.1.1 Reference Monitor Concept

According to the reference monitor concept, a computer system can be depicted in terms of subjects, objects, an authorization database, an audit trail, and a reference monitor, as shown in Figure 2-1. The reference monitor is the control center that authenticates subjects and implements and enforces the security policy for every access to an object by a subject.

Figure 2-1 Reference Monitor

The following table describes the elements shown in Figure 2-1:

Item	Element	Description
1	Subjects	Active entities, such as user processes, that gain access to information on behalf of people.
2	Objects	Passive repositories of information to be protected, such as files.
3	Authorization database	Repository for the security attributes of subjects and objects. From these attributes, the reference monitor determines what kind of access (if any) is authorized.
4	Audit trail	Record of all security-relevant events, such as access attempts, successful or not.

2.1.2 How the Reference Monitor Enforces Security Rules

The reference monitor enforces the security policy by authorizing the creation of subjects, by granting subjects access to objects based on the information in a dynamic authorization database, and by recording events, as necessary, in the audit trail. In an ideal system, the reference monitor must meet the following three requirements:

• Mediate every attempt by a subject to gain access to an object
• Provide a tamperproof database and audit trail that are thoroughly protected from unauthorized observation and modification
• Remain a small, simple, and well-structured piece of software so that it is effective in enforcing security requirements

These are the requirements proposed for systems that are secure even against penetration. In such systems, the reference monitor is implemented by a security-related subset, or security kernel, of the operating system.

2.2 Implementation of the Reference Monitor

While the OpenVMS operating system does not implement the reference monitor as a security-related subset, or security kernel, its interface to users and system managers does mirror the basic structure dictated by the reference monitor concept. Experience shows that incorporating such a structure is the best way to build a system resistant to probing and to most attempts at penetration.

The following sections describe the OpenVMS operating system's implementation of the reference monitor model.

2.2.1 Subjects

Subjects are the users or user agents (the user processes) that access information and, in some cases, may be prevented from accessing information. Subjects can be interactive processes, network processes, or batch jobs, and:

• Must pass security controls
  • During process creation
  • During information access
• Require identification
  • User names
  • Passwords
  • User identification codes
  • Rights identifiers

When a user logs in to use the operating system interactively or when a batch or network job starts, the operating system creates a process that includes the identity of the user. That process gains access to information as the agent for the user, as described in Chapter 4.

Processes are vulnerable to security breaches while they are being created and while they are accessing information. The system manages process access to information by using its authorization data and internal mechanisms, such as hardware controls. Because process creation has many areas of security vulnerability, many operating security features concentrate on the area of process (or subject) creation.

When a user attempts to log in to a system, the user provides a user name (a name that will be given to the resulting process) and a password. The password serves as an authenticator that should be known only to the user and to the operating system.

Because a short or obvious password is likely to fail this requirement, the system incorporates many password protection mechanisms that can be invoked by the user or required by the security administrator (see Chapter 7). The operating system is also capable of limiting the number of attempts that an intruder can make to guess a password.

The file of users' passwords is part of the security database that must be protected from unauthorized observation and modification. The system meets this requirement by storing the passwords in a file protected from general access, the system user authorization file (SYSUAF.DAT). The system takes the additional precaution of storing passwords in an encoded form that is hard to use if stolen.

Once the operating system creates a process for a user, it assigns a user identification code or UIC from the user authorization record to that process. The UIC corresponds to the name of the user who created the process (as authenticated by the user's password). In addition, the UIC indicates the user's membership in a group that can correspond to the user's department, project, or function. The system can also attach additional information to the process regarding the creation of the process and the affiliation of the process owner with other groups. This additional information plays a part in the application of the authorization database. (Both Chapter 4 and Chapter 8 discuss UICs.)

2.2.2 Objects

In the reference monitor concept, objects are passive repositories of information. In the OpenVMS system, there are many objects, such as files and devices, that are subject to protection, as described in Table 2-1. The operating system protects objects from unauthorized access and provides a variety of mechanisms (described in Chapter 4 and Chapter 5) for sharing them in a controlled manner. These mechanisms are also used to control access to system resources.

Table 2-1 Objects Protected by Security Controls

Class Name	Definition
Capability	A resource to which the system controls access; currently, the only defined capability is the vector processor.
Common event flag cluster	A set of 32 event flags that enable cooperating processes to post event notifications to each other.
Device	A class of peripherals connected to a processor that are capable of receiving, storing, or transmitting data.
File	Files-11 On-Disk Structure Level 2 (ODS-2) files and directories.
Group global section	A shareable memory section potentially available to all processes in the same group.
Logical name table	A shareable table of logical names and their equivalence names for the system or a particular group.
Queue	A set of jobs to be processed in a batch, terminal, server, or print job queue.
Resource domain	A namespace controlling access to the lock manager's resources.
Security class	A data structure containing the elements and management routines for all members of the security class.
System global section	A shareable memory section potentially available to all processes in the system.
Volume	A mass storage medium, such as a disk or tape, that is in ODS-2 format. Volumes contain files and may be mounted on devices.

2.2.3 Authorization Database

According to the reference monitor model, each subject's authorization to gain access to an object is based on an abstract authorization database. This database is a set of dynamic security attributes that govern a subject's access to an object at any given time. In the OpenVMS system, the database is distributed and stored in association with the objects that must be protected. For example, the authorization data for a file or directory is stored in the file header for that file or directory. Table 2-2 summarizes the information stored in the authorization database.

Table 2-2 Information Stored in the Authorization Database

File	Contents	Data Used to Interpret
SYSUAF.DAT	User names	Logins
	Passwords	Logins
	UICs	Access control checks
NETPROXY.DAT	User names	Logins
NET$PROXY.DAT	User names	Logins
RIGHTSLIST.DAT	Rights identifiers	Access control checks
VMS$OBJECTS.DAT	UICs	Access control checks
	Protection codes	Access control checks
	Access control lists	Access control checks
VMS$AUDIT_ SERVER.DAT	Auditable events	Reporting of events

As Section 2.2.2 suggests, different objects in the OpenVMS system can be shared with differing levels of flexibility. Protected objects are subject to a protection code. This code specifies whether access is allowed or denied to processes run on behalf of system users, the user who is owner of the object, other members of the UIC group of the owner, and all other users.

In addition to the protection code, objects can be shared under control of access control lists (ACLs). ACLs provide a finer granularity of access control than UIC-based protection, especially for user groups or subsets of groups. ACLs list individual users or groups of users who are to be allowed or denied particular types of access to the object. ACLs specify sharing on the basis of UIC identification as well as other groupings or identifiers that can be associated with a process. For example, it is possible to specify that a file should never be read by a process connected to a terminal on a dialup line. Section 2.2.6 uses an access matrix to explain the concept of an ACL. Section 4.4 gives a general discussion of ACLs and identifiers, and Chapter 8 explains how you, as security administrators, can create identifiers and construct ACLs for system resources.

2.2.4 Audit Trail

All security-relevant events can be recorded in an audit log file, sent to an operator terminal, or both. A terminal can be designated as a security operator terminal where all auditable events can be displayed. An audit log file provides a permanent record of security events. Many times a security administrator can find a pattern of activity, called an audit trail, by studying the log file.

The operating system audits the classes of security events shown in Table 2-3 by default. You can select other events for auditing, such as volume mounts or changes to system time.

Table 2-3 Security Auditing Overview

Destination	Events Audited by Default
Log file or terminal display	Authorization database changes
	Intrusion attempts
	Login failures
	Use of DCL command SET AUDIT
	Events triggered by Audit or Alarm ACEs

The audit log allows users and security administrators to record many events. Because it is time-consuming to examine every event, it is most efficient to audit events that will contribute the most information to your security picture. See Chapter 9 for a description of security auditing.

2.2.5 Reference Monitor

In the OpenVMS operating system, the executive performs the role of the reference monitor. All system programs that run in kernel and executive mode help implement the reference monitor, as do the command line interpreter and certain user-mode images that run with privilege. While the volume of code comprising the executive is large, Compaq attempts to ensure that none of the code can be used to bypass system security.

Some privileges can grant a user the authority to modify or subvert the reference monitor. For example, a process with the CMKRNL privilege can execute code of its own within the system kernel, gaining access to the reference monitor's internal data and the internal representation of protected objects. Clearly, granting such critical privileges should be severely limited.

Similarly, give privileges such as SYSPRV and SECURITY only to users whose processes help maintain the reference monitor and authorization database.

2.2.6 Authorization Database Represented as an Access Matrix

The reference monitor model specifies an authorization database, which describes all access authorizations in the system for all subjects and all objects. This database is often represented as an access matrix, which lists subjects on one axis and objects on the other (see Figure 2-2). Each crosspoint in the matrix thus represents the access that one subject has to one object.

Figure 2-2 Authorization Access Matrix

In this access matrix, an asterisk (*) denotes that the subject has access to that object. (Different types of access, such as read and write, are omitted from this example for simplicity.) Thus, subjects B, C, and D all have access to objects W, X, and Y. In addition, subject A has access to objects W and Z, subject D to object V, and subject E to object V.

Breaking up the access matrix by rows yields a capability-based model, in which each subject carries a list of the objects that it can access. Thus, a capability representation of this access matrix would appear as follows:

A: W, Z
B: W, X, Y
C: W, X, Y
D: V, W, X, Y
E: V

It is also possible to break up the access matrix by columns, listing for each object the subjects that have access to it. This results in an authority-based model, implemented in the OpenVMS system by ACLs (see Chapter 4). The ACL representation appears as follows:

V: D, E
W: A, B, C, D
X: B, C, D
Y: B, C, D
Z: A

The ACL and identifier controls used by the operating system combine the properties of both the capability- and authority-based systems. In OpenVMS systems, both subjects and objects carry identifiers. Subjects can access objects if they have matching identifiers and if the objects' access statements grant the requested access.

The result of combining properties of the capability- and authority-based systems is an extremely powerful and flexible system capable of representing complex access matrixes in a compact and convenient manner. Consider what happens to the previous example of an access matrix when some of the cross-points have labels, as shown in Figure 2-3.

Figure 2-3 Authorization Access Matrix with Labeled Cross-Points

Some labeled cross-points can be grouped and treated as a single entity. Thus, the points that are labeled Q in Figure 2-3 represent the access that subjects B, C, and D have to objects W, X, and Y. All the Q points can be considered as a single area of interest. The system provides the concept of identifiers to take practical advantage of this grouping of areas of interest.

You can define identifiers to represent the two groups of access, P and Q, in Figure 2-3. Note that two of the cross-points in the matrix remain unlabeled. Identifiers can also represent individual subjects and thus allow the traditional ACL facility.

To represent the access matrix, the OpenVMS operating system uses two structures, one for each dimension:

• The rights list (RIGHTSLIST.DAT) represents the rows of the access matrix and thus corresponds to the capability-based model. For the matrix in Figure 2-3, you would need the following rights list:
  B: Q
  C: Q
  D: P, Q
  E: P
• ACLs for the protected objects represent the columns of the access matrix. For this example, you would need the following ACLs:
  V: P
  W: A, Q
  X: Q
  Y: Q
  Z: A

Note that the system structures required to represent the access matrix are simpler than either the traditional capability- or authority-based model and require fewer terms in total. In the example, the difference is slight. However, complexity of the access matrix increases with the square of its size.

2.3 Summary: System Security Design

When designing an overall system security plan, ask yourself the following questions:

• How are users associated with subjects? What is the reliability of the authentication mechanism?
• What objects contain sensitive information in this system or application? Is access to those objects controlled?
• Does the authorization database reflect the site's security policy? Who is authorized to gain access to sensitive objects? Are adequate restrictions in place?
• Is the audit trail recording enough or too much information? Who will monitor it? How often will it be examined?
• What programs are functioning as part of the reference monitor? Which users can modify the security policy and the authorization database? Is this the desired configuration?

These considerations, as well as the underlying reference monitor design, apply equally to a timesharing system, a widespread network, or a single application on a system that grants access to records in a file or database. The operating system provides general mechanisms that users and security administrators must apply to achieve system security. See Chapter 6 for more information on designing and implementing a security policy.

Copyright © Compaq Computer Corporation 1998. All rights reserved. Legal

6346PRO_001.HTML