This chapter discusses the major issues in ACMS application performance, ACMS processes that affect system performance, and tools for monitoring your system. This chapter also provides a tuning checklist.

14.1 Major Issues in ACMS Application Performance

Sometimes performance issues are a matter of tuning the application and the system as a whole. For example, modifying SYSGEN and ACMSGEN parameters to ensure that ACMS processes have the resources they need can have a significant effect on the performance of both the application and the entire ACMS system. Take the following steps when tuning an ACMS system:

1. Ensure that ACMS processes have the resources they need, by doing such things as setting the working set size of the Application Execution Controller (EXC) to an appropriate value to decrease paging.
2. Ensure that limits to system resources, such as lookaside lists (preallocated pieces of dynamic memory for terminal I/O or network I/O) or global pages, do not cause sudden delays in processing. An inadequate lookaside list could:
   • Cause the ACMS system or task to hang
   • Prevent new users from signing in
   • Cause non-ACMS processes to stop
   • Exhaust dynamic memory
   • Exhaust workspace pools
   
   Although there is no way to predetermine the correct size for lookaside lists, you can determine adequate limits by routinely observing the ACMS system with the DCL SHOW MEMORY command, the OpenVMS Monitor Utility, or the Software Performance Monitor (SPM) software.
3. Ensure there is enough memory so that ACMS control processes are not subject to swapping.
4. Determine the optimal number of server processes for each server, minimizing the time that servers wait for tasks and that tasks wait for servers. Then, change the application definition to set the minimum and maximum values for server process count to their optional numbers. This eliminates excessive server process creation and deletion.
5. Determine whether or not you can trade off resources to benefit the work of most importance at the expense, if necessary, of other activities. For example, you can:
   • Use servers of lower priority for less important work
   • Require certain processing work, such as computations, to be run in batch mode
   • Set limits on the number of ACMS users and applications to ensure that resources are available on the system for other non-ACMS work

Most of the quotas and SYSGEN parameters established during post-installation are sufficient for your ACMS system. You might have to change them, however, if you modify your hardware, add or delete an application, or change the number of users on your system. If you change SYSGEN parameters or for some reason the system resources are insufficient, the ACMS system might notify you with an error message, or it might not have sufficient resources to send an error message.

The best way to determine the optimal settings for parameters is to monitor your system regularly and establish some base numbers to work with. Only through regular monitoring can you recognize normal and abnormal activity, the effect of different workloads on the system, and changes in throughput rates. Also, by monitoring your system before and after making changes to it, you can measure the effects of the change against a predefined goal.

14.2 Monitoring Tools

Tuning involves collecting and reporting on system processes to improve performance. A number of tools are available that can help you collect information about an active ACMS system and applications:

• ACMS SHOW commands
  The displays returned by these commands, especially by ACMS/SHOW APPLICATION and ACMS/SHOW APPLICATION/CONTINUOUS, provide information you can use to determine the optimal number of server processes required for your ACMS system. By examining the number of tasks waiting for servers and servers waiting for tasks, you can determine whether some servers are not being used, or too many servers are being created to meet increased demand. See Chapter 9 for a detailed discussion about these ACMS SHOW commands.
  In setting the minimum server processes, set the value as high as necessary to prevent the too frequent creation of additional server processes during the time of peak application use. Also set the value as low as you can to conserve memory. Once you have determined the optimal setting for the minimum value, you can set the maximum to that same value, to ensure that no additional servers are created, even at times of peak application load. Users queue for the available servers instead. See Compaq ACMS for OpenVMS ADU Reference Manual for more information about using ADU to set the optimal number of server processes.
• ACMS Audit Trail Logger
  The task selections logged in the Audit Trail Logger can help you determine which tasks are most heavily used and the times of heaviest use. This can be useful in determining how to modify an application. See Chapter 12 for a detailed discussion on the ACMS Audit Trail Logger and audit trail log files.
• OpenVMS Monitor Utility and DCL SHOW commands
  The OpenVMS Monitor Utility and the DCL SHOW commands are two important performance management tools provided with the OpenVMS operating system.
  The OpenVMS Monitor Utility returns information that can help identify and isolate resources causing either ACMS or system-level problems: inadequate values for small request packets (SRP), I/O request packets (IRP), and large request packets (LRP), for example, or inadequate values for paged and nonpaged pool. The paging information is useful in determining appropriate working set sizes for ACMS processes. See OpenVMS System Management Utilities Reference Manual for information about the OpenVMS Monitor Utility.
  The SHOW commands display the current status of a process, the system, and the devices on the system. Use the SHOW MEMORY command, for example, to ensure there is adequate memory available to prevent swapping of the ACMS control processes. See OpenVMS DCL Dictionary for information on the DCL SHOW commands.
• Software Performance Monitor (SPM)
  The SPM is an OpenVMS layered product that provides information on resource usage. SPM is particularly useful in determining working set sizes. Because the CPU cost of running SPM is low, it is a good tool for gathering routine performance information. Also, SPM provides extensive tools for reporting the data it gathers. The SPM reports can provide significant help in identifying and solving performance problems. SPM's graphic representation of the information gathered makes it easier to identify performance problems than with MONITOR. See the documentation provided with SPM for more information on this product.
• DEC/Test Manager (DTM)
  The DTM is an OpenVMS layered product that is most useful for regression and functional testing. DTM can also be useful in setting up a controlled simulation of application activity to allow you to isolate a performance problem. DTM allows you to have multiple processes on the same CPU run the same script, simulating multiple ACMS users. Although the resources DTM requires can affect the overall performance of the system, this controlled simulation can help you determine which resources are being used. See DTM documentation for more information on this product.

These tools can help identify areas in which adjustment of SYSGEN parameters or user name characteristics can improve ACMS or overall system performance.

14.3 ACMS Processes that Affect Performance

These processes control the ACMS system:

• ACMS Central Controller (ACC)
• Terminal Subsystem Controller (TSC)
• Command Process (CP)
• Application Execution Controller (EXC)
• Server Process
• Audit Trail Logger
• Queued Task Initiator (QTI)

Of these processes, the CP, EXC, and server process have the greatest effect on the performance of an application. In addition, the QTI process has a very significant effect on the performance of the ACMS Queued Task Facility.

The CP handles all ACMS sign-ins and ACMS menu displays, performs exchange steps for distributed applications, and interprets commands issued at the ACMS menu level. The EXC controls all server processes and handles all exchange steps for local applications. The server process handles all of the processing step work for an application, including all database I/O and computational work. The QTI dequeues queued task elements from one or more task queues and initiates queued tasks in ACMS applications.

Minimizing the number of ACMS processes without overloading them while providing them sufficient system resources results in the best performance. Since most of the ACMS processes control many users, they need larger working sets to function than a typical interactive process. For the same reason, set the OpenVMS priority of the ACMS processes at least as high as that of an interactive process.

14.4 Configuration of Hardware

This section discusses how the various processes and procedures of an ACMS system use the different components of a computer system. If you understand how ACMS uses memory, CPU power, and disk I/O, you can make better choices about how much and what kind of hardware you need. The section also includes a sample configuration suitable for distributed applications.

A properly configured computer system consists of three elements:

• Main memory sufficient to contain the working sets of active processes, thus minimizing paging and swapping
• CPU power sufficient to support operating system requirements and to carry out computation required by the application fast enough to sustain desired throughput
• Disks sufficient in size to store data, application sources, and images, and sufficient in number and speed to support I/O throughput of the application

Before proceeding with plans for hardware configuration, have a clear understanding of requirements in terms of application size, database requirements, and expected performance. Estimates of hardware requirements are much more meaningful if based on observation of an existing application, such as a prototype. All estimates should be based on peak load, not average load.

When you estimate how much memory your configuration should include, estimate the memory required for the operating system plus the working set sizes of all the processes in your application. The working set size for a process must be sufficient to avoid excessive page faulting.

To estimate how much memory OpenVMS uses, start with a base figure for OpenVMS itself and operating system processes such as ERRFMT, OPCOM, and JOB_CONTROL. Include additional memory for application processes. Remember to include the memory needs of layered products installed on your system, such as DATATRIEVE, TDMS, and COBOL.

When determining the amount of memory each application process needs, consider two factors: the number of global pages and the number of process-private pages. Global pages are pages that can be shared among several similar processes, and should only be counted once. Process-private pages are specific to a single process and should be counted for each process. The OpenVMS MONITOR PROCESS command can assist you in determining global and process-private pages for the processes you need to consider. Monitor a prototype application to estimate memory needs.

For an ACMS application, the following processes need to be considered when estimating required memory:

• ACC
  There is one ACC on your computer. The ACC is the central control point for the ACMS system.
• CPs
  How many Command Processes is your system going to run? Broadly speaking, you can count on 1 CP for every 20 users.
• EXC
  Consider both global pages used by the EXC and process-private pages needed by each user of the application. The amount of memory is determined by the number of active tasks, but at this level of estimation, the number of users is more likely to be known than the number of active tasks. If this process is short of memory, the performance of the application is adversely affected.
• Servers
  Include sufficient working sets for each server and for the processing work being done in the server. Memory requirements are determined by the code you write for the processing work, as well as the database management system you use.
• Non-ACMS processes
  Include global and process-private pages for any non-ACMS processes that may be running on your system.

Add up all of your estimates for an estimate of the total memory required for your application.

You can refine your requirements after the hardware configuration is in place and the system is up and running. Chapter 10 discusses means of monitoring and tuning memory usage for your system. It also contains information helpful in calculating quotas and parameters for the processes discussed in this section.

14.4.2 Estimating CPU Requirements

How big a CPU does your application require? Performance is almost always spoken of in terms of throughput, the rate of processing computer computations. For example, the number of transactions per second is a measure of the speed at which the system works. The performance of an ACMS system is determined by the size and power of the CPU and the specifics of your application design.

ACMS applications conserve CPU energy more than traditional applications because of the way in which ACMS applications share resources. For example, the server initialization procedure binds the database once for a server process that serves many users. This puts less load on the CPU than a traditional timesharing application, in which each user binds the database in individual processes.

An ACMS application can take an existing image, remove the database binds, and put them into an initialization procedure. The resulting image runs in a single-step task. The performance gains are impressive, even before the image is converted to a multiple-step task.

An ACMS application needs to create fewer server processes and to activate fewer images for a group of users than a traditional application, because the task group shares resources among many users. Also, the ACMS use of specialized processes to share memory among many users reduces memory management overhead, another factor in CPU usage.

The design of a task also has an effect on CPU usage. Avoiding scrolled regions and the processing required to map a large array to a TDMS form were discussed in the section on the user interface for the multiple-step task. Database access methods also affect CPU usage. Use DATATRIEVE, RDO, or DBQ to prototype your database access methods, and observe the effect of various methods on CPU usage. Because of the cost of communicating group and user workspaces from server process for computation to Command Process for I/O, the use of such workspaces often requires more CPU resources than the benefits of such use warrants. Take the costs of communication and server use into account as you estimate the size of CPU required for your application.

You may have different CPU requirements, depending on whether throughput or best response time is your primary goal. To get maximum throughput and best response time, you may need more CPU power than for just maximum throughput. Response time increases as the limit of CPU capacity (maximum throughput) is reached.

Again, it is useful to monitor a prototype and to determine from that the additional CPU power required for the fully implemented system.

14.4.3 Estimating Disk Requirements

Applications that are suitable for development with ACMS tend to be I/O-intensive, requiring a significant amount of disk resources. Update and inquiry tasks against a database, for example, use more disk I/O than an application doing scientific computation, which is a heavy user of CPU power.

When determining how many and what kind of disks you need for your hardware, take these factors into account:

• How much data, application, and operating system space is required
• What disks support the I/O throughput required by your system

Determining the number and type of disks needed to support your I/O throughput requirements depends on a number of factors. You need to understand the throughput rate of the disks you are considering. How many I/O requests per second can the disk process before applications have to begin queuing for disk access? The rate varies widely depending on the type of disk and the kind of I/O your application does---for example, whether it is synchronous or asynchronous. The database management system in use also affects the rate of I/O throughput.

Factors in disk configuration specific to ACMS include the location of the audit trail log file. Determine the location of this file by defining the logical name ACMS$AUDIT_LOG before you start up the ACMS system. See Chapter 12 for a discussion of the audit trail log file. Be careful not to locate this file on the same disk as your DBMS root file, or Rdb database file, or the file where the index of an RMS indexed file is located. The Audit Trail Logger should not have to contend with database processing for access to the same disk.

You can create a prototype of the database access involved in your application and monitor the I/O requirements. Use DATATRIEVE, RDO, or DBQ to model the transactions you plan to include in your application, and use the DCL command MONITOR/DISK to observe the actual I/O rate. Proceeding from observed rates of I/O throughput in the prototype can help you figure out what the disk requirements are when the system is fully implemented.

Once you have an idea of I/O requests per transaction and the desired processing throughput rate in terms of transactions per second, multiply the two figures together to get the total I/O rate. Divide the total I/O rate by the I/O throughput rate of the disk you are considering, and you should have an idea of how many disks you need to support the I/O throughput you require for your application.

((requests/transaction * transactions/second) / throughput rate) = disks

Keep in mind that:

• For good response time, the I/O throughput rate must be less than the disk's maximum capacity.
• There may be limits imposed on the I/O throughput by your database management system.

Figure 14-1 shows a sample configuration: distributed ACMS running on two MicroVAX II front-ends for terminal processing and an OpenVMS Cluster for disk processing on the back-end.

In Figure 14-1, terminal processing for a distributed ACMS system is handled by two MicroVAXs. For disk processing, two VAX 8200s and a VAX 8600 are clustered with two HSC50 disk controllers.

The dual HSC50s allow quick disk access. The use of RA81 disk drives provides large amounts of disk storage in a minimal amount of floor space. The MicroVAX processors allow the application manager to distribute the terminals to front-ends and reserve the capacity of the OpenVMS Cluster for database access and computation.

Figure 14-1 Sample Distributed ACMS Configuration

The CP handles many users at one time. The performance of the CP depends on the number of users it must control and the rate at which those users are selecting menus or tasks. The number of users that the CP controls is determined by the ACMSGEN parameter MAX_TTS_CP.

To determine whether or not the CP is configured properly, monitor one of the command processes during peak hour conditions using the DCL command SHOW PROCESS/CONTINUOUS. If the CP remains idle for long periods of time, it may be able to handle more users. Allowing it to control more users reduces the number of Command Processes ACMS needs and also the amount of system resources ACMS uses.

If the CP remains in a computable state for a period of time, it may be overloaded. An overloaded CP can make menu displays and response time seem slow to the users. When this is the case, reduce the number of terminals the CP controls by changing the TERMINALS_PER_CP parameter in the ACMVARINI.DAT file and running the ACMSPARAM.COM procedure. See Chapter 10 for information about running ACMSPARAM.COM.

If the CP seems to be paging heavily, its working set might be set too low. When this is the case, increase the working set size for the CP using the OpenVMS Authorize Utility. Restricting the CP by limiting its working set can result in poor performance for all the users it controls.

Another source of poor CP performance might be the priority at which the CP is running. The ACMSGEN parameter CP_PRIORITY controls the priority at which the CP runs. Se this value at least as high as an interactive user's priority.

As much as possible, perform such adjustments using the ACMSPARAM.COM procedure rather than the ACMSGEN Utility because ACMSPARAM adjusts other related parameters automatically. In some instances, such as adjusting dynamic system parameters, it might be necessary to use ACMSGEN (see Chapter 11).