Reliable Transaction Router

Application Design Guide

Order Number: AA–REPMB–TE

This guide explains how to design application programs for use with Reliable Transaction Router.

Compaq, the Compaq logo, AlphaServer, TruCluster, VAX, and VMS Registered in U.S. Patent and Trademark Office.

DECnet, OpenVMS, and PATHWORKS are trademarks of Compaq Information Technologies, L.P.

Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation.
Intel is a trademark of Intel Corporation.
UNIX and The Open Group are trademarks of The Open Group.

All other product names mentioned herein may be trademarks or registered trademarks of their respective companies.

Confidential computer software. Valid license from Compaq required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.

Compaq shall not be liable for technical or editorial errors or omissions contained herein.

The information in this publication is subject to change without notice and is provided "AS IS" WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK ARISING OUT OF THE USE OF THIS INFORMATION REMAINS WITH RECIPIENT. IN NO EVENT SHALL COMPAQ BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL, PUNITIVE OR OTHER DAMAGES WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION OR LOSS OF BUSINESS INFORMATION), EVEN IF COMPAQ HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE FOREGOING SHALL APPLY REGARDLESS OF THE NEGLIGENCE OR OTHER FAULT OF EITHER PARTY AND REGARDLESS OF WHETHER SUCH LIABILITY SOUNDS IN CONTRACT, NEGLIGENCE, TORT, OR ANY OTHER THEORY OF LEGAL LIABILITY, AND NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.

Reading Path

Using Concurrent Servers

Using Threads

Using Multiple Partitions

 Router Failover

Server Failover

        Concurrent Servers

        Standby Servers

        Standby Support

The Cluster Environment

             True Clusters

             Host-Based Clusters

Recognized Clusters

Unrecognized Clusters

Clustered Resource Managers and Databases

Failure Scenarios with RTR Standby Servers

Shadow Servers

Tolerating Site Disaster

The Role of Quorum

Surviving on Two Nodes

Partitioning

Transaction Serialization

Transaction Serialization Detail

            RTR Commit Group

            Independent Transactions

CSN Ordering

CSN Boundary

Batch Processing Considerations

Recovery after a Failure

Journal Accessibility

Journal Sizing

Replay Anomalies

Design for Performance

RTR Performance Tests

Summary

 Concurrent Servers

 Partitions and Performance

 Facilities and Performance

 Router Placement

 Broadcast Messaging

 Making Broadcasts Reliable

 Large Configurations

 Using Read-Only Transactions

 Making Transactions Independent

Configuration for Operability

Firewalls and RTR

Avoiding DNS Server Failures

Batch Procedures

RTR Requirements on Applications

Be Transaction Aware

Avoid Server-Specific Data

Be Independent of Time of Processing

Use Two Identical Databases for Shadow Servers

Make Transactions Self-Contained

Lock Shared Resources

Ensuring ACID Compliance

Ensuring Atomicity

Ensuring Consistency

Ensuring Isolation

Ensuring Durability

Transaction Dependencies with Concurrent Servers

Server-Side Transaction Timeouts

Two-Phase Commit Process

Prepare Phase

Nested Transactions

RTR Messaging

Transactional Messages

Broadcast Messages

Flow Control

Sequencing of Broadcasts

Sequencing Relative to Transaction Delivery

Recovery of Broadcasts

Lost Broadcasts

Coping with Broadcast Loss

Broadcast Messaging Processes

User Events

RTR Events

Location Transparency

Handling Error Conditions

Using Locks

Oracle Locking

Privileges Required

Overriding Default Locking

Oracle Explicit Data Locking

Table Locks

Acquiring Row Locks

Setting SERIALIZABLE and ROW_LOCKING Parameters

Using the LOCK TABLE Statement

Summary of Locking Options

Non-Default Locking Behavior

Distributed Deadlocks

Providing Parallel Processing

Establishing Read-Only Sites

Resolving Idempotency Issues

Designing for a Heterogenous Environment

Using the Multivendor Environment

Upgrading from RTR Version 2 to RTR Versions 3 and 4

Transactional Messaging with the C++ API

Data-Content Routing with the C++ API

Changing Transaction States

RTR Message Types

Transactional Message Processing

Message Processing Sequence

Accept Processing

Starting a Transaction

Identifying the Transaction

Accepting a Transaction

Rejecting a Transaction

Ending a Transaction

Processing Summary

Administering Transaction Timeouts

Two-Phase Commit

Initiating the Prepare Phase

Commit Phase

Explicit Accept, Explicit Prepare

Implicit Prepare, Explicit Accept

Server Transaction States

Router Transaction States

Transaction Recovery

Recovery Examples

Recovery: Before Server Accepts

Recovery: After Server Accepts

Recovery: After Database is Committed

Recovery: After Beginning a New Transaction

Exception Transaction Handling

Coordinating Transactions

Integration of RTR with Resource Managers

Distributed Transaction Model

Broadcast Messaging Processes

User Events

RTR Events

Authentication Using Callout Servers

Transportation Example

Stock Exchange Example

Banking Example

OpenVMS Cluster

Tru64 UNIX TruCluster

Windows NT Cluster

Server and Active Transaction States in a Shadow Server

Server and Transaction States in a Standby Server

Specifying Server Type

Server Failover

Concurrent Servers

Standby Servers

Shadow Servers

Making Transactions Independent

Specifying Server Type

Concurrent Servers

Diagnosing Performance Problems

Dual-Rail Setup

Physical Network Card Setup

RTR Facility Setup

DNS Name Server Support

Tunnel Configurations

Figure 2-1: Transaction Flow with Concurrent Servers and Multiple Partitions

Figure 2-2: Transaction Flow on Standby Servers

Figure 2-3: RTR Standby Servers

Figure 2-4: Transaction Flow on Shadow Servers

Figure 2-5: Two Sites with Shadowing and Standby Servers

Figure 2-6: Configuration with Quorum Node

Figure 2-7: Commit Sequence Number

Figure 2-8: CSN Vote Window for the C++ API

Figure 2-9: CSN Vote Window for the C API

Figure 2-10: Recovery after a Failure

Figure 2-11: Single-Node TPS and CPU Load by Number of Channels

Figure 2-12: Single-Node TPS and CPU Load by Message Size

Figure 2-13: Two-Node TPS and CPU Load by Number of Channels

Figure 2-14: Two-Site Configuration

Figure 2-15: Message Fan-Out

Figure 3-1: Transactional Shadow Servers

Figure 3-2: Shadow Servers and Third Common Database (not recommended)

Figure 3-3: Uncertain Interval for Transactions

Figure 3-4: Concurrent Server Commit Grouping

Figure 3-5: Transactional Messaging

Figure 3-6: Transactional Messaging Interaction

Figure 3-7: Deadly Embrace

Figure 3-8: Scenario for Distributed Deadlock

Figure 4-1: Transactional Messaging with the C++ API

Figure 4-2: C++ API Calls for Transactional Messaging

Figure 4-3: Flow of a Transaction

Figure 4-4: Accept Processing

Figure 5-1: RTR C API Calls for Transactional Messaging

Figure 5-2: MS DTC and RTR

Figure A-1: Transportation Example Configuration

Figure A-2: Stock Exchange Example

Figure A-3: Banking Example Configuration

Figure B-1: OpenVMS CI-Based Cluster

Figure B-2: Tru64 UNIX TruCluster Configuration

Figure B-3: Windows NT Cluster

Figure C-1: Server Events and States with Active Transaction Message Types

Figure C-2: Server States after Standby Events

Figure G-1: Dual-Rail Configuration with Network Cards on a Router

Figure G-2: Dual-Rail Configuration with Network Cards on a Frontend

Table 3-1: Backend Transaction States

Table 3-2: Frontend Transaction States

Table 3-3: RTR Error Codes

Table 3-4: LOCK TABLE Statements

Table 3-5: Summary of Locking Options

Table 3-6: Non-Default Locking Behavior

Table 4-1: RTR Message Types

Table 4-2: RTRServerTransactionController Methods

Table 4-3: RTRClientTransactionController Methods

Table 4-4: Prepare Phase Server States

Table 4-5: Commit Phase Server States

Table 4-6: Prepare Phase Router States

Table 4-7: Typical Server Application Transaction Sequences

Table 5-1: RTR C API Calls by Category

Table 5-2: RTR C API Calls

Table 5-3: How RTR Reports Aborted Transactions

Table 5-4: RTR Message Types

Table 5-5: Client Action Based on Journal State

Table 5-6: Server Participation

The goal of this document is to assist the experienced application programmer in designing applications for use with Reliable Transaction Router (RTR). Here you will find conceptual information and some implementation details to assist in:

• Creating new applications
• Updating existing applications

As an application programmer, you should be familiar with the following concepts:

• Distributed systems
• Client/server environment
• C or C++ programming
• Transaction processing

If you are not familiar with these software concepts, you will need to augment your knowledge by reading, taking courses, or through discussion with colleagues. You should also become familiar with the other books in the RTR documentation kit, listed in Related Documentation. Before reading this document, become familiar with the information in Reliable Transaction Router Getting Started.

This document is intended to be read from start to finish; later you can use it for reference.

Additional resources in the RTR documentation kit include:

Other books that can be helpful in developing your transaction processing application include:

• Philip A. Bernstein, Eric Newcomer, Principles of Transaction Processing, Morgan Kaufman, 1997
• Jim Gray, Andreas Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufman, 1992
• Oracle8 and Oracle 8i Application Developer's Guides

A useful web site for tips and tutorials on Oracle SQL is http://photo.net/sql .

You will find additional information on RTR and existing implementations on the RTR web site http://www.compaq.com/products/software/middleware/rtr/ .

Compaq welcomes your comments on this guide. Please send your comments and suggestions by email to rtrdoc@compaq.com. Please include the document title, date from the title page, order number, section and page numbers in your message.

Figure 1 shows the reading path to follow when using the Reliable Transaction Router information set.

This document is for the application programmer who is developing an application for use with Reliable Transaction Router (RTR). Here you will find information on using RTR in the design and development of an application. The main emphasis is on providing design suggestions and considerations for writing the RTR application. Example designs describing existing applications that use RTR show implementations exploiting RTR features, and provide real examples where RTR is in use.

The C++ object-oriented interface is new with RTR V4.0. Most of the material in this document is generic to RTR and not specific to either the C or C++ API (application programming interface). However, some implementation specifics for each API are shown in appendices. Each API also has its own reference document, the RTR C++ Foundation Classes manual for the C++ API, and the RTR C Application Programmer's Reference Manual for the C API. This document also contains material specific to each API in separate chapters.

In designing your application, Compaq recommends that you use object-oriented analysis and design techniques, whether or not you are using the new object-oriented API. This methodology includes the following:

• Performing use-case analysis
• Tracing scenarios
• Determining actors and classes
• Establishing object interactions

Discussing this methodology is outside the scope of this document, but you can find more information in the following books, among others:

• James Rumbaugh, Michael Blaha, William Premerlani, Frederick Eddy, William Lorensen, Object-Oriented Modeling and Design, Prentice-Hall, 1991
• Martin Fowler, Kendall Scott, UML Distilled Applying the Standard Object Modeling Language, Addison-Wesley, 1997

When designing your application:

• Consider your application requirements fully.
• Work through the entire design and its operational results.
• Understand both the logical and physical design of your database, including any partitioning of your database.

A design flaw can be very expensive or impossible to correct in your application, so doing a thorough design, fully discussed and understood by your team, is essential to the ultimate success of your application in operation. One goal of this document is to help you understand some of the finer subtleties of the product, to help you design an optimum application.

Reliable Transaction Router (RTR) is transactional messaging middleware used to implement highly scalable, distributed applications with client/server technologies. With RTR, you can design and implement transaction management applications, take advantage of high availability, and implement failure-tolerant applications to maximize uptime. RTR helps ensure business continuity across multivendor systems and helps maximize uptime.

Implementing an application to use RTR embeds high availability capabilities in the application. Furthermore, RTR greatly simplifies the design and coding of distributed applications because, with RTR, client-server interactions can be bundled together in transactions. In addition, RTR applications can easily be deployed in highly available configurations.

RTR supports applications that run on different hardware and different operating systems. RTR has not been built to work with any specific database product or resource manager (file system, communication link, or queuing product), and thus provides an application the flexibility to choose the best product or technology suited for its purpose. There are, however, some resource managers with which RTR provides tight coupling. (For more information on using this tighter coupling, see the section in this manual on Using XA.) For specifics on operating systems, operating system versions, and supported hardware, see the Reliable Transaction Router Software Product Description for each supported operating system.

RTR provides several benefits for your application:

• High availability

• High security

• Data retention

• High transaction performance

RTR provides programming interfaces for use in implementing applications that work with RTR, including:

• The C++ API
• The C API

The C++ API is described in the RTR C++ Foundation Classes manual and used in descriptions and examples in this manual. The C API is described in the RTR C Application Programmer's Reference Manual and used in examples in this manual.

The C++ API is an object-oriented application programming interface for RTR. With the C++ API, as an application developer, you can design and implement applications to take advantage of object-oriented (OO) design principles, including patterns and code reuse. The C++ API is backwards compatible and can be used for new applications or integrated into existing RTR applications.

As Figure 1-1 illustrates, the C++ API sits on top of the C API, which is a layer on top of RTR.

Client applications and server applications can use both the C++ API and the C API or just the C++ API. The C++ API maps RTR data elements, data structures, and C API functionality into the object-oriented design world.

The C++ API:

• Is 100% compatible with existing applications.
• Provides the features of the flat C API, plus many more.
• Allows RTR applications to focus on their business logic instead of the details of RTR.
• Provides all the benefits of OO design.
• Allows existing applications to benefit from the new interface in many ways.

C++ API and RTR Technology

The C++ API code resides beneath application or business logic code. Thus, the C++ API interfaces directly with RTR while application code does not. This transparency simplifies the development and implementation of transaction processing solutions with RTR.

OO Benefits

The C++ API was created to assist RTR customers who:

• Need to create RTR system management routines.
• Are writing common application code.
• Can take advantage of the benefits of OO design and development.
• Write some form of OO wrapper to the existing API.

The benefits include:

• Higher quality software.
• Lower maintenance costs.
• Reduced development time.
• Ease of extensibility.

With the C++ API, applications can inherit functionality from RTR.

The C++ API Provides Ease of Use

The C++ API provides a simplified way for you to implement RTR solutions. With the C++ API:

• Each major RTR concept is represented by its own individual class.
• Class factory support is provided for data objects.
• Clients and servers connect through transaction controller objects (automates and hides C API channel use).
• You do not need to provide handling for all messages and events; default handling is provided.
• The sending and receiving of data is abstracted to a high level.
• Simple methods let you obtain RTR internal information without a need to know the internals of RTR.

C++ API Design

The C++ API upgrades RTR technology by providing a set of classes that streamlines the development and implementation of RTR transaction processing applications. The C++ API has been designed to:

• Provide 100% functional and binary compatibility (backward compatibility) with existing applications.
• Provide an object model that can be implemented in any OO language.
• Provide an object model from which new and existing applications can benefit.
• Perform "common" tasks for the application (for example, for an application to receive a message for a client/server connection).
• Provide default implementation for applications where appropriate.
• Provide an easy-to-use framework for handling messages and events that applications can extend to suit their own business logic.

Use of the C++ API

The C++ API can be used to:

• Develop entirely new applications.
• Upgrade existing applications on a single tier of their application - client or server.
• Integrate individual C++ API classes into existing applications. For example, existing applications can easily use the property classes.
• Develop system management routines for both new and existing applications.

• Add diagnostics to the application that can be viewed in an integrated display with RTR counters.

The C API enables applications to be written with RTR calls using the C programming language. The C API is described in the Reliable Transaction Router C Application Programmer's Reference Manual.

To assist you in designing fault-tolerant RTR transaction processing applications, this chapter addresses configuration and design topics. Specifying how your RTR components are deployed on physical systems is called configuration. Developing your application to exploit the benefits of using RTR is called design or application design.

The following topics are addressed:

• Tolerating Process Failure
• Tolerating Storage Device Failure
• Tolerating Node Failure
• Tolerating Site Disaster
• Design for Performance
• Configuration for Operability

Short examples for both C++ and C APIs are available in appendices. The RTR C++ Foundation Classes manual and the RTR C Application Programmer's Reference Manual provide longer code examples. Code examples are also available with the RTR software kit in the examples directory.

Tolerating Process Failure

Using Concurrent Servers

Using Threads

To configure a system that tolerates storage device failure well, consider incorporating the following in your configuration and software designs:

• Shadowed storage devices
• RAID storage devices

Further discussion of these devices is outside the scope of this document.

Tolerating Node Failure

RTR failover employs concurrent servers, standby servers, shadow servers, and journaling, or some combination of these. To survive node failure, you can use standby and shadow servers in several configurations.

If the application starts up a second server for the partition, the server is a standby server by default.

Consider using a standby server to improve data availability, so that if your backend node fails or becomes unavailable, you can continue to process your transactions on the standby server. You can have multiple standby servers in your RTR configuration.

The time-to-failover on OpenVMS and Tru64 UNIX systems is virtually instantaneous; on other operating systems, this time is dictated by the cluster configuration that is in use.

The C++ API includes management classes that enable you to write management routines that can specify server type, while the C API uses flags on the open channel calls.

Router Failover

RTR deals with router failures automatically and transparently to the application. In the event of a router failure, neither client nor server applications need to do anything, and do not see an interruption in service. Consider router configuration when defining your RTR facility to minimize the impact of failure of the node where a router resides. If possible, place your routers on independent nodes, not on either the frontend or backend nodes of your configuration. If you do not have enough nodes to place routers on separate machines, configure routers with backends. This assumes a typical situation with many client applications on multiple frontends connecting to a few routers. For tradeoffs, see the sections on Design for Performance and Design for Operability sections of this chapter.

Provide multiple routers for redundancy. For configurations with a large number of frontends, the failure of a router causes many frontends to seek an alternate router. Therefore, configure sufficient routers to handle reconnection activity. When you configure multiple routers, one becomes the current router. If it fails, RTR automatically fails over to another.

For read-only applications, routers can be effective for establishing multiple sites for failover without using shadowing. To achieve this, define multiple, nonoverlapping facilities with the same facility name in your network. Then provide client applications in the facility with the list of routers. When the router for the active facility fails, client applications are automatically connected to an alternate site. Read-only transactions can alternatively be handled by a second partition running on a standby server. This can help reduce network traffic.

When a router fails, in-progress transactions are routed to another router if one is available in that facility.

Server Failover

Server failover in the RTR environment can occur with failure of concurrent, standby, transactional shadow servers, or a combination of these. Servers in a cluster have additional failover attributes. Conceptually, server process failures can be contrasted as follows:

• A concurrent server employs a different process running on the same node.
• A standby server that becomes active employs a different process on a different node in the same cluster.
• A shadow server employs a different process on a different node in a different cluster.

Failover of any server is either event-driven or timer-based. For example, server loss due to process failure is event-driven and routinely handled by RTR. Server loss due to network link failure is timer-based, with timeout set by the SET LINK/INACTIVITY timer (default: 60 seconds). For more information on setting the inactivity timer, see the SET LINK command in the RTR System Manager's Manual.

The server type of a specific server depends on whether it is in a cluster environment and what other servers are declared for the same key range. In a cluster environment, a server declared as a standby and as a shadow becomes a standby server if there is another server for the same key range on the cluster. In a non-cluster environment, the server becomes a shadow server. For example, Figure 2-1 illustrates the use of concurrent servers to process transactions for Partition A.

When one of the concurrent servers cannot service transactions going to partition A, another concurrent server (shown by the dashed line) processes the transaction. Failover to the concurrent server is transparent to the application and the user.

Concurrent Servers

Concurrent servers are useful in environments where more than one transaction can be performed on a database partition at one time to increase throughput.

Standby Servers

Standby servers provide additional availability and node-failure tolerance. Figure 2-2 illustrates the processing of transactions for two partitions using standby servers.

Figure 2-2: Transaction Flow on Standby Servers

When the configuration is operating normally, the primary servers send transactions to the designated partition (solid lines); transactions "A" proceed through primary server A to database partition A and transactions "B" proceed through primary server B to database partition B. However, when the primary server fails, the router reroutes transactions "A" through the standby server A’ to partition A, and transactions "B" through the standby server B’ to database partition B. Note that standby servers for different partitions can be on different nodes to improve throughput and availability. For example, the bottom node could be the primary server for partition B, with the top node the standby. The normal route is shown with a solid line, the standby route with a dashed line.

When the primary path for transactions intended for a specific partition fails, the standby server can still process the transactions. Standby servers automatically take over from the primary server if it fails, transparently to the application. Standby servers recover all in-progress transactions and replay them to complete the transactions.

As shown in Figure 2-2, there can be multiple standby servers for a partition.

Standby Support

Failover and transaction recovery behave differently depending on whether server nodes are in a cluster configuration. Not all "cluster" systems are recognized by RTR; RTR recognizes only the more advanced or "true" cluster systems.

Figure 2-3 shows the components that form an RTR standby server configuration. Two nodes, N1 and N2, in a cluster configuration are connected to shared disks D1, D2 and D3. Disks D1 and D2 are dedicated to the RTR journals for nodes N1 and N2 respectively, and D3 is the disk that hosts the clustered database. This is a partitioned database with two partitions, P1 and P2. Under normal operating conditions, the RTR active server processes for each partition, P1_A and P2_A run on nodes N1 and N2 respectively. The standby server processes for each partition run on the other node, that is, P1_S runs on N2 and P2_S runs on N1. In this way, both nodes in the cluster actively participate in the transactions and at the same time provide redundancy for each other. In case of failure of a node, say N1, the standby server P1_S is activated by RTR and becomes P1_A and continues processing transactions without any loss of service or loss of in-flight transactions. Both the active and standby servers have access to the same database and therefore both can process the same transactions.

Failover between RTR standby servers behaves differently depending on the type of cluster where RTR is running. Actual configuration and behavior of each type of cluster depends on the operating system and the physical devices deployed. For RTR, configurations are either true clusters or host-based clusters.

True Clusters

True clusters are systems that allow direct and equal access to shared disks by all the nodes in the cluster, for example OpenVMS and Tru64 UNIX (Version 5.0). Since concurrent access to files must be managed across the cluster, a distributed lock manager (DLM) is typically used as well. Since all cluster members have equal access to the shared disks, a failure of one member does not affect the accessibility of other members to the shared disks. This is the best configuration for smooth operation of RTR standby servers. In such a cluster configuration, RTR failover occurs immediately if the active node goes down.

Host-Based Clusters

Host-based clusters include MSCS on Windows NT, Veritas for Solaris, IBM AIX and Tru64 UNIX (versions before 5.0). These clusters do not allow equal access to the disks among cluster members. There is always one host node that mounts the disk locally. This node then makes this disk available to other cluster members as a shared disk. All cluster members accessing this disk communicate through the host. In such a configuration, failure of the host node affects accessibility of the disks by the other members. They will not be able to access the disks until the host-based cluster software appoints another host node and this node has managed to mount the disks and export them. This will cause a delay in the failover, and also introduces additional network overhead for the other cluster members that need to access the shared disks.

Recognized Clusters

The cluster systems currently recognized by RTR are: OpenVMS, and TruCluster systems on Tru64 UNIX. Cluster behavior affects how the standby node fails over and how transactions are recovered from the RTR journal. For RTR to coordinate access to the shared file system across the clustered nodes, it uses the Distributed Lock Manager on both OpenVMS and Tru64 UNIX.

Failover

When the active server goes down, RTR fails over to the standby server. Before that, RTR on the upcoming active node attempts to perform a scan of the failed node’s journal. Since this is a clustered system, the cluster manager fails over the disks as well to the new active node. RTR will wait for this failover to happen before it starts processing new transactions.

Transaction Recovery

In all the recognized clusters, whenever a failover occurs, RTR attempts to recover all the in-doubt transactions from the failed node’s journal and replay them to the new active node. If RTR on the upcoming active server node cannot access the journal of the node that failed, it waits until the journal becomes accessible. This wait allows for any failover time in the cluster disks. This is particularly relevant in host-based clusters (for example, NT clusters) where RTR must allow time for a new host to mount and export the disks. If after a certain time the journal is still inaccessible, the partition state goes into local recovery fail. In such a situation, the system manager must complete the failover of the cluster disks manually. If this succeeds, RTR can complete the recovery process and continue processing new transactions.

Unrecognized Clusters

The current version of RTR does not recognize the cluster systems available for Sun Solaris, NT, HP-UX or IBM AIX.

Failover

When the active server goes down, RTR fails over to the standby server. RTR treats unrecognized cluster systems as independent non-clustered nodes. In this case, RTR scans for the failed node's journal among the valid disks accessible to it. However if it does not find it, it does not wait for it, as with recognized clusters. Instead, it changes to the active state and continues processing new transactions.

Transaction Recovery

As in the case of recognized clusters, whenever a failover occurs, RTR attempts to recover all the in-doubt transactions from the failed node’s journal and replay them to the new active node. If the failover of the cluster disks happens fast enough so that when the new active node does the journal scan, the journal is visible, RTR will recover any in-doubt transactions from the failed node’s journal. However, if the cluster disk failover has not yet happened, RTR does not wait. RTR changes the standby server to the active state and continues processing new transactions. Note that this does not mean that the transactions in the failed node’s journal have been lost, as they will remain in the journal and can be recovered. See the RTR System Manager’s Manual for information on how to do this.

Clustered Resource Managers and Databases

When RTR standby servers work in conjunction with clustered resource managers such as Oracle RDB or Oracle Parallel Server, additional considerations apply. These affect mainly the performance of the application and are relevant only to those cluster systems that do not have "true" cluster file systems. OpenVMS and Tru64 UNIX V5.0 both have "true" cluster file systems.

Other cluster file systems host their file systems on one node and export the file system to the remaining nodes. In such a scenario, the RTR server could be working with a resource manager on one node that has its disks hosted on another node; not an optimal situation. Ideally, disks should be local to the RTR server that is active. Since RTR only waits for the journals to become available, this is not synchronized with the failover of the Resource Manager’s disks. An even worse scenario occurs if both the RTR journal and the database disks are hosted on a remote node. In this case, the use of failover scripts is recommended to assist switching over in the most optimal way. Subsequent sections discuss this in more detail.

Failure Scenarios with RTR Standby Servers

In this section the various failure situations are analyzed in more detail. This can help system managers to configure the system in an optimal way.

A transactional shadow server handles the same transactions as the primary server, and maintains an identical copy of the database on the shadow. Both the primary and the shadow server receive every transaction for their key range or partition. If the primary server fails, the shadow server continues to operate and completes the transaction. This helps to protect transactions against site failure. For greater reliability, a shadow server can have one or more standby servers. Figure 2-4 shows two primary servers, A and B, and their shadow servers, A^s and B^s.

Tolerating Site Disaster

Quorum is used by RTR to ensure facility consistency and deal with potential network partitioning. A facility achieves quorum if the right number of routers and backends in a facility (referred to in RTR as the quorum threshold), usually a majority, are active and connected.

RTR computes a quorum threshold based on the distributed view of connected roles. The minimum value can be two. Thus a minimum of one router and one backend is required to achieve quorum. If the computed value of quorum is less than two, quorum cannot be achieved. In exceptional circumstances, the system manager can reset the quorum threshold below its computed value to continue operations, even when only a minimum number of nodes, less than a majority, is available. Note, however, that RTR uses other heuristics, not based on simple computation of available roles, to determine quorum viability. For instance, if a missing (but configured) backend’s journal is accessible, that journal is used to count for the missing backend.

A facility without quorum cannot complete transactions. Only a facility that has quorum, whose nodes/roles are quorate, can complete transactions. A node/role that becomes inquorate cannot participate in transactions.

Your facility definition also has an impact on the quorum negotiation undertaken for each transaction. To ensure that your configuration can survive a variety of failure scenarios (for example, loss of one or several nodes), you may need to define a node that does not process transactions. The sole use of this node in your RTR facility is to make quorum negotiation possible, even when you are left with only two nodes in your configuration. This quorum node prevents a network partition from occurring, which could cause major update synchronization problems.

Note that executing the CREATE FACILITY command or its programmatic equivalent does not immediately establish all links in the facility, which can take some time and depends on your physical configuration. Therefore, do not use a design that relies on all links being established immediately.

Quorum is used to:

• Detect inconsistencies in how a facility has been defined on the routers and backends of the facility. RTR checks that the facility definition on its nodes is consistent and will disallow operations if it discovers inconsistencies.
• Ensure that frontends route their transactions through a router that is properly connected to all running backends.
• Ensure that shadow servers do not operate independently if a network partition occurs. This could cause databases connected to these servers to diverge.

Surviving on Two Nodes

If your configuration is reduced to two server nodes out of a larger population, or if you are limited to two servers only, you may need to make some adjustments in how to manage quorum to ensure that transactions are processed. Use a quorum node as a tie breaker to ensure achieving quorum.

For example, with a five-node configuration (Figure 2-6) in which one node acts as a quorum node, processing still continues even if one entire site fails (only two nodes left). When an RTR configuration is reduced to two nodes, the system manager can manually override the calculated quorum threshold. For details on this practice, see the Reliable Transaction Router System Manager’s Manual.

In some applications that use RTR with shadowing, the implications of transaction serialization need to be considered.

Given a series of transactions, numbered 1 through 6, where odd- numbered transactions are processed on Frontend A (FE A) and even- numbered transactions are processed on Frontend B (FE B), RTR ensures that transactions are passed to the database engine on the shadow in the same order as presented to the primary. This is serialization. For example, the following table represents the processing order of transactions on the frontends.

The order in which transactions are committed on the backends, however, may not be the same as their initial presentation. For example, the order in which transactions are committed on the primary server may be 2,1,4,3,5,6, as shown in the following table.

The secondary shadowed database needs to commit these transactions in the same order. RTR ensures that this happens, transparently to the application.

However, if the application cannot take advantage of partitioning, there can be situations where delays occur while the application waits, say, for transaction 2 to be committed on the secondary. The best way to minimize this type of serialization delay is to use a partitioned database. However, because transaction serialization is not guaranteed across partitions, to achieve strict serialization where every transaction accepts in the same order on the primary and on the shadow, the application must use a single partition.

Not every application requires strict serialization, but some do. For example, if you are moving $20, say, from your savings to your checking account before withdrawing $20 from your checking account, you will want to be sure that the $20 is first moved from savings to checking before making your withdrawal. Otherwise you will not be able to complete the withdrawal, or perhaps, depending upon the policies of your bank, you may get a surcharge for withdrawing beyond your account balance. Or a banking application may require that checks drawn be debited first on a given day, before deposits. These represent dependent transactions, where you design the application to execute the transactions in a specific order.

If your application deals only with independent transactions, however, serialization will probably not be important. For example, an application that tracks payment of bills for a company would consider that the bill for each customer is independent of the bill for any other customer. The bill-tracking application could record bill payments received in any order. These would be independent transactions. An application that can ignore serialization will be simpler than one that must include logic to handle serialization and make corrections to transactions after a server failure.

In addition to dependent transactions that can make serialization more complex, if the application uses batch processing or concurrent servers, there may be difficulties with ensuring strict serialization, if the application requires it.

In a transactional shadow configuration using the same facility, the same partition, and the same key-range, RTR ensures that data in both databases are correctly serialized, provided that the application follows a few rules. For a description of these rules, see the Application Implementation chapter, later in this document. The shadow application runs on the backends, processes transactions based on the business and database logic required, and hands off transactions to the database engine that updates the database. The application can take advantage of multiple CPUs on the backends.

Transaction Serialization Detail

Transactions are serialized by accept committing them in chronological order within a partition. Do not share data records between partitions because they cannot be serialized correctly on the shadow site.

Dependent transactions operate on the same record and must be executed in the same order on the primary and the secondary servers. Independent transactions do not update the same data records and can be processed in any order.

RTR relies on database locking during its accept phase to determine if transactions executing on concurrent servers within a partition are dependent. A server that holds a lock on a data record during its vote call (AcceptTransaction for the C++ API or rtr_accept_tx for the C API) blocks other servers from updating the same record. Therefore only independent transactions can vote at the same time.

RTR tracks time in cycles using windows; a vote window is the time between the close of one commit cycle and the start of the next commit cycle.

RTR Commit Group

RTR commit grouping enables independent transactions to be scheduled together on the shadow secondary. A group of transactions that execute an AcceptTransaction (or rtr_accept_tx call for the C API) call within a vote window form an RTR commit group, identified by a unique commit sequence number (CSN). For example, given a router (TR), backend (BE), and database (DB), each transaction sent by the backend to the database server is represented by a vote. When the database receives each vote, it locks the database and responds as voted. The backend responds to the router in a time interval called the vote window, during which all votes that have locked the database receive the same commit sequence number. This is illustrated in Figure 2-7.

Figure 2-7: Commit Sequence Number

To improve performance on the secondary server, RTR lets this commit group of transactions execute in any order on the secondary.

RTR reuses the current CSN if it determines that the current transaction is independent of previous transactions. This way, transactions can be sent to the shadow in a bunch.

In a little more detail, RTR assumes that transactions within the vote window are independent. For example, given a router and a backend processing transactions, as shown in Figure 2-8, transactions processed between execution of AcceptTransaction and the final Receive that occurs after the SQL commit or rollback will have the same commit sequence number.

Figure 2-9 illustrates the vote window for the C API. Transactions processed between execution of the rtr_accept_tx call and the final rtr_receive_message call that occurs after the SQL commit or rollback will have the same commit sequence number.

Not all database managers require locking before the SQL commit operation. For example, some insert calls create a record only during the commit operation. For such calls, the application must ensure that the table or some token is locked so that other transactions are not incorrectly placed by RTR in the same commit group.

All database systems do locking at some level, at the database, file, page, record, field, or token level, depending on the database software. The application designer must determine the capabilities of whatever database software the application will interface with, and consider these in developing the application. For full use of RTR, the database your application works with must, at minimum, be capable of being locked at the record level.

                Independent Transactions

When a transaction is specified as being independent (using the SetIndependentTransaction parameter set to true in the AcceptTransaction method (C++ API) or with the INDEPENDENT flag (C API)), the current commit sequence number is assigned to the independent transaction. Thus the transaction can be scheduled simultaneously with other transactions having the same CSN, but only after all transactions with lower CSNs have been processed.

RTR tracks time in cycles using windows; a vote window is the time between the close of one commit cycle and the start of the next commit cycle. For example, independent transactions include transactions such as zero-hour ledger posting (posting of interest on all accounts at midnight), and selling bets (assuming that the order in which bets are received has no bearing on their value).

RTR examines the vote sequence of transactions executing on the primary server, and determines dependencies between these transactions. The assumption is: if two or more transactions vote within a vote window, these transactions could be processed in any order and still produce the same result in the database. Such a group of transactions is considered independent of other transaction groups. Such groups of transactions that are mutually independent may still be dependent on an earlier group of independent transactions.

RTR tracks these groups through CSN ordering. A transaction belonging to a group with a higher CSN is considered to be dependent on all transactions in a group with a lower CSN. Because RTR infers CSNs based on run-time behavior of servers, there is scope for improvement if the application can provide hints regarding actual dependence. If the application knows that the order in which a transaction is committed within a range of other transactions is not significant, then using independent transactions is recommended. If an application does not use independent transactions, RTR determines the CSN grouping based on its observation of the timing of the vote.

The goal for this environment is to be able to survive a "double hit," without any loss of performance. While A is down, there is a window during which there is a single point of failure in the system. To meet this need, a standby server can be launched on machine B as a new P1, and the transactions being journaled in [P1] on C can be played across to Site 1. This can be done without any downtime, and P1 on C can continue to accept new transactions. When the playing across is finished, recovery is complete because all new transactions will be sent to both [P1] on C and P1 on B.

In more detail, the following sequence of events occurs:

1. Node A fails with the P1.
2. A standby server on B is started and takes over P1 for A.
3. Node C assumes the primary role for P1 and starts remembering transactions.
4. RTR starts its local recovery processing. To do so, it will try to access any nodes (defined as backend nodes in the RTR configuration) in its own cluster to locate journals that may have recovery information on them. Because A and B are not in the same cluster, it does not look for A’s journal.
5. After completing local recovery processing (with zero transactions found in its own journal) it proceeds to do shadow catch-up recovery. For this it seeks a backend node outside its own cluster (that is, any of A, C or D will be suitable candidates) and checks whether that journal has any remembered transactions for this partition. Only node C will respond positively to this search. Node B will then proceed to do shadow recovery from node C’s journal.

The fact that node A is not accessible does not prevent B from being able to shadow P1 on node C. In this configuration, the absence of node A is unlikely to cause a quorum outage.

Journal Accessibility

The RTR journal on each node must be accessible to be used to replay transactions. When setting up your system, consider both journal sizing and how to deal with replay anomalies.

To size a journal, use the following rough estimates as guidelines:

Use of large transactions generally causes poor performance, not only for initial processing and recording in the database, but also during recovery. Large transactions fill up the RTR journals more quickly than small ones.

Replay Anomalies

You should also consider how the application is to handle secondary or shadow server errors and aborts, and write your application accordingly.

In designing for performance, take the following into account:

An important part of your application design will concern performance considerations: how will your application perform when it is running with RTR on your systems and network? Providing a methodology for evaluating the performance of your network and systems is beyond the scope of this document. However, to assist your understanding of the impact of running RTR on your systems and network, this section provides information on two major performance parameters:

• Number of client channels for the C API or transaction controllers (client side) and partitions (server side) for the C++ API.
• Size of messages (in bytes)

This information is roughly scalable to other CPUs and networks. The material is based on empirical tests run on a relatively busy Ethernet network operating at 700 to 800 Kbps (kilobytes per second). This baseline for the network was based on FTP tests (doing file transfer using a File Transfer Protocol tool) because in a given configuration, network bandwidth is often a limiting factor in performance. For a typical CPU (for example, a Compaq AlphaServer 4100 5/466 4 MB) opening 80 to 100 channels with a small (100 byte) message size, a TPS (transactions per second) rate of 1400 to 1600 is usual.

Tests were performed using simple application programs (TPSREQ - client and TPSSRV - server) that use RTR Version 3 C API application programming interface calls to generate and accept transactions. (TPSREQ and TPSSRV are supplied on the RTR software kit.) The transactions consisted of a single message from client to server. The tests were conducted on OpenVMS Version 7.1 running on AlphaServer 4100 5/466 4 MB machines. Two hardware configurations were used:

• A single node, both client and server running on the same machine
• Two nodes, one configured as a frontend, the other as a router and backend

In each configuration, transactions per second (TPS) and CPU-load (CPU%) consumed created by the application (app-cpu) and the RTR ACP process (acp-cpu) were measured as a function of the:

• Number of client channels opened by the TPSREQ test program
• Size of the message sent

The transactions used in these tests were regular read/write transactions; there was no use of optimizations such as READONLY or ACCEPT_FORGET. The results for a single node with an incrementing number of channels are shown in Figure 2-11.

This test using 100-byte messages suggests the following:

• CPU saturation limited the maximum TPS at about 2500.
• CPU resource cost per transaction goes down rapidly as offered load increases (probably due to more effective use of RTR optimizations to ‘batch’ I/Os for disk and interprocess communication (IPC) as more transactions are being processed concurrently).
• In an SMP environment, the RTRACP will likely limit the maximum TPS per system to about 3000, regardless of the number of CPUs added.

The results for a single node with a changing message size are shown in Figure 2-12.

This test using 80 client and server channels suggests that:

• CPU saturation appears to limit TPS for small message sizes.
• Disk I/O rates appear to limit TPS for large messages.

The results for the two-node configuration are shown in Figure 2-13.

This two-node test using 100-byte messages provides CPU usage with totals for frontend and backend combined (hence a maximum of 200 percent). This test suggests that the constraint in this case appears to be network bandwidth. The TPS rate flattens out at a network traffic level consistent with that measured on the same LAN by other independent tests (for example, using FTP to transfer data across the same network links).

Summary

Determining the factors that limit performance in a particular configuration can be complex. While the previous performance data can be used as a rough guide to what can be achieved in particular configurations, they should be applied with caution. Performance will certainly vary depending on the capabilities of the hardware, operating system, and RTR version in use, as well as the work performed by the user application (the above tests employ a dummy application which does no real end-user work.)

In general, performance in a particular case is constrained by contention for a required resource. Typical resource constraints are:

• CPU saturation
• Disk storage I/O bandwidth and latency
• Network bandwidth and delays
• Server application I/O delays
• Database tuning
• Optimum database connection bandwidth
• Size of messages
• Number of transaction controllers or channels

Additionally, achieving a high TPS rate can be limited by:

• Lack of applied client load

For suggestions on examining your RTR environment for performance, see Appendix F in this document, Evaluating Application Resource Requirements.

Concurrent Servers

Use concurrent servers in database applications to optimize performance and continue processing when a concurrent server fails.

When programming for concurrency, you must ensure that the multiple threads are properly synchronized so that the program is thread-safe and provides a useful degree of concurrency without ever deadlocking. Always check to ensure that interfaces are thread-safe. If it is not explicitly stated that a method is thread-safe, you should assume that the routine or method is not thread-safe. For example, to send RTR messages in a different thread, make sure that the methods for sending to server, replying to client and broadcasting events are safe. You can use these methods provided that the:

• Sending thread owns the object being sent.
• Transaction controller has been completely constructed before any other threads use it.
• Transaction controller is not destructed before other threads have stopped using it.

Partitions and Performance

Partitioning data enables the application to balance traffic to different parts of the database on different disk drives. This achieves parallelism and provides better throughput than using a single partition. Using partitions may also enable your application to survive single-drive failure in a multi-drive environment more gracefully. Transactions for the failed drive are logged by RTR while other drives continue to record data.

Facilities and Performance

To achieve performance goals, you should establish facilities spread across the nodes in your physical configuration using the most powerful nodes for your backends that will have the most traffic.

In some applications with several different types of transactions, you may need to ensure that certain transactions go only to certain nodes. For example, a common type of transaction is for a client application to receive a stock sale transaction, which then proceeds through the router to the current server application. The server may then respond with a broadcast transaction to only certain client applications. This exchange of messages between frontends and backends and back again can be dictated by your facility definition of frontends, routers, and backends.

Router Placement

Placement of routers can have a significant effect on your system performance. With connectivity over a wide-area network possible, do not place your routers far from your backends, if possible, and make the links between your routers and backends as high speed as possible. However, recognize that site failover may send transactions across slower-speed links. For example, Figure 2-14 shows high-speed links to local backends, but lower-speed links that will come into use for failover.

Additionally, placing routers on separate nodes from backends provides better failover capabilities than placing them on the same node as the backend.

In some configurations, you may decide to use a dual-rail or multihome setup for a firewall or to improve network-related performance. (See Appendix G for information on this setup.)

When a server or client application sends out a broadcast message, the message passes through the router and is sent to the client or server application as appropriate. A client application sending a broadcast message to a small number of server applications will probably have little impact on performance, but a server application sending a broadcast message to many, potentially hundreds of clients, can have a significant impact. Therefore, consider the impact of frequent use of large messages broadcast to many destinations. If your application requires use of frequent broadcasts, place them in messages as small as possible. Broadcasts could be used, for example, to inform all clients of a change in the database that affects all clients.

Figure 2-15 illustrates message fan-out from client to server, and from server to client.

You can also improve performance by creating separate facilities for sending broadcasts.

Making Broadcasts Reliable

To help ensure that broadcasts are received at every intended destination, the application might number them with an incrementing sequence number and have the receiving application check that all numbers are received. When a message is missing, have a retransmit server re-send the message.

Very large configurations with unstable or slow network links can reduce performance significantly. In addition to ensuring that your network links are the fastest you can afford and put in place, examine the volume of inter-node traffic created by other uses and applications. RTR need not be isolated from other network and application traffic, but can be slowed down by them.

Using Read-Only Transactions

Read-only transactions can significantly improve throughput because they do not need to be journaled. A read-only database can sometimes be updated only periodically, for example, once a week rather than continuously, which again can reduce application and network traffic.

When using transactional shadowing, it can enhance performance to process certain transactions as independent. When transactions are declared as independent, processing on the shadow server proceeds without enforced serialization. Your application analysis must establish what transactions can be considered independent, and you must then write your application accordingly. For example, bets placed at a racetrack for a specific race are typically independent of each other. In another example, transactions within one customer's bank account are typically independent of transactions within another customer’s account.

For examples of code snippets for each RTR API, see the appendices of samples in this manual.

To help make your RTR system as manageable and operable as possible, consider several tradeoffs in establishing your RTR configuration. Review these tradeoffs before creating your RTR facilities and deploying an application. Make these considerations part of your design and validation process.

• Define your facilities with an eye to the number and placement of frontends, routers, and backends.
• To avoid problems with quorum resolution, design your configuration with an odd number of routers to ensure that quorum can be achieved.
• Separate your routers from your backends to improve failover, so that failure of one node does not take out both the router and the backend.
• If your application requires frontend failover when a router fails, frontends must be located on separate nodes from the routers, but frontends and routers must of course be in the same facility. For frontend failover, a frontend must be in a facility with multiple routers. You use frontend failover with nested transactions.
• To identify a node used only for quorum resolution, define the node as a router or as a router and frontend. Define all backends in the facility, but no other frontends.
• With a widely dispersed set of nodes, for example, nodes distributed across an entire country, use local routers to deal with local front ends. This can be more efficient than having many dispersed frontends connecting to a small number of distant routers.
• In many configurations, it may be more effective to place routers near backends.

Firewalls and RTR

For security purposes, your application transactions may need to pass through firewalls in the path from the client to the server application. RTR provides this capability within the CREATE FACILITY syntax. See the RTR System Manager's Manual, Network Transports, for specifics on how to specify a node to be used as a firewall, and how to set up your application tunnel through the firewall.

Avoiding DNS Server Failures

Nodes in your configuration are often specified with names and IP or DECnet addresses fielded by a name server. When the name server goes down or becomes unavailable, the name service is not available and certain requests may fail. To minimize such outages, declare the referenced node name entries in a local host names file that is available even when the name server is not. Using a host names file can also improve performance for name lookups.

Batch Procedures

Operations staff often create batch or command procedures to take snapshots of system status to assist in monitoring applications. The character cell displays (ASCII output) of RTR can provide input to such procedures. Be aware that system responses from RTR can change with each release, which can cause such command procedures to fail. If possible, plan for such changes when bringing up new versions of the product.

In addition to understanding the RTR run-time and system management environments, you must also understand the RTR applications environment and the implications of that environment on your implementation. This section provides information on requirements that transaction processing applications must take into account and deal with effectively. It also cites rules to follow that can help prevent your application from violating the rules for ensuring that your transactions are ACID compliant. The requirements and rules complement each other and sometimes repeat a similar concept. Your application must take both into account.

Applications written to operate in the RTR environment should adhere to the following rules:

• Be transaction aware
• Avoid server-specific data
• Optionally, have independent transactions
• Optionally, use two identical databases for transactional shadow servers
• Make transactions self-contained
• Lock shared resources

Be Transaction Aware

RTR expects server applications to be transaction aware; an application must be able to roll back an appropriate amount of work when asked. Furthermore, to preserve transaction integrity, rollback must be all or nothing. Each transaction incurs some overhead, and the application must be prepared to deal with failures and concomitant rollback gracefully.

When designing your client and server applications, note the outcome of transactions. Transactional applications often store data in variables that pertain to the operation taking place outside the control of RTR. Depending on the outcome of the RTR transaction, the values of these variables may need to be adjusted. RTR guarantees delivery of messages (usually to a database), but RTR does not know about any data not passed through RTR.

The rule is: Code your application to preserve transaction integrity through failures.

Avoid Server-Specific Data

The client and server applications must not exchange any data that makes sense on only one node in the configuration. Such data can include, for example, a memory reference pointer, whose purpose is to allow the client to reference this context in a later transaction, indexes into files, node names, or database record numbers. These values only make sense on the machine on which they were generated. If your application sends data to another machine, that machine will not be able to interpret the data correctly. Furthermore, data cannot be shared across servers, transaction controllers, or channels.

The rule is: How you track state must be meaningful on all nodes where your application runs.

Be Independent of Time of Processing

Transactions are assumed to contain all the context information required to be successfully executed. An RTR transaction is assumed to be independent of time of processing. For example, in a shadow environment, if the secondary server cannot credit an account because it is past midnight, but the transaction has already been successfully committed on the primary server, this would cause an inconsistency between the primary and secondary databases. Or, in another example, Transaction B cannot rely on the fact that Transaction A performed some operation before it.

Make no assumptions about the amount of time that will occur between transactions, and avoid using a transaction to establish a session with a server application that can time out. Such a timeout might occur in a client application that logs onto a server application that sets a timer to determine when to log the client off. If a crash occurs after a successful logon, subsequent transactions may fail because the logon session is no longer valid.

The rule is: If you have operations that must not be shadowed, identify them and exclude them from your application. Furthermore, do not keep a state that can become stale over time.

In your application, you can define transactions as independent with the C++ API, using the SetIndependentTransaction method in your transaction controller AcceptTransaction or SendApplicationMessage calls. Using the C API, you use the independent transaction flag in your rtr_accept_tx or rtr_reply_to_client calls.

For more information on the independent transaction methods in the RTRServerTransactionController class, see the RTR C++ Foundation Classes manual. For more information on the independent transaction flag and the different uses of these calls, see the RTR C Application Programmer’s Reference Manual.

Use Two Identical Databases for Shadow Servers

Shadow server use is aimed at keeping two identical copies of the database synchronized. For example, Figure 3-1 illustrates a configuration with a router serving two backends to two shadow databases. The second router is for router failover.

Figure 3-1: Transactional Shadow Servers

Figure 3-2: Shadow Servers and Third Common Database (not recommended)

While a server application is processing a transaction, and particularly before it "accepts" the transaction, it must ensure that all shared resources accessed by that transaction are locked. Failure to do so can cause unpredictable results in shadowing or recovery.

The rule is: Lock shared resources while processing each transaction.

Ensuring ACID Compliance

To ensure that your application deals with transactions correctly, its transactions must be:

• Atomic
• Consistent
• Isolated
• Durable

Ensuring Atomicity

A transaction either creates a new and valid state of data, or, if any failure occurs, returns all data to its state as it was before the start of the transaction. This is called consistency.

Several rules must be considered to ensure consistency:

• Shadowed applications must lock records being updated and hold these locks while RTR performs commit processing. Note that shadowed transactions can be executed at different times and in a different order between two sites because RTR assumes that transactions are independent.
• Client applications that rely on system-dependent information sent from the server application should use the REPLYDIFF option in RTR to ensure that a single transaction does not span multiple nodes. For example, if data are returned to the client application before the server crashes, the recovered transaction may produce results that conflict with the message previously returned to the client.
• Applications should never access shadowed data sets outside the RTR environment. This could, for example, be the case if your application relies on batch processing that does not use RTR or database maintenance. Updating between the two sites will not be serialized if all updates do not pass through RTR. Updating two sites only after quiescing shadow systems is impractical in most mission-critical applications.
• Partition your data so that all access is done through a single RTR partition (key range). It is unusual to partition by function or use shared tables. Although partitioning is recommended for improving concurrency, RTR serializes transactions only on a partition basis. If two partitions have overlapping data records, these records might be updated in differing order across the two sites, unless the application deals with them correctly. Do not overlap partitions; make them discrete.
• Perform all data-constraint checking before committing the database. This can be an issue with certain database optimizations.

Ensuring Isolation

The changes to shared resources that a transaction causes do not become visible outside the transaction until the transaction commits. This makes transactions serializable. To ensure isolation:

• Hold record locks throughout the RTR commit cycle. If the server crashes, the transaction could be recovered after changes to the data by a dependent transaction, generating results that are different from those already sent to the client. Also, new transactions can overtake the completion of a previous transaction from the same client if shared records are not locked.
• Do not use RTR concurrent servers if your data manager does not support record locking. This can be important, for example, with in-memory databases. Concurrency relies on the independence of two operations that may affect common data records. Record locking ensures that a concurrent transaction cannot affect the consistency of data being operated on by another transaction.
• Avoid certain site-dependent actions when running RTR shadow servers. For example, using transaction sequence numbers or time and date comparisons, can introduce problems. Shadowed transactions are serialized based on commit groups. If your application requires absolute transaction serialization, you cannot run concurrent servers. For example, Figure 3-4 illustrates the serialization of commit groups. The first commit group is A1, processed on the primary backend; it is followed by commit group A2, followed by A3. Commit to the database, however, is in the order A3, A1, A2, as shown in the diagram. On the shadow site, commit to the database will be in the order A3, A2, A1, due to use of concurrent servers. If absolute serialization of CSNs is important in your application, you cannot use concurrent servers.

RTR commit grouping allows independent transactions to be scheduled together on the shadow secondary.

Ensuring Durability

For a transaction to be durable, the changes caused by transaction commitment must survive subsequent system and media failure. Thus transactions are both persistent and stable.

For example, your bank deposit is durable if, once the transaction is complete, your account balance reflects what you have deposited.

The durability rule is:

Standby servers that update the database must have access to each other’s RTR journal, and use cluster-aware data managers such as Oracle Parallel Servers. If a node running as a standby server crashes, in-progress transactions will be recovered from the failed node’s journal files.

If there are dependencies between separate RTR transactions, these should be considered carefully because the locking mechanisms of resource managers can cause unexpected behavior. These issues around locking mechanisms occur only if there is more than one server for the same partition.

For example, consider the case where there is a transaction T1 which inserts a record in the database and a subsequent transaction T2 which uses that record to make another update. If the partition has been configured with concurrent servers, it can happen that the update transaction T2 which has been given to a free server will begin executing and reach the database before the insert operation issued by transaction T1 has completed the commit process. In this scenario, the inserted record is not yet visible to the update transaction T2 because the commit is not yet complete. This will cause transaction T2 to fail. However, if the database table being updated is locked for the duration of the insert, transaction T2 will block (wait) until the insert has committed and there will be no possibility of transaction T2 overtaking transaction T1.

In another example, the first transaction T1 makes an update to the table and a second transaction T2 uses the updated value in its transaction. If the resource manager does not lock the row being accessed by transaction T1 right at the start of the update, that row can be queried by the second transaction T2 which has started on a concurrent server. However, transaction T2 will in this case be working with the old and not the updated value that was the result of T1. To prevent such unexpected and potentially undesirable behavior, check the locking mechanisms of the resource managers being used before using concurrent servers.

RTR provides client applications the option to specify a transaction timeout, but has no provision for server applications to specify a timeout on transaction duration. If there is a scarcity of server application processes, all other client transactions remain queued. If these transactions have also specified timeouts, they are aborted by RTR (assuming that the timeout value is less than 2 minutes).

To avoid this problem, the application designer has two choices:

• Use concurrent server processes
• Have the application administer its own timeout

The first (and easier) option is to use concurrent server processes. This allows transaction requests to be serviced by other free servers, even if one server is occupied by such a transaction that is taking a long time to disappear. The second option is to design the server application so that it can abort the transaction independently.

There are three cases where this use of concurrent servers is not ideal. First, there is an implicit assumption about how many such lingering transactions might remain on the system. In the worst case, this could exceed or equal the number of client processes. But having so many concurrent server processes to cater to this contingency is wasteful of system resources. Second, use of concurrent servers is beneficial when the servers do not need to access a common resource. For instance, if all these servers needed to update the same record in the database, they would simply be waiting on a lock taken by the first server. Additional servers do not resolve this issue. Third, it must make business sense to have additional servers. For example, if transactions must be executed in the exact order in which they entered the system, concurrent servers may introduce sequencing problems.

Take the example of the order matcher in a stock trading application. Business rules may dictate that orders be matched on a first-come, first-matched basis; using concurrent servers would negate this rule.

The RTR frontend sees several transaction states during accept processing. Table 3-2 lists frontend transaction states that represent the steps in the prepare phase.