Reliable Transaction Router
Application Design Guide


Previous Contents Index

The information in this table comes from the Oracle 8i Application Developer's Guide.

Distributed Deadlocks

A deadlock or deadly embrace can occur when transactions lock data items in a database. The typical scenario is with two transactions txn1 and txn2 executing concurrently with the following sequence of events:

  1. txn1 write-locks data item A.
  2. txn2 write-locks data item B.
  3. txn1 requests a lock on data item B but must wait because txn2 still has a lock on data item B.
  4. txn2 requests a lock on data item A but must wait because txn1 still has a lock on data item A.

Neither txn1 nor txn2 can proceed; they are in a deadly embrace. Figure 3-7 illustrates a deadly embrace.

Figure 3-7 Deadly Embrace


With RTR, to avoid such deadlocks, follow these guidelines:

  1. Always engage servers in the same order, and wait for the reply before each send.
  2. Provide several concurrent servers to minimize contention. Estimate the number of concurrent servers needed by determining the volume of transactions the servers must support, considering periods of maximum activity, and allowing for growth. The larger the volume on your servers, the more likely it is that your application will benefit from using concurrent servers.

RTR attempts to resolve deadlocks by aborting one deadlocked transaction path with error code RTR_STS_DLKTXRES and replaying the transaction. Other paths are not affected. Server applications should be written to handle this status appropriately.

The RTR status code RTR_STS_DLKTXRES can occur under several environmental conditions that RTR detects and works around. The application need not take any explicit action other than releasing the resources connected with the active transaction such as doing a rollback on the database transaction.

For example, RTR may issue an RTR_STS_DLKTXRES status when:

As an example of the first case, consider clients A and B both performing transactions TCA and TCB, where both TCA and TCB include a message to both server X and server Y followed by an ACCEPT. There is only one instance of Server X and Server Y available, and due to the quirks of distributed processing, only Server X receives the message belonging to TCA and only Server Y receives the message belonging to TCB. Figure 3-8 reflects this scenario. Because Server Y has no chance of accepting TCA until TCB is processed to completion and Server X has no chance of accepting TCB until TCA is processed to completion, Server X and Y are in a distributed deadlock. In such a case, RTR selects TCA or TCB to abort with DLKTXRES and replays it in a different order.

Figure 3-8 Scenario for Distributed Deadlock


Sometimes RTR needs to abort a transaction and reschedule it. For example, it can happen that a state change is needed after the primary server started to process a transaction but RTR had to change its role to secondary before the transaction was completed. Thus the transaction would be executed on the other node as primary and later played to this server as secondary. RTR uses the same status code RTR_STS_DLKTXRES when aborting the transaction.

Providing Parallel Processing

One method for improving response time is to send multiple messages from clients without waiting for a reply. The messages can be sent to different partitions to provide parallel processing of transactions.

Establishing Read-Only Sites

For certain read-only applications, RTR can be used without shadowing to establish sites to protect against site failure. The method is to define multiple non-overlapping facilities with the same facility name across a network that is geographically dispersed. In the facility, define a failover list of routers, for example, some in one city, some in another. Then when the local router fails, a client is automatically reconnected to another node. If all local nodes in the facility are unavailable, the client is automatically connected to a node at the alternate site.

Another method is to define a partition on a standby server for read-only transactions. This minimizes network traffic to the standby. A read-only partition on a standby server can reduce node-to-node transaction locking.

Resolving Idempotency Issues

Generally, databases (and applications built to work with them) are required to be idempotent. That is, given a specific state of the database, the same transaction applied many times would always produce the same result. Because RTR relies on replays and rollbacks, if there is a server failure before a transaction is committed, RTR assumes the database will automatically roll back to the previous state, and the replayed transaction will produce results identical to the previous presentation of the transaction. RTR assumes that the database manager and server application provide idempotency.

For example, consider an internet transaction where you log into your bank account and transfer money from one account to another, perhaps from savings to checking. If you interrupt the transfer, and replay it two hours later, the transfer may not succeed because it would be required to have been done within a certain time interval after the login. Such a transaction is not idempotent.

Designing for a Heterogenous Environment

In a heterogeneous environment, you can use RTR with several hardware architectures, both little endian and big endian. RTR does data marshalling in your application so that you can take advantage of such a mixed environment.

If you are constructing an application for a heterogeneous environment:

Using the Multivendor Environment

With RTR, applications can run on systems from more than one vendor. You can mix operating systems with RTR, and all supported operating systems and hardware architectures can interoperate in the RTR environment. For example, you can have some nodes in your RTR configuration running OpenVMS and others running Windows NT.

To develop your applications in a multivendor environment:

Upgrading from RTR Version 2 to RTR Versions 3 and 4

An existing application written using RTR Version 2 with OpenVMS will still operate with RTR Versions 3 and 4. See the Reliable Transaction Router Migration Guide for pointers on using RTR Version 2 applications with RTR Version 3, and moving RTR Version 2 applications to RTR Version 3.


Chapter 4
Design with the C++ API

This chapter provides information on RTR transaction model and recovery concepts for client and server applications implemented with the C++ API. Topics include :

Additional information on RTR transactions and recovery can be found in the Application Implementation chapter of this guide and in RTR Getting Started .

Transactional Messaging with the C++ API

Figure 4-1 illustrates frontend/backend interaction with pseudo-code for transactions and shows transaction brackets. The transaction brackets show the steps in completing all parts of a transaction, working from left to right and top to bottom. In the figure, TC stands for transaction controller.

Figure 4-1 Transactional Messaging with the C++ API


The transaction is initiated at "Start transaction" on the frontend, and completed after the "Commit transaction" step on the backend. The transaction ID is encoded to ensure its uniqueness throughout the entire distributed system. In the prepare phase on the server, the application should lock the relevant database (DB) records. The commit of a transaction hardens the commit to the database. Figure 4-2 illustrates a typical call sequence between a client and server application. These calls are RTRClientTransactionController and RTRServerTransactionController class methods. The first call in both the client and server transaction controllers is to create a new transaction controller, for example, in the server, use RTRServerTransactionController::RTRServerTransactionController.

Figure 4-2 C++ API Calls for Transactional Messaging


For a client, an application typically uses the following methods in this order:

Client first creates a transaction controller and a facility.

The client sends a request to the server.

After the server has processed the request, the client calls Receive and the data object contains the RTR message rtr_mt_reply , causing the client message handler OnInitialize and OnApplicationMessage methods to be called.

The client calls AcceptTransaction, if all went well.

For a server, an application typically uses the following methods in this order:

On the first server transaction controller Receive, RTRData contains rtr_mt_msg1 . With event-driven processing (the default behavior) the server message handler calls OnInitialize and then calls OnApplicationMessage.

On a second Receive from a client SendApplicationMessage, the RTR message received in the data object contains rtr_mt_msgn , causing OnApplicationMessage to be called by the server message handler.

After processing the client's request, the server calls SendApplicationMessage.

When the client accepts the transaction, the server Receive call includes rtr_mt_prepare from RTR.

The server accepts the transaction.

RTR sends rtr_mt_accepted

With event-driven processing, the appropriate methods in the RTRClientMessageHandler and RTRServerMessageHandler classes are called by RTRData::Dispatch. You must use the RTRClientTransactionController::RegisterHandlers and RTRServerTransactionController::RegisterHandlers methods to enable this default handling.

Data-Content Routing with the C++ API

Data-content routing is the capability provided by RTR to support the partitioned data model. With the C++ API, you define partitions with the RTRPartitionManager class. Partitions are defined with partition name, facility name and KeySegment attributes. Using RTR data-content routing, message content determines message destination. The routing key, defined in the C++ API RTRKeySegment class is embedded within the application message. When a server is started, the server informs RTR that it serves a particular partition.

When a client sends a message that references that particular partition, RTR knows where to find a server for that partition and routes the message accordingly. Even if the server is moved to another location, RTR can still find it. The client and the application do not need to worry about locating the server. This is location independence.

The benefits of data-content routing are simpler application development and flexible, scalable growth without application changes. RTR hides the underlying network complexity from the application programmer, thus simplifying development and operation.

Changing Transaction States

With Set State methods within the RTRServerTransactionProperties class, users can change a transaction state in their application servers. With the event-driven processing model, your states are shown through the message handlers. In the polling model, the states are accessed with an RTRData::GetMessageType() call.

Consider the following scenario: upon receiving an rtr_mt_accepted message indicating that a transaction branch is committed, the server normally performs an SQL commit to write all changes to the underlying database. However, the server that is to detect the SQL commit or the underlying database may be temporarily unavailable.

With the C++ API, you can change the state of a transaction to EXCEPTION using the SetStateToException method, to temporarily put this transaction in the exception queue. When things get better, users can call SetStateToCommit or SetStateToDone to change the transaction state back to COMMIT or DONE, respectively.

The following example shows how a transaction state can be changed using the set state methods. See the RTRServerTransactionProperties class documentation in the Reliable Transaction Router C++ Foundation Classes manual for more details.

RTRServerTransactionProperties::SetStateToException();
RTRServerTransactionProperties_object_name.SetStateToException(stCurrentTxnState);

The parameter stCurrentTxnState is a transaction state of type rtr_tx_jnl_state_t .

Normally at the RTR command level, users need to provide at least facility name and partition name qualifiers to help RTR select a desired set of transactions to be processed. Because a transaction's TID (rtr_tid_t) is unambiguously unique, a user needs only to specify the transaction's current state and its TID.

Note that if a transaction has multiple branches running on different partitions simultaneously on this node, RTR will reject this set transaction request with an error. RTR can only change state for one branch of a multiple partition transaction at a time.

RTR Message Types

RTR calls (and responses to them) contain RTR message types (mt) such as rtr_mt_reply or rtr_mt_rejected . There are four groups of message types:

Table 4-1 lists all RTR message types.

Table 4-1 RTR Message Types
Transactional Status Event-related Informational
rtr_mt_msg1 rtr_mt_accepted rtr_mt_user_event rtr_mt_opened
rtr_mt_msg1_uncertain rtr_mt_rejected rtr_mt_rtr_event rtr_mt_closed
rtr_mt_msgn rtr_mt_prepare   rtr_mt_request_info
rtr_mt_reply rtr_mt_prepared   rtr_mt_rettosend

Applications should include code for all RTR expected return message types. Message types are returned to the application in the message status block. For more detail on message types, see the Reliable Transaction Router C Application Programmer's Reference Manual .

Transactional Message Processing

Figure 4-3 illustrates transactional messaging interaction between a client application and a server application using C++ API calls.

Figure 4-3 Flow of a Transaction


In Figure 4-3:

  1. The first send call in the client application starts a transaction and sends a message.
  2. The server processes the transaction then sends a message back to the client.
  3. The client receives the message and either accepts or rejects the transaction. The client's vote goes to RTR, not the server application.
  4. The server then votes after receiving a message or event from RTR.

RTR sends both the server and the client a message that either declares that the transaction was accepted or rejected.

All the above steps comprise two parts of a transaction:

Message Processing Sequence

Message processing involves a message being sent by a client and a reply coming back from one or more servers. This is where the business application processing takes place.

The typical steps during message processing are:

Accept Processing

Figure 4-4 illustrates the accept-processing portion of a completed RTR transaction.

Figure 4-4 Accept Processing


The steps in accept processing are:

  1. Client sends accept message to RTR.
  2. Server receives prepare message.
  3. Server accepts or rejects transaction.
  4. Client and server receive final transaction state from RTR.

Starting a Transaction

There is one way to start a transaction - explicitly by creating a server transaction controller object, registering a facility, and using StartTransaction. The SendMessage call sends a message as part of a transaction from a client. If there is no transaction currently active on the transaction controller, a new one is started. The AcceptTransaction can be bundled with the last message. The SendMessage call also sends a reply message from a server to the client. The reply message is part of the transaction initiated by the client. A transaction is defined as a group of messages initiated by the client. The server knows that a transaction has begun when it receives a message of one of the following types:

If transaction timeouts are not required, the transaction starts on the next SendMessage call.

The Register call enables the transaction to join another transaction or set a transaction timeout. When a transaction is started implicitly, the timeout feature is disabled. A client has two options for message delivery after a failure:

Identifying the Transaction

When a message is received, the message status block contains the transaction identifier (TID).

You can use the GetTID call to obtain the RTR transaction identifier for the current transaction. This identifier is a unique number generated by RTR for each transaction. The application can use the TID if the client needs to know the TID to take some action before receiving a response.

Accepting a Transaction

The AcceptTransaction call by the client begins the prepare phase of the two- phase commit protocol. An accepted transaction is not complete until a message of type rtr_mt_accepted or rtr_mt_rejected is received.

The application can specify a reason on the AcceptTransaction method so that the caller can specify an accept reason that is passed on to all participants in the transaction. If more than one transaction participant specifies a reason, the reason values are ORed together by RTR. The accept is final: the caller cannot reject the transaction later. The caller cannot send any more messages for this transaction.

A client can accept a transaction in one of two ways: with the AcceptTransaction call or by using the SetAcceptTransaction method. Using the SetAcceptTransaction method removes the need to issue an AcceptTransaction method and can help optimization of client traffic. Merging the data and accept messages in one call puts them in a single network packet. This can make better use of network resources and improve throughput.

Rejecting a Transaction

Any participant in the transaction can call RejectTransaction. The reject is final and it is impossible for the caller to accept the transaction later. The RejectTransaction method rejects a transaction. Once the transaction has been rejected, the caller receives no more messages for this transaction.

The server can call the ForceRetry method to have RTR redeliver the transaction beginning with rtr_mt_msg1 without aborting the transaction for other participants. Using the RejectTransaction method, the application can specify a reason that the caller can pass on to all participants in the transaction. If more than one transaction participant specifies a reason, the reason values are ORed together by RTR.

Ending a Transaction

A server application can end a transaction by either accepting or rejecting the transaction. RTR can reject a transaction at any time after the transaction is started but before it is committed. For example, if RTR cannot deliver a transaction to its destination, it rejects the transaction and delivers the reject completion status to all participants that know about the transaction.

A transaction is accepted explicitly with the AcceptTransaction method, and rejected explicitly with the RejectTransaction method. RTR can reject a transaction at any time once the transaction is started but before it is committed. If RTR cannot deliver a transaction to its destination, it rejects the transaction explicitly and delivers the reject completion status to all participants.

A transaction participant can specify a reason for an accept or reject on the AcceptTransaction and RejectTransaction methods. If more than one transaction participant specifies a reason, RTR uses the OR operator to combine the reason values together. For example, with two servers, A and B, each providing a reason code of 1 or 2, respectively, the client receives the result of the OR operation, reason code 3, in its message buffer in RTRData.

A transaction is done once a client or server application receives a completion message, either an rtr_mt_accepted or rtr_mt_rejected message from RTR. An application no longer receives messages related to a transaction after receiving a completion message or if the application uses RejectTransaction A client or server can also specify SetForgetTransaction to signal its acceptance and end its involvement in a transaction early. RTR returns no more messages (including completion messages) associated with the transaction; any such messages received will be returned to the caller.

A client or server application no longer receives messages related to a transaction after it receives an OnAccepted or OnRejected message from RTR, or if the application called RejectTransaction.


Previous Next Contents Index