Server and Active Transaction States in a Shadow Server

Figure C-1 shows server states after delivery of a primary or secondary event, and message types used with primary and secondary servers.

Figure C-1 Server Events and States with Active Transaction Message Types

Server and Transaction States in a Standby Server

Figure C-2 shows server states after delivery of a standby event, and message types used with transactions that are active or in recovery.

Figure C-2 Server States after Standby Events

Specifying Server Type

The application must specify server type with boolean attributes using the CreateBackEndPartition method in the RTRManager class. For example, the following declaration establishes a standby server with concurrency:

CreateBackEndPartition( *pszPartitionName, pszFacility, pKeySegment bShadow=false bConcurrent=true bStandby=true);

To add a transactional shadow server, use: bShadow = true

To disallow a standby server, use: bStandby = false

Server Failover

With the C++ API, you enable RTR failover behavior with the CreateBackEndPartition method in the RTRPartitionManager management class.

Concurrent Servers

For the C++ API, concurrent servers can be implemented as many server transaction controllers in one process or as one or many server transaction controllers in many processes.

RTR delivers transactions to any open transaction controllers, so each application thread must be ready to receive and process transactions.

An application creates a transaction controller and registers a partition with the RegisterPartition method. To specify whether or not a server is concurrent, the application uses the CreateBackendPartition method in the RTRPartitionManager class. The rules are as follows:

• Set the bConcurrent parameter to true for the server to have other concurrent servers.
• Set the bConcurrent parameter to false for the server not to be concurrent.

For example, the following declaration establishes a concurrent server that is also a standby:

CreateBackEndPartition( *pszPartitionName, pszFacility, pKeySegment bShadow=false bConcurrent=true bStandby=true);

The following declaration establishes a server with no concurrency:

CreateBackEndPartition( *pszPartitionName, pszFacility, pKeySegment bShadow=false bConcurrent=false bStandby=true);

For more information on the CreateBackEndPartition method, see the Reliable Transaction Router C++ Foundation Classes manual.

Standby Servers

RTR manages the activation of standby servers at run time.

When an application creates a server partition with the CreateBackEndPartition method in the RTRPartitionManager class, it specifies whether a server is to be standby or not as follows:

• Sets the bStandby parameter true so the server can have standby servers.
• Sets the bStandby parameter to false to specify that the server is not to be a standby nor to have standbys. For example, the following declaration establishes a concurrent server that is not a standby.

CreateBackEndPartition ( *pszPartitionName, pszFacilityName, *pKeySegment, bShadow = false, bConcurrent = true, bStandby = false);

Shadow Servers

When an application creates a server partition with the CreateBackEndPartition method in the RTRPartitionManager class, it specifies whether a server is to be a shadow or not as follows:

• Sets the bShadow parameter to false so the server is not a shadow server.
• Specifies that the server is to be a shadow by setting the bShadow parameter to true. For example:

CreateBackEndPartition ( *pszPartitionName, pszFacilityName, *pKeySegment, bShadow = true, bConcurrent = true, bStandby = false);

Only one primary and one secondary shadow server can be established. Shadow servers can have both standby and concurrent servers.

When partition state is important to an application, the application can determine if a shadow server is in the primary or secondary partition state after server restart and recovery following a server failure. The application does this using methods in the RTRServerEventHandler class such as OnServerIsPrimary, OnServerIsStandby, and OnServerIsSecondary. For example:

OnServerIsPrimary(*pRTRData, *pController);

Making Transactions Independent

Within your application server code, you identify those transactions that can be considered independent, and set the state of the transaction controller object with the bIndependent attribute in the AcceptTransaction method, as appropriate. The following example illustrates how to set the bIndependent parameter to true with the AcceptTransaction method to make a transaction independent.

RTRServerTransactionController *pController= new RTRServerTransactionController(); pController->AcceptTransaction(RTR_NO_REASON, true);

Another example:

RTRServerTransactionController stc; /* Determine from our business logic if this transaction is independent of our other transactions. */ If (true == Independent()) { stc.AcceptTransaction(RTR_NO_REASON,true) } else { stc.AcceptTransaction() }

Specifying Server Type

The application specifies the server type in the rtr_open_channel call as follows:

rtr_status_t rtr_open_channel ( . rtr_ope_flag_t

To add a transactional shadow server, include the following flags: flags = RTR_F_OPE_SERVER RTR_F_OPE_SHADOW;

To disallow concurrent and standby servers, use the following flags:

flags = RTR_F_OPE_SERVER | RTR_F_OPE_NOCONCURRENT | RTR_F_OPE_NOSTANDBY;

Server Failover

With the C API, you enable RTR failover behavior with flags set when your application executes the rtr_open_channel statement or command.

Concurrent Servers

For the C API, concurrent servers can be implemented as many channels in one process or as one or many channels in many processes. By default, a server channel is declared as concurrent.

RTR delivers transactions to any open channels, so each application thread must be ready to receive and process transactions. The main constraint in using concurrent servers is the limit of available resources on the machine where the concurrent servers run.

When an application opens a channel with the rtr_open_channel call, it specifies whether the server is to be concurrent or not, as follows:

• Does nothing (omits the flag) so the server can have other concurrent servers. This is the default.
• Uses the RTR_F_OPE_NOCONCURRENT flag to indicate that the server is not to be concurrent.

For example, the following code fragment establishes a server with concurrency:

rtr_open_channel(&Channel, RTR_F_OPE_SERVER, FACILITY_NAME, NULL, RTR_NO_PEVTNUM, NULL, Key.GetKeySize(), Key.GetKey() != RTR_STS_OK);

If an application starts up a second server for a partition on the same node, the second server is a concurrent server by default.

The following example establishes a server with no concurrency:

rtr_open_channel(&Channel, RTR_F_OPE_SERVER|RTR_F_OPE_NOCONCURRENT, FACILITY_NAME, NULL, RTR_NO_PEVTNUM, NULL, Key.GetKeySize(), Key.GetKey() != RTR_STS_OK);

When a concurrent server fails, the server application can fail over to another running concurrent server, if one exists.

Concurrent servers are useful both to improve throughput using multiple channels on a single node, and to make process failover possible. Concurrent servers can also help to minimize timeout problems in certain server applications. For more information on this topic, see the section later in this manual on Server-Side Transaction Timeouts.

For more information on the rtr_open_channel call, see the Reliable Transaction Router C Application Programmer's Reference Manual and the discussion later in this document.

Standby Servers

RTR manages the activation of standby servers at run time.

When an application opens a channel, it specifies whether or not the server is to be standby, as follows:

• Does nothing (omits the flag) so the server can have standby servers. This is the default.
• Includes the RTR_F_OPE_NOSTANDBY flag so the server is not to be a standby nor to have standbys.

Shadow Servers

When an application opens a channel, it specifies whether the server is to have the capability to be a transactional shadow server or not, as follows:

• Does nothing (omits the flag) so the server is not a shadow server. This is the default.
• Includes the RTR_F_OPE_SHADOW flag so the server is to be a shadow server.

Only one primary and one secondary shadow server can be established. Shadow servers can have both standby and concurrent servers.

When partition state is important to an application, the application can determine if a shadow server is in the primary or secondary partition state after server restart and recovery following a server failure. The application does this using RTR events in the rtr_open_channel call, specifying the events RTR_EVTNUM_SRPRIMARY and RTR_EVTNUM_SRSECONDARY . For example, the following is the usual rtr_open_channel declaration:

rtr_status_t rtr_open_channel ( rtr_channel_t *p_channel, //Channel rtr_ope_flag_t flags, //Flags rtr_facnam_t facnam, //Facility rtr_rcpnam_t rcpnam, //Name of the channel rtr_evtnum_t *p_evtnum, //Event number list //(for partition states) rtr_access_t access, //Access password rtr_numseg_t numseg, //Number of key segments rtr_keyseg_t *p_keyseg //Pointer to key-segment data )

To enable receipt of RTR events that show shadow state, used if an application needs to include logic depending on partition state, the application enables receipt of RTR events that show shadow state.

The declaration includes the events as follows:

rtr_evtnum_t evtnum = { RTR_EVTNUM_RTRDEF, RTR_EVTNUM_SRPRIMARY, RTR_EVTNUM_SRSECONDARY, RTR_EVTNUM_ENDLIST }; rtr_evtnum_t *p_evtnum = &evtnum;

Broadcasts deliver using name and subscription name. For details, see the descriptions of rtr_open_channel and rtr_broadcast_event in the RTR C Application Programmer's Reference Manual.

Making Transactions Independent

Within your application server code, you identify those transactions that can be considered independent, and process them with the independent transaction flags on rtr_accept_tx or rtr_reply_to_client calls, as appropriate. For example, the following code fragment illustrates use of the independent transaction flag on the rtr_accept_tx call:

case rtr_mt_prepare: /* if (txn is independent).*/ status = rtr_accept_tx (channel, RTR_F_ACC_INDEPENDENT, RTR_NO_REASON\\); if (status != RTR_STS_OK)

You can also use the independent flag on the rtr_reply_to_client call. For example,

rtr_reply_to_client(channel, RTR_F_REP_INDEPENDENT, pmsg, msglen, msgfmt);

User Events Example

For example, the following snippet specifies named events for all subscribers:

rtr_status_t rtr_open_channel { . rtr_rcpnam_t rcpnam = "*"; rtr_evtnum_t evtnum = { RTR_EVTNUM_USERDEF, RTR_EVTNUM_USERBASE, RTR_EVTNUM_UP_TO, RTR_EVTNUM_USRMAX, RTR_EVTNUM_ENDLIST }; rtr_evtnum_t *p_evtnum = &evtnum;

For a broadcast to be received by an application, the recipient name specified by the subscribing application on its rtr_open_channel call must match the recipient specifier used by the broadcast sender on the rtr_broadcast_event call.

An application receives broadcasts with the rtr_receive_message call. A message type returned in the message status block informs the application of the type of broadcast received.

For example,

rtr_receive_message(.pmsg,maxlen,.pmsgsb);

The user event would be in msgsb.msgtype == rtr_mt_user_event . User broadcasts can also contain a broadcast message. This message is returned in the message buffer provided by the application. The size of the user's buffer is determined by the maxlen field. The number of bytes actually received is returned by RTR in the msglen field of the message status block.

RTR Events

An application subscribes to an RTR event with the rtr_open_channel call. For example,

rtr_status_t rtr_open_channel( . rtr_rcpnam_t rcpnam = RTR_NO_RCPNAM; rtr_evtnum_t evtnum = { RTR_EVTNUM_RTRDEF, RTR_EVTNUM_SRPRIMARY, RTR_EVTNUM_ENDLIST }; rtr_evtnum_t *p_evtnum = &evtnum; )

You read the message type to determine what RTR has delivered. For example,

rtr_status_t rtr_receive_message ( . rtr_msgsb_t *p msgsb )

Use a data structure of the following form to receive the message:

typedef struct { rtr_msg_type_t msgtype; rtr_usrhdl_t usrhdl; rtr_msglen_t msglen; rtr_tid_t tid; rtr_evtnum_t evtnum; /*Event Number*/ } rtr_msgsb_t;

The event number is returned in the message status block in the evtnum field. The following RTR events return key range data back to the client application:

RTR_EVTNUM_KEYRANGEGAIN RTR_EVTNUM_KEYRANGELOSS

These data are included in the message ( pmsg ); size is msglen_sizeof(rtr_msgsb_t) . Other events do not have additional data.

Diagnosing Performance Problems

Use the following brief checklist to help diagnose a particular performance problem:

1. Check the CPU load on the machines involved. A machine loaded over 60% is generally suspect, if reasonable response times are desired.
   Possible fixes:
   • Buy a more powerful CPU.
   • Add more SMP processors.
   • Partition the application workload over multiple machines.
   • Profile application CPU usage and optimize code hotspots.
2. Measure:
   • Disk I/O rate
   • Data rate on the disks used for the RTR journal
   • Data rate of the application
   
   These rates should be comfortably below the rated capacity of the controller and drives. If not, you may be on the trail of a performance constraint. Try:
   • Using faster disks or perhaps using disk caching. Note that disk caching can be vulnerable to data loss unless backed up by adequate auxiliary power supply.
   • Spreading the load over more disks, for example, using separate disks for RTR journal I/O and for application I/O.
   • Getting more concurrency in RTR journal I/O by increasing the number of RTR server processes or threads. RTR can combine the data transfers for multiple transactions into a single disk I/O, if these are being processed concurrently.
   • Reducing the size of messages sent from RTR client to server (if the problem is data-rate rather than I/O rate). Client-to-server user data are written to the RTR journal file, so removing redundancy from messages may help lower the data rate on the journal disk.
3. Measure RTR network traffic generated by the application. Use RTR MONITOR TRAFFIC for this while the application is running under load. Add the total bytes/second sent and received, and subtract the bytes/second sent and received from the local node to itself (intra-node data does not use the network). This total should be substantially lower than the measured capacity of the network.
   A rough-and-ready way to measure available network capacity is to do a file- transfer of a large file using FTP or some other program between the nodes, and divide the file size by the time taken. Note that multiple network connections may share the same hardware infrastructure, so you may need to try multiple simultaneous measurements between different node-pairs.
   If the RTR network traffic measured is not substantially less than the measured capacity of the network, then this may be the cause of the performance constraint for which you are looking. Try:
   • Using faster network hardware
   • Reducing the size of the application messages sent with RTR
   • Often, networks are tuned for high performance when transferring large files, but perform badly for bursty traffic. Buffering of either side of the transfer and of intermediate hops ensures smooth data flow. Check each hop to see if packets are being retransmitted due to excessive loss, and tune your network accordingly.
4. Measure delays in transmission through the network. Use "ping" to measure delay times between nodes whilst the system is under load. If reported round-trip delays are not in the low-millisecond range, you may be on to something. Additionally, use RTR MONITOR STALLS to measure whether delays are taking place in the acceptance of outgoing data by the network.
   If MONITOR STALLS shows a large number of stalls, especially in the columns for stalls longer than three seconds, then you very likely have a packet-loss problem in the network. Try:
   • Getting rid of that satellite link
   • Increasing the capacity of the network hardware
   • Checking that sufficient buffers are available for the network drivers on your machines
   • Upgrading or tuning network routers, bridges, and so on, if they are reporting packet losses.
     (If MONITOR STALLS reports lots of long stalls, but standard network analysis indicates that the network is operating as expected, check network utilization more closely. Packet losses which cause these glitches are usually caused by overload peaks in network traffic. You may still see disturbing long delays or link-losses when the system gets busy, even if average traffic is well below the capacity of the hardware.)
   
   Network monitors generally look at overall performance, measured over a period of time. It is often possible to show a 20 percent utilization of network bandwidth over time plotted at 5 minute intervals, but miss the peaks that last for 5 seconds and lose 50 packets. It is those 50 packets that account for the odd transaction getting a response time of 45 seconds instead of the usual 200 msec.
5. Check whether the throughput on your backend machines is being limited because all the servers are busy. Measure this by issuing the command RTR SHOW PARTITION/BACKEND /FULL on the backend machines. To observe this information with automatic updating of the display, use the MONITOR QUEUE or MONITOR GROUP command.

   Note Excessive use of a MONITOR command can be disruptive to the system. For example, running several MONITOR commands simultaneously steals cycles away from RTR to do real work. To minimize the impact of using MONITOR commands, increase the sample size interval using /INTERVAL= no-of-seconds .

   
   If the SHOW PARTITION command consistently shows the number of "Free Servers" as zero and the number of "Txns Active" larger than the number of servers, then a performance problem may be caused by queues building up because an inadequate number of server applications are ready to process incoming transactions. Try the following:
   • Fix any resource constraints limiting the application server's ability to do its work (CPU-load, disk-saturation, DB lock contention).
   • Increase the number of server processes or threads, so more work can be done concurrently.
6. If none of the above results in the TPS-rate you would like to see, are you sure that you are generating enough work for the servers to do? To check this, try increasing the number of clients accessing the system.