Reliable Transaction Router
C Application Programmer's
Reference Manual

To illustrate this, the example code evaluates your input parameters and places them into a message_data_t structure called ‘send_msg’. Message_data_t is not an RTR structure, but created by the programmer who wrote this client. The message_data_t structure is defined in ADG_HEADER.H.

typedef struct { rtr_uns_8_t routing_key; rtr_uns_32_t server_number; rtr_uns_32_t client_number; rtr_uns_32_t sequence_number; String31 text; } message_data_t;

You’ll notice that the data types which make up message_data_t aren’t your standard data types --- they are RTR data types. And they are generic enough to be able to be used on any operating system: 8 bit unsigned, 32 bit unsigned, and a string.

Earlier, we looked at the receive message code when the client opened a channel. The structure it used to obtain information, receive_message_t, is also one created by the programmer, and not a standard RTR structure. If you look at its definition in the ADG_HEADER.H file, you’ll see that it’s the same as the message_data_t structure, plus it contains a location for RTR status. There will be more on this in the next section.

Send/Receive Message Loop:
As mentioned earlier, the sample code for the RTR client application contains a message loop that sends messages to the server via the RTR router, and handles messages that come from the server via the router, or from RTR itself.

The following discussion will reference a simplified version of that loop; code in the sample to add time stamps and print to a log file has been removed here for clarity.

When you run the sample client, the client expects three parameters: a client number, a partition range, and the number of messages to send, in that order. We will talk more about partition ranges later when we look at the server application, but for now it is enough to know that we’ll use one character, the letter h.

The input command parameters are evaluated and placed in the message_data_t structure named send_msg. The ‘number of messages’ parameter which you’ll input on the command line is placed in the ‘txn_cnt’ variable. The ‘for’ loop which sends and receives messages will execute this number of times.

The message_data_t structure also holds a ‘sequence number’ value that is incremented each time the loop is executed; so now our loop begins:

for ( ; txn_cnt > 0; txn_cnt--, send_msg.sequence_number++ ) { status = rtr_send_to_server( channel, RTR_NO_FLAGS , &send_msg, sizeof(send_msg), RTR_NO_MSGFMT ); check_status( "rtr_send_to_server", status );

The first message has been sent to the server in the third parameter of the rtr_send_to_server call. As you will see, this is part of the flexibility and power of RTR. This third parameter is no more than a pointer to a block of memory containing your data. RTR doesn’t know what it’s a pointer to --- but it doesn’t need to know this. You, as the programmer, are the only one who cares what it is. It’s your own data structure that carries any and all of the information your server will need in order to do your bidding. We’ll see this in detail when we look at the server code.

In the fourth parameter, you must tell RTR how big the piece of memory being pointed to by the third parameter is. RTR needs to know how many bytes to move from your client machine to your server machine, so that your server application has access to the data being sent by the client.

The rest of the parameters bear some looking at, as well: there’s the channel again. You’ll see the channel parameter in almost every RTR call. You may be becoming suspicious about the channel, and think that it’s really more than just a line for communicating. And you’d be right. RTR uses the channel much like you use that third parameter in this call. The RTR developers are the only ones who know what’s in it, and it contains much of the information they need to make RTR work.

You’ll recognize two more default parameters, RTR_NO_FLAGS and RTR_NO_MSGFMT.

And now, the client waits for a response from the server.

/* * Get the server's reply OR an rtr_mt_rejected */ status = rtr_receive_message( &channel, RTR_NO_FLAGS, RTR_ANYCHAN, &receive_msg, sizeof(receive_msg), RTR_NO_TIMOUTMS, &msgsb); check_status( "rtr_receive_message", status );

Again you see the channel and the default flags; the receive_msg parameter is a pointer to another data structure created by you as the programmer, and can carry any information you need your server to be able to communicate back to the your client. In your own application, you would actually create this data structure in your application’s header file. You can see what the example receive message looks like by checking out the receive_msg_t in the ADG_HEADER.H file. RTR picks it up from your server and writes it here for your client to read.

The msgsb parameter is an RTR data structure: you saw this message status block earlier when we looked at the open channel code. Its msgtype field contains a code that tells you what kind of a message you are now receiving. If msgsb.msgtype contains the value rtr_mt_reply, then you are receiving a reply to a message you already sent, and your receive message data structure has been written to with information from your server.

switch (msgsb.msgtype) { case rtr_mt_reply: fprintf(fpLog, " sequence %10d from server %d\n ", receive_msg.receive_data_msg.sequence_number, receive_msg.receive_data_msg.server_number); break;

If msgsb.msgtype contains the value rtr_mt_rejected, then something has happened that caused your transaction to fail after you sent it to the router. You can find out what that ‘something’ is by looking at the status returned by the rtr_receive_message call. You will recall that making the rtr_error_text call and passing the status value will return a human readable null terminated ASCII string containing the error message.

case rtr_mt_rejected: fprintf(fpLog, " txn rejected at: %s", ctime( &time_val)); fprint_tid(fpLog, &msgsb.tid );

This is where you’ll need to make a decision about what to do with this transaction. You can abort and exit the application, issue an error message and go onto the next message, or resend the message to the server. This code resends a rejected transaction to the server.

/* Resend message with same sequence_number after reject */ send_msg.sequence_number--; txn_cnt++; break; default: fprintf(fpLog, " unexpected msg”); fprint_tid(fpLog, &msgsb.tid ); fflush(fpLog); exit((int)-1); }

When your client application receives an rtr_mt_reply message, your message has come full circle. The client has made a request of the server on behalf of the user; the server has responded to this request. If you’re satisfied that the transaction has completed successfully, you must notify RTR so that it can do its own housekeeping. To this point, this transaction has been considered "in progress", and its status kept track of at all times. If all parties interested in this transaction (this includes the client AND the server) notify RTR that the transaction has been completed, RTR will stop tracking it, and confirm to all parties that it has been completed. This is called ‘voting’.

if (msgsb.msgtype == rtr_mt_reply) { status = rtr_accept_tx( channel, RTR_NO_FLAGS, RTR_NO_REASON ); check_status( "rtr_accept_tx", status );

And now the client waits to find out what the result of the voting is.

status = rtr_receive_message( &channel, RTR_NO_FLAGS, RTR_ANYCHAN, &receive_msg, sizeof(receive_msg), receive_time_out, &msgsb); check_status( "rtr_receive_message", status ); time(&time_val);

If everyone voted to ‘accept’ the transaction, the client can move on to the next one. But if one of the voters rejected the transaction, then another decision must be made regarding what to do about this transaction. This code attempts to send the transaction to the server again.

switch ( msgsb.msgtype ) { case rtr_mt_accepted: fprintf(fpLog, " txn accepted at : %s", ctime( &time_val)); break; case rtr_mt_rejected: fprintf(fpLog, " txn rejected at : %s", ctime( &time_val)); /* Resend same sequence_number after reject */ send_msg.sequence_number--; txn_cnt++; break; default: fprintf(fpLog, " unexpected status on rtr_mt_accepted message\n"); fprintf(fpLog, " %s\n", rtr_error_text(receive_msg.receive_status_msg.status); break; } } } /* end of for loop */

All of the requested messages, or transactions, have been sent to the server, and responded to. The only RTR cleanup we need to do before we exit the client is to close the channel. This is similar to signing off, and RTR releases all of the resources it was holding for the client application.

close_channel ( channel );

Now, that wasn’t so bad, was it? Of course not. And what has happened so far? The client application has sent a message to the server application. The server has responded. RTR has acted as the messenger by carrying the client’s message and the server’s response between them.

Next, let’s see how the server gets these messages, and sends a response back to the client.

Server App:
The files shipped with the RTR kit used in the server application for this tutorial are ADG_SERVER.C, ADG_SHARED.C and ADG_HEADER.H. You’ll notice that ADG_SHARED.C and ADG_HEADER.H are used in both client and the server applications. This is for a number of reasons, but most importantly that both the client and the server will use the same definition for the data structures they pass back and forth as messages.

With the exception of only two items, there will be nothing in this server that you haven’t already seen in the client. It’s doing much the same things as the client application is doing. It opens a channel to the router, telling the router that it is a server application; waits to hear that the open channel request has been successfully executed; runs a loop that receives messages from the client; carries out the client’s orders; sends the response back to the client. And the server gets to vote, too, on whether each message/response loop is completed.

One of the differences is the types of messages a server can receive from RTR; we’ll go through some of them in this section of the tutorial about the server application.

The other difference is the declaration of a partition which is sent to RTR in the open channel call. We mentioned partitions while discussing the client application, but said we’d discuss them later. Well, it’s later...

Initialize RTR:
Just like the client, the server opens a channel to the router, causing RTR to initialize a number of resources for use by the server, as well as to gather information about the server. In the declare_server function in the server example application, ADG_SERVER.C, you’ll find the example server calling rtr_open_channel.

Immediately, you see that the code initializes an RTR data structure called rtr_keyseg_t. In the example server code, the variable name of the structure is p_keyseg. This structure is a required parameter in the server open channel call in order to implement data partitioning.

Data Partitions:
What is data partitioning, and why would you wish to take advantage of it?

It is possible to run a server application on each of multiple backend machines, and to run multiple server applications on any backend machine. When a server opens a channel to begin communicating with the RTR router, it uses the rtr_keyseg_t information in its last two parameters to tell RTR that it is available to handle certain key segments. A key segment can be "all last names that start with A to K" and "all last names that start with L to Z", or "all user identification numbers from zero to 1000" and "all user identification numbers from 1001 to 2000".

Each key segment describes a data partition. Data partitions allow you to use multiple servers to handle the transactions all of your clients are attempting to perform; in this way, they don’t all have to wait in line to use the same server. They can get more done in less time.

The RTR Application Design Guide and API reference manual go into much more detail about data partitioning.

This is how the example server application defines the key segment that it will handle:

p_keyseg[0].ks_type = rtr_keyseg_string; p_keyseg[0].ks_length = 1; p_keyseg[0].ks_offset = 0; p_keyseg[0].ks_lo_bound = &outmsg->routing_key; p_keyseg[0].ks_hi_bound = &outmsg->routing_key;

It tells RTR that this server is interested only in records containing a string of 1 byte at the beginning of the record; this actually makes it a single character. The value of that character is from and including the value of the character in the routing_key field of outmsg, to and including the value of the character in the routing_key field of outmsg. As you can see, this too describes only one character.

The structure ‘outmsg’ is actually a msg_data_t structure, which is the structure you saw the client application using to pass data to the server application. The value of this character is input when you start the server. Since we decided to use the letter h when we start the client, it would be really nice if the server we started identified itself as one who could handle the client’s request. So we’ll start the server using h as well; in this way, the h gets into outmsg->routing_key. The complete server command line for both the client and the server is documented later in this tutorial.

status = rtr_open_channel( channel, RTR_F_OPE_SERVER, "RTRTutor", NULL, RTR_NO_PEVTNUM, NULL, 1, p_keyseg); check_status( "rtr_open_channel", status);

The server has requested a channel on which to communicate with RTR, and advertised itself as handling all requests from clients in the RTRTutor facility that have a key segment value of h. The remaining parameters contain defaults.

Now the server waits for a message confirming that RTR opened the channel successfully. If it did, the server can then begin receiving requests from the client, via RTR.

status = rtr_receive_message( channel, RTR_NO_FLAGS, RTR_ANYCHAN, &receive_msg, sizeof(receive_msg), receive_time_out, &msgsb); check_status( "rtr_receive_message", status);

Again, we use the RTR rtr_msgsb_t structure that RTR will place information in, and the user- defined receive_msg_t data structure (see ADG_HEADER.H) that the client’s data will be copied into. But at this point, the server is talking with RTR only, not the client, so it is expecting an answer from RTR in msgsb; all the server really wants to know is that the channel has been successfully opened. If it isn’t, the server application will write out an error message and exit with a failure status.

if ( msgsb.msgtype != rtr_mt_opened ) { fprintf(fpLog, " Error opening rtr channel : \n"); fprintf(fpLog, "%s", rtr_error_text(receive_msg.receive_status_msg.status); fclose (fpLog); exit(-1) } fprintf(fpLog, " Server channel successfully opened \n"); return;

And now that the channel has been established, the server waits to receive messages from the client application and the RTR router.

Receive/Reply Message Loop:
The server sits in a message loop receiving messages from the router, or from the client application via the router. Like the client, it must be prepared to receive various types of messages in any order and then handle and reply to each appropriately. But the list of possible messages the server can receive is different than that of the client. This example includes some of those.

First, the server waits to receive a message from RTR.

while ( RTR_TRUE ) /* always, or until we exit */ { status = rtr_receive_message( &channel, RTR_NO_FLAGS, RTR_ANYCHAN, &receive_msg, sizeof(receive_msg), receive_time_out, &msgsb); check_status( "rtr_receive_message", status);

Like the client, upon receiving the message the server checks the rtr_msgsb_t structure’s msgtype field to see what kind of message it is. Some are messages directly from RTR and others are from the client. When the message is from the client, your application will read the data structure you constructed to pass between your client and server and, based on what it contains, do the work it was written to do. In many cases, this will involve storing and retrieving information using your favorite database.

But when the message is from RTR, how should you respond? Let’s look at some of the types of messages a server gets from RTR, and what should be done about them.

switch (msgsb.msgtype) { case rtr_mt_msg1_uncertain: case rtr_mt_msg1:

The first message this server application prepares to handle is the rtr_mt_msg1_uncertain message. This is combined with the handler for the rtr_mt_msg1 message.

The msg1 messages identify the beginning of a new transaction. Rtr_mt_msg1 says that this is a message from the client, and it’s the first in a transaction. When you receive this message type, you will find the client data in the structure pointed to by the fourth parameter of this call. The client and server have agreed on a common data structure that the client will send to the server whenever it makes a request: this is the message_data_t we looked at in the client section of this document. RTR has copied the data from the client’s data structure into the one whose memory has been supplied by the server. The server’s responsibility when receiving this message is to process it.

Recovered Transactions:
The rtr_mt_msg1_uncertain message type tells the server that this is the first message in a recovered transaction. The original server the application was communicating with failed, possibly leaving some of its work incomplete, and now the client is talking to the standby server. What happens to that incomplete work left by the original server?

Looking back at the client you will recall that everyone got to vote as to whether the transaction was accepted or rejected, and then the client waited to see what the outcome of the vote was. While the client was waiting for the results of this vote, the original server failed, and the standby server took over. RTR uses the information it kept storing to the recovery journal, which you also created earlier, to replay to the standby server so that it can recover the incomplete work of the original server.

When a server receives the ‘uncertain’ message, it knows that it is stepping in for a defunct server that had, to this point, been processing client requests. But it doesn’t know how much of the current transaction has been processed by that server, and how much has not, even though it receives the replayed transactions from RTR. The standby server will need to check in the database or files to see if the work represented by this transaction is there and, if not, then process it. If it has already been done, the server can forget about it. In the examples, rtr_msgsb_t must be declared as a variable, as rtr_msgsb_t msgsb;.

if (msgsb.msgtype == rtr_mt_msg1_uncertain) replay = RTR_TRUE; else replay = RTR_FALSE; if ( replay == TRUE ) /* The server should use this opportunity to * clean up the original attempt, and prepare * to process this request again. */ else /* * Process the request. */

The server then replies to the client indicating that it has received this message and handled it.

reply_msg.sequence_number = receive_msg.receive_data_msg.sequence_number; status = rtr_reply_to_client ( channel, RTR_NO_FLAGS, &reply_msg, sizeof(reply_msg), RTR_NO_MSGFMT);

The rtr_reply_to_client call is one you haven’t seen before. Obviously, it is responding to a client’s request. This call may not be used in an application that has declared itself a client.

The server is using the rtr_reply_to_client call to answer the request the client has made. In some cases, this may mean that data needs to be returned. This will be done in the ‘reply_msg’ structure which, like the send_msg structure, has been agreed upon by both the client and the server. RTR will copy ‘sizeof’ bytes from the server’s copy of the reply_msg into the client’s copy.

check_status( "rtr_reply_to_client", status); break; case rtr_mt_prepare:

Prepare Transaction:
The rtr_mt_prepare message tells the server to prepare to commit the transaction. All messages from the client which make up this transaction have been received, and it is now almost time to commit the transaction in the database.

This message type will never be sent to a server which has not requested an explicit prepare. To make this request, the server must use the RTR_F_OPE_EXPLICIT_PREPARE flag in the ‘flags’ parameter when opening the channel.

After determining whether it is possible to complete the transaction based on what has occurred to this point, the server can either call rtr_reject_tx to reject the transaction, or set all of the required locks on the database before calling rtr_accept_tx to accept the transaction.

Because this example code is not dealing with a database, nor is it bundling multiple messages into a transaction, the code here immediately votes to accept the transaction.

status = rtr_accept_tx ( channel, RTR_NO_FLAGS, RTR_NO_REASON); check_status( "rtr_accept_tx", status); break; case rtr_mt_rejected:

Transaction Rejected:
The rtr_mt_rejected message is from RTR, telling the server application that a participant in the transaction voted to reject it. If one participant rejects the transaction, it fails for all. The transaction will only be successful if all participants vote to accept it.

When it receives this message, the server application should take this opportunity to roll back the current transaction if it is processing database transactions.

break; case rtr_mt_accepted:

Transaction Accepted:
RTR is telling the server that all participants in this transaction have voted to accept it. If database transactions are being done by the server, this is the place at which the server will want to commit the transaction to the database, and release any locks it may have taken on the database.

break; } /* end of switch */ } /* end of while loop */

Note that there is no close_channel call in the server. This is because the RTR router closes the channel and stops the server when it sees fit. RTR makes this decision.

That’s it. You now know how to write a client and server application using RTR as your network communications, availability and reliability infrastructure. Congratulations!

Build and Run the Servers:
Compile the ADG_SERVER.C and ADG_SHARED.C module on the operating system that will run your server applications. If you are using two different operating systems, then compile it on each of them.

To build on UNIX, issue the command:

cc -o server adg_server.c adg_shared.c /usr/shlib/librtr.so -DUNIX

You should start the servers before you start your clients. They will register with the RTR router so that the router will know where to send client requests. Start your primary server with the appropriate ‘run’ command for its operating system along with the two parameters ‘1’ and ‘h’. To run on UNIX: