$ RTR RTR> CLEAR /ALL (1) RTR> DISPLAY TEXT /X=25 /Y=1 "THE TEST PICTURE AT $TIME" (2) RTR> DISPLAY NUMERIC RTR_SOME_DATA_ITEM - (3) _RTR> /X=1 /Y=3 /LABEL ="SOME DATA ITEM: " RTR> DISPLAY NUMERIC RTR_ANOTHER_DATA_ITEM - (4) _RTR> /X=1 /Y=4 /LABEL ="OTHER DATA ITEM: " RTR> MONITOR (5) RTR> CLEAR /X=25 /Y=1 (6) RTR> DISPLAY TEXT /X=25 /Y=1 "THE NEW PICTURE AT $TIME" (7) RTR> SHOW DISPLAY /OUTPUT=MYPICTURE.MON (8) |
- Any items from previously displayed pictures are cleared.
- A title is displayed on the first line. The symbol $TIME is substituted by the current system time when the picture is displayed. (See Section A.2, Substitution Symbols).
- The data item called RTR_SOME_DATA_ITEM is displayed on the third line.
- The data item called RTR_ANOTHER_DATA_ITEM is displayed on the next line. Note that if /X and /Y are omitted, the item is displayed on the next free line and in the same column as the previous item.
- (5) The monitor picture is displayed on the screen. Figure A-1 shows what would appear on the screen at this point.
- The title is removed from the picture definition.
- A new title is added. This technique can also be used to make small alterations to standard monitor pictures.
- The definition of the picture is saved in the text file MYPICTURE.MON. The saved picture can be redisplayed by entering MONITOR MYPICTURE.
Figure A-1 Interactively Defined Monitor Picture
THE TEST PICTURE AT 11:49:24
SOME DATA ITEM: 0
OTHER DATA ITEM: 0
|
Because monitor file definitions depend on the internal structure and data items of RTR, they may need to be changed for future versions of RTR. |
A.2 Substitution Symbols
The text and labels in the DISPLAY commands that are used to define monitor pictures can contain symbols which are substituted when the picture is displayed. Table A-2 contains a list of these symbols.
| Symbol | Description |
|---|---|
| $TIME | Current time |
| $DATE | Current date |
| $NODE_NAME | Name of node where values were measured |
| $LINK_NAME | Name of link for which values were measured |
| $FACILITY_NAME | Name of facility for which value was measured |
| $PROCESS_ID | ID of process for which value was measured |
| $PROCESS_NAME | Name of process for which value was measured |
| $IMAGE_NAME | Image file name of process for which value was measured |
| $FULL_IMAGE_NAME | The full file specification of the image for the process for which values were measured |
The field width for these predefined symbols can be specified by
appending a colon (":") and then the required width. For example, $TIME displays the current system time in the
form "10:34:03", $TIME:5 displays the
first five characters of the system time, in this case, "10:34". If the
field width specified is larger than the field, then the data is left
justified and space filled to the required width.
A.3 Arithmetic Expressions and Operators
When using data items and constants as parameters for display commands, or as Booleans in the BELL, BLANK, BLINK, BOLD, SELECT, REVERSE and UNDERLINE qualifiers, you may use the operators shown in Table A-3.
| Characters | Meaning |
|---|---|
| - | Minus |
| + | Plus |
| * | Multiply |
| / | Divide |
| & | Logical AND |
| | | Logical OR |
| <= | Less than or equal |
| >= | Greater than or equal |
| != | Not equal |
| <> | Not equal |
| < | Less than |
| > | Greater than |
| = | Equal |
Example A-2 shows how these operators are used.
| Example A-2 Arithmetic Operators Examples |
|---|
RTR> DISPLAY NUMERIC RTR_SOME_DATA_ITEM+2 (1)
RTR> MONITOR
RTR> DISPLAY NUMERIC RTR_SOME_DATA_ITEM
_RTR> /SELECT="RTR_SOME_OTHER_DATA_ITEM>42" (2)
RTR> MONITOR
RTR> DISPLAY NUMERIC RTR_SOME_DATA_ITEM
_RTR> /SELECT="RTR_SOME_OTHER_DATA_ITEM>42" -
_RTR> /BOLD="RTR_SOME_OTHER_DATA_ITEM>84" -
_RTR> /UNDERLINE="RTR_SOME_OTHER_DATA_ITEM>=1952" (3)
RTR> MONITOR
RTR> DISPLAY SYMBOLIC RTR_SOME_DATA_ITEM "zero","one" -
_RTR> "two","three","four,"five","six"/BLINK (4)
|
- Two is added to the value of RTR_SOME_DATA_ITEM before displaying it.
- RTR_SOME_DATA ITEM is only displayed if RTR_SOME_OTHER_DATA_ITEM is greater than 42.
- Same as the one above, but displayed in bold if RTR_SOME_OTHER_DATA_ITEM is greater than 84, and underlined if its greater than 1952.
- The value of RTR_SOME_DATA_ITEM is displayed in text form if it is less than six, otherwise it is displayed as a numeric.
DISPLAY commands which select multiple instances of a data item (for example, multiple instances of a process counter) can use the following keywords to control the way the items are aggregated.
- _MIN---select the data item instance with the lowest value
- _MAX---select the data item instance with the higheest value
By default, data items are totaled unless the /AVERAGE=() qualifier is specified.
The following example from SYSTEM.MON shows the use of the _MIN keyword:
! Warn if any messages have been pending for more than 30 seconds
DISPLAY TEXT /X=32 /Y=17 -
"X" -
/BOLD -
/SELECT="...
((rtr:rtr_current_time - _MIN(prc:rtr_mt_received_time)) <= 30)"
DISPLAY TEXT /X=40 /Y=17 -
"X" -
/BOLD /BLINK -
/SELECT="...
((rtr:rtr_current_time - _MIN(prc:rtr_mt_received_time)) > 30)"
|
Appendix B
Server Shadowing and Recovery
RTR shadowing gives you the ability to recover from a site disaster without the need for special coding within your application program. This appendix is an introduction to RTR shadowing.
A server for a database partition is said to be shadowed when two copies of the same server perform identical actions on identical database copies on separate nodes. It is possible to declare servers to be members of a pair of shadow sites for any particular RTR key-range. Shadowed sites can either be two nodes within a single cluster, or can be two separate clusters.
Note that concurrent servers handle similar transactions, (that is, in the same
key-range but not the same transactions). Standby servers do not handle
transactions at all (for the given key-range) and shadow servers handle
the same transactions.
B.1 Primary and Secondary Roles
There is a concept of primary and secondary role for the shadow server pair, although in most cases this is transparent to the user when the processing is the same on both sites.
Initial role assignment is arbitrary, in that the first server of a
shadow pair to start is given the primary role, and the second the
secondary. The assigned roles may change, as servers come and go. Roles
are required, since RTR needs to determine the voting order on the
primary site before the transaction is presented to the secondary site.
B.2 Automatic Features
Shadow sites each have an identical copy of the customer's database.
Transactions are sent by RTR to both sites, and RTR ensures that they are processed by the servers in equivalent order on each of these sites, so that both copies of the customer database remain up-to-date.
A transaction is sent to the secondary site only after the primary has accepted it, or if the primary fails before being asked to vote.
RTR suppresses replies and broadcasts issued by the secondary shadow
server.
B.2.1 Shadow Events
RTR provides the following shadowing events:
- RTR_EVTNUM_SRPRIMARY - Server is in primary mode
- RTR_EVTNUM_SRSTANDBY - Server is in standby mode
- RTR_EVTNUM_SRSECONDARY - Server is in secondary mode
- RTR_EVTNUM_SRSHADOWLOST - Server has lost its shadow partner
- RTR_EVTNUM_SRSHADOWGAIN - Server has gained its shadow partner
- RTR_EVTNUM_SRRECOVERCMPL - Server has completed recovery
The shadow events are delivered with no special status and no data. They are delivered only to the server(s) whose state has changed.
A server receives RTR_EVTNUM_SRPRIMARY under the following circumstances:
- On initial startup if servers for the key-range are not already running on other nodes.
- If the server had previously been standby and the previous primary has failed.
- If the server had previously been shadow secondary and the previous primary has failed.
A server receives RTR_EVTNUM_SRSTANDBY when it starts up and servers already exist for the same key-range on another node in the same cluster.
A server receives RTR_EVTNUM_SRSECONDARY when it starts up and a shadow primary set of servers exist elsewhere.
A server receives RTR_EVTNUM_SRSHADOWLOST if it is running as primary and the secondary goes away.
A server receives RTR_EVTNUM_SRSHADOWGAIN if it is running as primary and a secondary node starts up.
A server receives RTR_EVTNUM_SRRECOVERCMPL when it has finished doing
recovery operations, and is hence ready to start processing new
transactions.
B.3 The RTR Journal System
The RTR journal is used for two purposes:
- RTR stores data about a transaction so that if the server processing a transaction fails for any reason, another server can transparently continue from where the previous server failed (if it is able to access the same database and journal).
- RTR also stores data about transactions when a shadow site is known to be missing. In this case, RTR stores all transaction data which can result in a database update. The RTR journal stores this data until the secondary shadow site is available again. RTR then transparently replays this data to the shadow site, after which the data is deleted from the journal.
The amount of space required for the journal depends upon:
- the size of the messages in a transaction,
- the number of messages in the transaction,
- the rate of generation of transactions,
- the maximum time a shadow site can be out of commission.
The /MAXIMUM_BLOCKS qualifier on the CREATE JOURNAL command controls how large a journal may become. The /MAXIMUM_BLOCKS qualifier defines the maximum number of blocks which the journal is allowed to occupy on any one disk. Note that RTR does not check if this amount of space is actually available, as the disk space specified by /MAXIMUM_BLOCKS is used only on demand by RTR when insufficient space is available in the space allocated by the /BLOCKS qualifier.
The number of blocks specified by the /BLOCKS qualifier specifies the size of the journal that RTR attempts to keep to. (The actual number of blocks used may vary, depending upon the load on RTR.)
The command MODIFY JOURNAL also accepts the /BLOCKS and /MAXIMUM_BLOCKS qualifiers.
Journal file extension occurs on demand when RTR detects that a "write to journal" would otherwise fail due to lack of space. Journal file truncation takes place automatically when blocks are freed.
Refer to MODIFY JOURNAL for the syntax description of the MODIFY JOURNAL command.
RTR> show journal/files/full RTR journal:- Disk: /dev/rz3a Blocks: 2500 Allocated: 1253 Maximum: 3500 File: //rtrjnl/anders/BRONZE.J00 RTR> |
B.4 Shadow Site Failure and Journaling
If a shadow-site fails, RTR allows transactions to continue to be processed on the remaining site. The intermediate transactions processed by the remaining server or servers are retained by RTR; when the failed site restarts these transactions are sent to this site as part of a shadow-recovery operation, thus bringing the failed site back up-to-date.
Since the transactions are stored in the RTR journal, it must be created with enough disk space in reserve to store data for the longest expected outage. It can be calculated using:
( Nr. transaction messages per second
multiplied by ( transaction message length + 70 )
multiplied by seconds of outage
) + 5% file overhead.
|
The overhead required when calculating journal size comes from internal journal data (block stamping) of approximately 3%. In addition, there is internal transaction data per (client to server) transactional message, and some further data per transaction (concerning voting and transaction completion).
Also, note that RTR prevents further transactional data being written to the journal when it is nearly full, but continues to allow deletes from the journal (deletes also cause data to be written to the journal). Ten segments are held in reserve for storing information about deleted transactions even when RTR cannot accept further transactions because the journal is full.
If the journal disk becomes full, transactions are aborted until the shadow partner re-starts and empties the journal of transactions to be replayed. |
B.4.1 Maximum Journal Size
The current maxima for the size of a journal are:
- Number of blocks per disk: 524288
(This is max_segments_per_disk * disk_blocks_per_segment, or 16384 times 32.) - Number of disks per journal: 16.
B.5 Standby for Shadows
Shadowed sites can either be two nodes within a single cluster, or can be two separate clusters. In the second case it is also possible to configure standby servers on another node of each of these clusters, so that failure of a single node within one of the shadow sites does not stop the shadow site from functioning. Multiple concurrent copies of the server processes are allowed on each site.
This implies that up to four nodes can be configured to serve a particular key range. (see Figure B-1).
Figure B-1 Four Node Shadow / Standby Configuration
B.6 Performance
The performance of a shadow pair compares with a transaction which
spans two nodes, with the addition of one extra protocol message which
is required to ensure that the transactions are presented in the
equivalent order.
Note that RTR does not have to wait for the secondary shadow server to complete its processing. It only needs to know that the primary has committed the transaction and that the journal file of the secondary shadow server contains the final vote status.
The two partners in a shadow pair should be connected with sufficient
bandwidth to cater for the possibly large amounts of data which may
need to be transferred during a shadow catchup operation.
B.7 Shadows in Action
The first node on which a shadow server for a particular key-range starts is arbitrarily designated by RTR to be the primary site for that key-range.
Initially RTR searches the journals of other backend sites to find any recoverable transactions left over from a previous invocation of the server. Once these have been processed (or RTR is able to determine that no such transactions exist), the server becomes active and available to handle new transactions sent by clients.
While no other server site for this key-range is available the server runs in REMEMBER mode, and RTR saves transactions processed on this site in the RTR journal (together with the order in which they should be committed), so that when the other site server(s) start they can be sent to this site.
When a server starts on a second site it begins processing the transactions saved in the primary site's journal. These are deleted from the journal as they are processed. When the second site server(s) have caught up, the second site enters SECONDARY state and the original site servers enter ACTIVE state. In this mode, new transactions are sent to both sites in parallel. They are executed first on the primary site, and shortly afterwards on the secondary site in equivalent order. The primary site commits transactions as soon as it knows that the secondary has hardened (i.e. written to the journal) the order in which the transaction is to be committed.
In the event of a failure at this point the remaining site executes a short tidy-up operation, and as soon as it has done this and determined that the other site is really down, it reverts to the REMEMBER state and carries on processing new transactions autonomously, saving the transaction information in its journal for when the other site restarts.
The execution order is determined for transactions issued to concurrent servers on a particular node by recording the order in which the individual servers issue rtr_accept_tx() calls. RTR knows that at the time a correctly written server application calls rtr_accept_tx() , it has already accessed (and hence locked) any database records it uses, and that it will release these records after RTR causes the rtr_accept_tx() call to complete. Any conflicting transaction would not be able to issue rtr_accept_tx() concurrently. Hence a correct serialisation order for issuing the transactions on the shadow site can be determined.
| Previous | Next | Contents | Index |