hp Reliable Transaction Router
System Manager's Manual

Displays a list of the links to other nodes. For each link, the total number of bytes received and sent on that link and the number of bytes received and sent per second are displayed. Derived from the NIO_BYTES_RCVD and NIO_BYTES_SENT counters. The Max field represents the maximum rate since the link started.

Example 7-25 Monitor Netbytes

LINK TRAFFIC IN BYTES Fri Apr 16 1999 17:41:12, NODE: nodea Bytes received Bytes sent -------------------------- -------------------------- Count Rate Max Count Rate Max Total 59201466 6776.0 - 1002579480 113857.0 - nodea->nodea 3072248 336.0 336.0 3072248 336.0 336.0 nodea->nodeb 42569496 4974.0 4974.0 717457678 84438.0 84438.0 nodea->nodec 13559722 1466.0 1466.0 282049554 29083.0 29083.0

7.2.27 Monitor Netstat

Displays the link status for connected links in detail and the fail code for any links on which a connection has failed. Unconnected links where connections have been lost are highlighted. Link aborts, rejects, loss, gain, restarts, state and architecture of the remote node are also displayed. Displays more detail than MONITOR CONNECTS.

Example 7-26 Monitor Netstat

C o n n e c t i o n S t a t u s D e t a i l Node: NODEA Mon March 15 1999 09:50:28 Ini Cnf Acc Abo Rej Loss Gain Ctmo Rstr AFO(1) TFO(2) State Type FailCode Node and Link 12 0 2 12 12 1 3 2074 2073 863 864 NODEA ->nodeb 0 0 0 0 0 0 1 0 0 0 0 up alpha NODEA ->nodec 6 0 0 6 6 0 0 0 0 0 0 down ? 76490676 NODEA ->noded 6 0 0 6 6 0 0 0 0 0 0 down ? 76490676 NODEA ->nodee 2591 0 0 863 2590 0 1 2074 2073 863 864 cinit ? Remote node unreachable

1. Address failover
2. Transport failover

Displays lists of server and client transaction controllers.

Example 7-27 Monitor Ortr

oRTR Transaction Controller List on node : rtrdoc Image: -ALL- at 15:42:55 Server Controllers Client Controller STC000 CTC000 STC001 CTC001 STC002 CTC002 STC003 CTC003 STC004 CTC004

7.2.29 Monitor Partit

Displays partitions as partition name if a partition name has been specified using the SET PARTITION command. The number of servers and key segments are shown for each partition. The least-significant byte of the partition's low and high bound is also shown, and callout type (if any). Table 7-3 shows partition state meanings.

Example 7-28 Monitor Partit

PARTITION DEFINITIONS Tue Apr 6 1999 10:40:31, NODE: NODEA Partition name # # Bounds Callout State Svrs Segs lo hi type RTR$DEFAULT_PARTITION_16777218 active 1 1 "A" "A" - RTR$DEFAULT_PARTITION_16777221 active 1 1 "B" "B" - RTR$DEFAULT_PARTITION_16777217 active 1 1 0 429496 -

Table 7-3 Monitor Partition States

State	Meaning
wt_tr_ok	Server is waiting for routers to accept it
wt_quorum	Server is waiting for backend to be quorate
lcl_rec	Local recovery
lcl_rec_fail	Primary server waiting for access to a restart journal
lcl_rec_icpl	Getting next journal to recover from
lcl_rec_cpl	Processed all journals for local recovery
shd_rec	Shadow recovery
shd_rec_fail	Shadow server waiting for access to a restart journal
shd_rec_icpl	Shadow getting next journal to recover from
shd_rec_cpl	Processed all journals for shadow recovery
catchup	Secondary is catching up with primary
standby	Server is declared as standby
active	Server is active
pri_act	Server is active as primary shadow
sec_act	Server is active as secondary shadow
remember	Primary is running without shadow secondary

7.2.30 Monitor Queues

Shows transaction queues by partition. Uses counters from Transaction Manager (TM) and the Requester/Server configurator (RSC). A delay is the delay in seconds for the most recent transaction handed to a server, which on average will be the same for all servers.

Example 7-29 Monitor Queues

TRANSACTION QUEUES BY PARTITION Fri Sep 08 2000 12:42:53, NODE: NODEA Partition name Processed Queued # Txns Msgs Rplys Txn Msg Svrs Dly RTR$DEFAULT_PARTITION_16842753 5792 5794 0 2 6 3 0

7.2.31 Monitor Quorum

Displays for each node and facility, the state and the number of roles RTR sees as configured (CNF), reachable (RCH) and quorate (QRT). The State column shows quorum states for router (TR) and backend (BE) roles.

Example 7-30 Monitor Quorum

QUORUM STATUS BY NODE AND FACILITY Tue Dec 04 2001 10:50:24, NODE: NODEA (Counts can be inaccurate for incorrectly configured facilities) States: bad configuration,not connected,minority,uncertain,quorate Node & Facility State CNF RCH QRT NODEA RTR$DEFAULT_FACILITY TR:quorate,BE:quorate 2 2 2 NODEA shadow TR:quorate,BE:minority(node12) 4 2 1 NODEA payroll TR:bad_config(NODEB) 1 0 0

7.2.32 Monitor Recovery

Shows the progress of transaction recovery. Last recovery backend is the last backend accessed to recover transactions. If the server state is lcl_rec_fail or shd_rec_fail , the Last Recovery Backend is the backend that could not be accessed. Journal scans is the number of journal files searched. Transactions recovered is the number of transactions found for this partition.

Table 7-4 shows the meaning of entries in the Server State column.

Example 7-31 Monitor Recovery

RECOVERY INFORMATION at Tue Apr 6 1999 10:54:50, on NODEA Last Restart-Recovery Shadow-Recovery Server Recovery Journal Txns Journal Txns Partition State Backend Scans Recovered Scans Recovered ------------ ------ ------- ------- --------- ------- --------- RTR_PARTITA active NODEA 1 0 0 0 RTR_PARTITB active NODEA 1 0 0 0 RTR_PARTITC active NODEA 1 0 0 0

Table 7-4 Monitor Recovery States

State	Meaning
wt_tr_ok	Server is waiting for routers to accept it
wt_quorum	Server is waiting for backend to be quorate
lcl_rec	Local recovery
lcl_rec_fail	Primary server waiting for access to a restart journal
lcl_rec_icpl	Getting next journal to recover from
lcl_rec_cpl	Processed all journals for local recovery
shd_rec	Shadow recovery
shd_rec_fail	Shadow server waiting for access to a restart journal
shd_rec_icpl	Shadow getting next journal to recover from
shd_rec_cpl	Processed all journals for shadow recovery
catchup	Secondary is catching up with primary
standby	Server is declared as standby
active	Server is active
pri_act	Server is active as primary shadow
sec_act	Server is active as secondary shadow
remember	Primary is running without shadow secondary

7.2.33 Monitor Rejects

Displays the last rtr_mt_rejected message received by each running process. Table 7-5 gives the meaning of each column.

Example 7-32 Monitor Rejects

Rejected Transaction Summary NODE: NODEA PROCESS: 20413894 Fri Apr 9 1999 10:26:14 Time Pid Chan Reason Status Text ------------------- ------ ------ -------- ----------------------------- Fri Apr 9 10:18:43 20417266 client 0 No server available to handle Fri Apr 9 10:17:47 20417274 server 0 Client aborted tx

Table 7-5 Monitor Rejects Fields

Field	Meaning
Time	Date and time when the rtr_mt_rejected message was received
Pid	ID of the process that received the message
Chan	Type of channel (client or server) that received the message
Reason	Reason field returned in the rtr_status_data_t buffer, if any
Status Text	Text describing the reject reason

7.2.34 Monitor Rejhist

Displays the last 10 rtr_mt_rejected messages received by the selected process. Always invoke this picture with the /ID qualifier. The transaction identifier associated with the rejected transaction can be displayed with the SHOW PROCESS < id>/COUNTER=api_reject* command. Table 7-6 shows the meaning of Rejhist columns.

Example 7-33 Monitor Rejhist

Rejected Transaction History NODE: NODEA PROCESS: 38009A8B Mon Mar 9 1999 10:26:14 Time Chan Reason Status Text ------------------- ------ ------- -------------------------------------- Mon Mar 15 18:06:06 client 0 Client aborted tx Mon Mar 15 18:06:41 server 0 Normal successful completion Mon Mar 15 18:06:41 client 0 Server aborted tx ------------------------------------------------------------------------------ number of reject messages: 3 number of accept messages: 0 rejected / total txns (%): 100% This display should be invoked with the /ID qualifier.

Table 7-6 Monitor Rejhist Fields

Field	Meaning
Time	Date and time when the rtr_mt_rejected message was generated
Chan	Type of channel (client or server) that received the message
Reason	Reason field returned in the rtr_status_data_t buffer, if any
Status Text	Text describing the reject reason

7.2.35 Monitor Response

Displays the elapsed time that a transaction has been active on the opened channels of a process. On the client, transaction duration is measured between the rtr_start_tx or rtr_send_tx call and the receipt of the final rtr_mt_accepted or rtr_mt_rejected message. A call to rtr_reject_tx also marks the end of a transaction. On the server, transaction duration is measured between receipt of an rtr_mt_msg1 or rtr_mt_msg1_uncertain message and the receipt of the final rtr_mt_rejected message or rtr_reject_tx call. Accepted transaction end times are recorded when the server issues an rtr_receive_message call to request a new transaction for processing.

Example 7-34 Monitor Response

TRANSACTION DURATION AT 10:24:51 Fri Apr 9 1999 Process ID, Image Name Client Response Time Server Response Time seconds 0---1---2---3---4 seconds 0---1---2---3---4 20413894 SERVER.EXE;4 0.000 3.670 20417266 RTR.EXE;75 2.200 3.440 20417274 SERVER.EXE;4 0.000 1.160

7.2.36 Monitor Rolequor

Displays for each facility the number of roles RTR sees as configured (CNF), reachable (RCH), and quorate (QRT), from the point of view of the routers and the backends. Compare Section 7.2.31.

Example 7-35 Monitor Rolequor

QUORUM COUNTS BY FACILITY Tue Dec 04 2001 14:32:48, NODE: NODEA Facility name Router View of Backend View of Backends Routers Backends Routers CNF RCH QRT CNF RCH QRT CNF RCH QRT CNF RCH QRT RTR$DEFAULT_FACILITY 1 1 1 1 1 1 1 1 1 1 1 1 funds_transfer 1 0 0 0 0 0

7.2.37 Monitor Routers

Displays information on a router node. It gives an indication of the utilization of the router in terms of transactions and broadcasts routed through this node. Useful to monitor performance or locate problems. Uses counters in the Transaction Manager (TM) subsystem.

Example 7-36 Monitor Routers

ROUTER TRANSACTION COUNTERS AT 14:33:29 Fri Sep 08 2000 Node: NODEA Facility: -ALL- Count Rate 10 20 30 40 50 60 70 80 90 100 Starts: 116641 25.7 Enqueues: 116641 25.7 Commits: 116641 25.6 Aborts: 0 0.0

7.2.38 Monitor Routing

Displays statistics of transaction and broadcast traffic by facility. Rate is the number of transactions or broadcasts per second within the monitoring interval. A value in the No KR column indicates the absence of an appropriate key range server. A value in the NoDst column indicates the absence of an appropriate broadcast destination. Counts in these columns probably mean that a required server application is missing or perhaps that there is a problem with application configuration.

Example 7-37 Monitor Routing

ROUTING STATISTICS BY FACILITY Tue Dec 04 2001 14:34:20, NODE: NODEA Facility name Transactions Broadcasts Count Rate NoKR Count Rate NoDst Total 118489 39.2 0 68444 994.0 1 RTR$DEFAULT_FACILITY 118489 39.2 0 68444 994.0 1

7.2.39 Monitor Rscbe

Displays, by partition, the history of the most recent request-to-server messages issued to a backend from each router. The RSC calls column shows the message type, the Router column gives the router node name, the State column shows the partition state for messages to a router and the target state for messages from a router. T-delta is the elapsed time in seconds since the partition was established, and Seq_nr gives the partition sequence number or priority.

Example 7-38 Monitor Rscbe

Most Recent RSC Dclsrv Calls History on Backend NODEA Thu Mar 4 1999,15:19:41 Partition name: RTR$DEFAULT Partition Start Time: THU MAR 4 15:18:22 1999 Image Name: RTR.EXE T-delta RSC calls Router State Seq_nr 0 send_dcl_to_master nodez wait_for_response 0 1 recv_status_ok nodez rstart_rvy 1 1 send_dcl_to_master nodez rstart_rvy_incomp 1 1 recv_status_ok nodez rstart_rvy 1 1 send_dcl_to_master nodez rstart_rvy_incomp 1 1 recv_status_ok nodez active 1 1 send_dcl_to_other node10 active 1 1 recv_status_ok node10 active 1

7.2.40 Monitor Stalls

Displays in real time any network links that are currently stalling (that is, waiting to transmit outbound traffic) and provides a history of the stalls encountered by the various links during their lifetime. The display shows:

• Node-to-node path of link analyzed
• The outgoing network message count per link
• The outgoing network message rate per link
• The number of bytes sent per link
• The number of link disconnects
• Total accumulated network write-completion wait time, in seconds
• Count of stalls between 1 and 3 seconds
• Count of stalls between 3 and 10 seconds
• Count of stalls between 10 and 30 seconds
• Count of stalls longer than 30 seconds
• Count of stalled network write completions

Example 7-39 Monitor Stalls

NETWORK STALLS AT Fri Sep 08 2000 15:35:03, ON NODE: TR1 Messages Bytes Link Stalls Issued Rate Sent Drops Secs <3s <10s <30s >30s Totals Total 5467 0.0 327148 2 33 23 1 0 0 24 TR1 -> TR1 29 0.0 3718 0 0 0 0 0 0 0 TR1 -> FE2 509 0.0 20707 0 4 4 0 0 0 4 TR1 -> BE1 303 0.0 13707 0 3 3 0 0 0 3 TR2 -> TR2 111 0.0 11682 0 0 0 0 0 0 0 TR2 -> BE1 504 0.0 22743 0 18 8 1 0 0 9 FE1 -> FE1 64 0.0 6645 0 0 0 0 0 0 0 FE1 -> FE2 373 0.0 18890 0 2 2 0 0 0 2 FE1 -> BE1 310 0.0 24487 0 0 0 0 0 0 0 FE2 -> FE2 231 0.0 18900 0 0 0 0 0 0 0 FE2 -> BE1 284 0.0 22503 1 1 1 0 0 0 1 FE2 -> TR1 536 0.0 28166 0 0 0 0 0 0 0 FE2 -> FE1 396 0.0 23643 0 0 0 0 0 0 0 BE1 -> BE1 355 0.0 28121 0 0 0 0 0 0 0 BE1 -> FE2 284 0.0 13014 1 0 0 0 0 0 0 BE1 -> TR2 515 0.0 27502 0 2 2 0 0 0 2 BE1 -> TR1 328 0.0 25698 0 1 1 0 0 0 1 BE1 -> FE1 335 0.0 17022 0 2 2 0 0 0 2