Deep Dive into Cluster Waits in RAC - Part 2

Block Oriented waits

Block oriented waits are the most common wait events in the cluster wait events. The block oriented wait event statistics indicate that the requested block was served from the other instances. In a two – node cluster environment a message is transferred to the current holder of the block and holder ships the block to the requestor. In a cluster environment with more than two nodes, the request for the block is sent to the holder of the block through the resource master and includes an additional message.

The average wait time and total wait time should be considered when being alerted to performance issues where these particular waits have a high impact. Usually , either interconnect or load issues of SQL execution against a large shared working ser can be found the root cause.The following are the most common block oriented waits

 gc current block 2 – way

gc current block 3 – way

gc cr block 2 – way

gc cr block 3 – way

Message Oriented Waits

The message oriented wait events statistics indicated that no block was received because it was not cached in any instance. Instead, a global grant was given, allowing the requesting instance to read the block from disk or modify it. These waits are normally followed with a disk read event such as “ db file sequential read” or “ db file scattered read”

If the time consumed by these  events is high, it may be assumed that the frequently executed SQL caused a lot of disk I/O ( in the event of the CR grant) or that the workload inserts a lot of data and needs to find and format a new blocks frequently ( in the event of the current grant).

The following are the most common message oriented waits

 gc current grant 2 – way

gc current grant 3 – way

gc cr grant 2 – way

gc cr grant 3 – way 

Contention – Oriented Waits

The contention oriented wait event statistics indicated that a block was received that was pinned by a session on another node, was deferred because a change had not yet been flushed to disk, or was deferred because of high concurrency, and there could not be shipped immediately. This typically happens with concurrent DMLs on index blocks where the other instance is waiting for the current block to add the index entry while the index block is busy in the holder. In this case the contention oriented wait “ gc current block busy” will be followed by the “gc current split” or “gc buffer busy” event.

A buffer may also be busy locally when a session has already initiated a Cache Fusion operation and is waiting for its completion when another session on the same node is trying to read or modify the same data.

High service times for blocks exchanged in the global cache may increase the contention, which can be caused by frequent concurrent read and write accesses to the same data. The following are the most common contention oriented waits

  gc current block busy

gc cr block busy

gc current buffer busy

Wait event “block busy” suggests that a single session required access to a specific block that is deemed busy on a remote node. A buffer busy indicates that multiple sessions are waiting for a block that is deemed busy on a remote node. In other words, dealing with a buffer busy situation involves multiple sessions contending for the same block requires tuning that is similar to dealing with buffer busy waits or read by other sessions in Oracle RAC environment.

Load Oriented Waits

The load – oriented wait events indicate that a delay in processing has occurred in the GCS, which is usually caused by high load, by CPU saturation, and would have to be solved by additional CPUs, load balancing , or off loading processing at different times or to a new cluster node. For the events mentioned , the wait time encompasses the entire roundtrip from the time a session starts to a wait after initiating a block request until the block arrives.

Whenever a request for any block transfer or message transfer waits for more than 1 ms in the internal queue after the interprocess communication (IPC) layer, the system is considered “ overaloaded”. This could happen when the LMS process is not able to catch up with the arrival rate and the messages are queued for LMS processing. The following are the most common load oriented waits, and existence of these waits indicated that the system is overloaded or underpowered.

gc current block congested

gc cr block congested

gc current grant congested

gc cr grant congested