Search Oracle Related Sites

Wednesday, September 21, 2011

Deep Dive into Cluster Waits in RAC - Part 2


Block Oriented waits

Block oriented waits are the most common wait events in the cluster wait events. The block oriented wait event statistics indicate that the requested block was served from the other instances. In a two – node cluster environment a message is transferred to the current holder of the block and holder ships the block to the requestor. In a cluster environment with more than two nodes, the request for the block is sent to the holder of the block through the resource master and includes an additional message.

The average wait time and total wait time should be considered when being alerted to performance issues where these particular waits have a high impact. Usually , either interconnect or load issues of SQL execution against a large shared working ser can be found the root cause.The following are the most common block oriented waits

 gc current block 2 – way
gc current block 3 – way
gc cr block 2 – way
gc cr block 3 – way

Message Oriented Waits

The message oriented wait events statistics indicated that no block was received because it was not cached in any instance. Instead, a global grant was given, allowing the requesting instance to read the block from disk or modify it. These waits are normally followed with a disk read event such as “ db file sequential read” or “ db file scattered read”

If the time consumed by these  events is high, it may be assumed that the frequently executed SQL caused a lot of disk I/O ( in the event of the CR grant) or that the workload inserts a lot of data and needs to find and format a new blocks frequently ( in the event of the current grant).

The following are the most common message oriented waits

 gc current grant 2 – way
gc current grant 3 – way
gc cr grant 2 – way
gc cr grant 3 – way 

Contention – Oriented Waits

The contention oriented wait event statistics indicated that a block was received that was pinned by a session on another node, was deferred because a change had not yet been flushed to disk, or was deferred because of high concurrency, and there could not be shipped immediately. This typically happens with concurrent DMLs on index blocks where the other instance is waiting for the current block to add the index entry while the index block is busy in the holder. In this case the contention oriented wait “ gc current block busy” will be followed by the “gc current split” or “gc buffer busy” event.

A buffer may also be busy locally when a session has already initiated a Cache Fusion operation and is waiting for its completion when another session on the same node is trying to read or modify the same data.

High service times for blocks exchanged in the global cache may increase the contention, which can be caused by frequent concurrent read and write accesses to the same data. The following are the most common contention oriented waits

  gc current block busy
gc cr block busy
gc current buffer busy

Wait event “block busy” suggests that a single session required access to a specific block that is deemed busy on a remote node. A buffer busy indicates that multiple sessions are waiting for a block that is deemed busy on a remote node. In other words, dealing with a buffer busy situation involves multiple sessions contending for the same block requires tuning that is similar to dealing with buffer busy waits or read by other sessions in Oracle RAC environment.

Load Oriented Waits

The load – oriented wait events indicate that a delay in processing has occurred in the GCS, which is usually caused by high load, by CPU saturation, and would have to be solved by additional CPUs, load balancing , or off loading processing at different times or to a new cluster node. For the events mentioned , the wait time encompasses the entire roundtrip from the time a session starts to a wait after initiating a block request until the block arrives.

Whenever a request for any block transfer or message transfer waits for more than 1 ms in the internal queue after the interprocess communication (IPC) layer, the system is considered “ overaloaded”. This could happen when the LMS process is not able to catch up with the arrival rate and the messages are queued for LMS processing. The following are the most common load oriented waits, and existence of these waits indicated that the system is overloaded or underpowered.

gc current block congested
gc cr block congested
gc current grant congested
gc cr grant congested


Deep Dive into Cluster Waits in RAC Part 1


When a session requests access to a data block in CR/CUR ( Consistent Read / Current) mode , it sends a request to the lock master for proper authorization. However , whether it will receive the block via the cache fusion mechanism or a permission to read from the disk is not known until the request is satisfied. Two placeholder events – “gc cr request” and “gc current request”  - keep track of the time a session spends in this state.

For example, in a two node cluster when a session sends a request for CR block, its wait time is counted against “gc cr request”.  Let’s assume that the lockmaster grants the lock to the requestor, there by authorizing it to read the lock from the disk so all the wait time is now logged against the wait event “ gc cr grant 2 –way”. Similarly, the wait time would be logged against “gc cr block 2-way” if the requesting instance received the CR copy of the block from the master.

The cluster wait events belong to one of the following categories

1)      Block Oriented waits

2)      Message – Oriented waits

3)      Contention Oriented waits

4)      Load Oriented waits

Tuesday, September 20, 2011

Top 5 Timed Events - RAC GC Wait Events

gc current grant 2 –way

In a 2 Node or 3 Node RAC environment when a instance needs a block in current mode, it sends the request to the master instance. The master instance finds that currently no instances, including self , has any lock on the requested block. It sends a message back to the requesting instance granting it the shared lock on the block. The requesting instance then reads the block from the disk. This event does not indicate any contention. The presence of this event in the Top 5 list indicated that the instance is spending a significant amount of time in obtaining the locks.

·         Tune the SQL to optimize the number of blocks accessed by the application, thereby reducing the no of blocks it is requesting.









gc current block 2 –way

An instance requests authorization for a block to be accessed in the current mode to modify the block. The instance mastering the corresponding resource received the request.  The master has the current version of the block and sends the current copy of the block to the requestor via the Cache Fusion mechanism. This event indicates write/write contention. The appearance of gc current block 2 way in the Top 5 event section does not necessarily indicate performance issues. It merely indicated that the instances are accessing copies of data present in each other’s cache  and that the cache fusion mechanism is being used to transfer copies of the data among the instances. However if the average wait time for each event is very high, it might be impacting performance and needs further analysis.

·         Analyze the contention. Segments in the Current Blocks Received section of the AWR report should help to identify the top contentious objects.

·         Ensure the good database object design practices, database object layout, and space management practices are being followed for the top contentious objects.
In the next blog i will come out with a clear cut definitions and various types of RAC Wait Events.