AWR TOP 5 Timed Event Analysis

enq: TX - row lock contention
Enqueues are locks that coordinate access to database resources. This event indicates that the session is waiting for a lock that is held by another session.

The name of the enqueue is included as part of the wait event name, in the form enq: enqueue_type - related_details. In some cases, the same enqueue type can be held for different purposes, such as the following related TX types:
enq: TX - allocate ITL entry
enq: TX - contention
enq: TX - index contention
enq: TX - row lock contention

These are acquired exclusive when a transaction initiates its first change and held until the transaction does a COMMIT or ROLLBACK.

• Waits for TX in mode 6: occurs when a session is waiting for a row level lock that is already held by another session. This occurs when one user is updating or deleting a row, which another session wishes to update or delete. This type of TX enqueue wait corresponds to the wait event enq: TX - row lock contention.

•Waits for TX in mode 4 can also occur if a session is waiting due to potential duplicates in UNIQUE index. If two sessions try to insert the same key value the second session has to wait to see if an ORA-0001 should be raised or not. This type of TX enqueue wait corresponds to the wait event enq: TX - row lock contention.

Problem: The performance of overall database will be effected with this kind of Wait Event and DB Time in AWR will be very high.

Solution:The solution is to have the first session already holding the lock perform a COMMIT or ROLLBACK. In our situation we have seen extensive usage of select for update statements. Reducing these we have reduced a lot of wait events. Also we have found couple of procedures where commit is being kept at the end rather immediately after the update statement. After placing the commits closure to the update statement the wait event has drastically reduced.

AWR Timed Event Analysis

Top 5 Timed Event --- direct path read




direct path read and direct path read temp
When a session is reading buffers from disk directly into the PGA (opposed to the buffer cache in SGA), it waits on this event. If the I/O subsystem does not support asynchronous I/Os, then each wait corresponds to a physical read request.
If the I/O subsystem supports asynchronous I/O, then the process is able to overlap issuing read requests with processing the blocks already existing in the PGA. When the process attempts to access a block in the PGA that has not yet been read from disk, it then issues a wait call and updates the statistics for this event. Hence, the number of waits is not necessarily the same as the number of read requests (unlike db file scattered read and db file sequential read).

Causes

This happens in the following situations:
•The sorts are too large to fit in memory and some of the sort data is written out directly to disk. This data is later read back in, using direct reads.

•Parallel slaves are used for scanning data.

•The server process is processing buffers faster than the I/O system can return the buffers. This can indicate an overloaded I/O system.

Actions
This wait event can be because of large sorts to disk or full table scans by parallel slaves. This is the biggest wait for large data warehouse sites. However, if the workload is not a DSS workload, then examine why this is happening.
In the above scenario we identified a query using parallel slaves performing multiple full scans on couple of tables having 20 million records each. If tables are defined with a high degree of parallelism, then this could skew the optimizer to use full table scans with parallel slaves and result into direct path read wait event on the top.

AWR / TOP 5 Timed Event

Top 5 Timed Event ---SGA: allocation forcing component growth

Problem: Database Hangs/ Some of the jobs like export fails.




Solution : The waits for "SGA: allocation forcing component growth" indicates that SGA is inadequate. It looks like there is no enough memory available in SGA for the growth of components. SGA has to be increased or disable the SGA automation by setting SGA_TARGET=0. I will write more about the problems of SGA Resizing as there are many bugs related to the functionility.One has to query the view V$Sga_Resize_Ops to know how many times Oracle Resizes are happening.

Oracle 10g Top 5 Timed Events - backup timed events

In the TOP 5 Times Events, i have highlighted couple of events related to backup job. IMM OP wait event falls under the other category but related to the RMAN backup. The other one you see is Backup : sbtbackup falls under the administrative category also realted to the RMAN. These wait events will occur when there is RMAN backup running on the server.

imm op

Meaning: Waiting for an IMMEDIATE I/O request to a slave process to end.The event "imm op" may occur if I/O slaves are used,

that is, if parameters such as DBWR_IO_SLAVES or BACKUP_TAPE_IO_SLAVES are used.

BACKUP_TAPE_IO_SLAVES is set to TRUE in the case of a RMAN backup if dedicated I/O slave processes are to copy backup write

processes to tape and not the Oracle shadow processes. In this case, "imm op" waits only affect the backup runtime, but not the live system.

Backup: sbtbackup

Meaning: Wait situations during RMAN backup. The wait events that begin with "Backup: sbt" are associated with wait situations during a RMAN backup. This means they only affect the backup runtime.

One has to be careful with these events especially during the peak time, they should not be in the TOP 5. If they are found in the peak time, one has to look at scheduling the backup during the non peak time and also these will occur when RMAN is backing up the database directly to the TAPE instead to a disk. As backup is a heavy DISK and CPU intensive job, it has to be done during the non peak time only.

AWR Analysis - Oracle "Wait Class" from 10g

Top 5 Timed Events and Wait Class Categories from 10g

Oracle 10g includes many enhancements to the wait event interfaces. Among them the introduction of wait classes has made the life of the DBA a lot easier in identifying the problem. The color coding given to these wait classes helps the DBA's to identify and correlate the problem to a particular category. And also the same wait class concept has been used in AWR reports as well as in many screens of the enterprise manager for example in monitoring the TOP Activity under Performance TAB Of Enterprise Manager.

In Oracle 10g wait events are classified into categories which can help the DBA to more easily determine the likely cause of the problem. The categories are:
Administrative, Idle, Application , Network , Cluster Scheduler , Commit , System I/O , Concurrency , User I/O , Configuration , Other

DBA can identify from a distance by looking at the color of the activity in the enterprise manager.

The first diagram is the TOP Activity of Enterprise Manager showing the activity in colors and right side you can see the color coding against each category.The second diagram shows the AWR report TOP 5 Times Events and Wait Class Statistics.

Happy tuning.

AWR / Statspack Analysis

Problem : Performance and High CPU

Solution / Interpretation : The TOP Event log file sync This wait event is seen in most of the high transactional databases where more frequent commits happens. Let us look deep into the concept and then see how we can reduce that, definitely we cannot avoid it.

At commit time, process creates a redo record [ containing commit op codes] and copies that redo record in to log buffer. Then that process signals LGWR to write contents of log buffer. LGWR writes from log buffer to log file and signals user process back completing a commit. Commit is considered successful after LGWR write is successful.

Commit is not complete until LGWR writes log buffers including commit redo recodes to log files. In a nutshell, after posting LGWR to write, user or background processes waits for LGWR to signal back with 1 sec timeout. User process charges this wait time as ‘log file sync’ event.


LGWR is unable to complete writes fast enough for one of the below reasons..

1)Disk I/O performance to log files is not good enough.

2)LGWR is starving for CPU resource. If the server is very busy, then LGWR can starve for CPU too. This will lead to slower response from LGWR, increasing ‘log file sync’ waits.

3)LGWR is unable to complete writes fast enough due to file system or unix buffer cache limitations.

4)LGWR is unable to post the processes fast enough, due to excessive commits. It is quite possible that there is no starvation for cpu or memory and I/O performance is decent enough. Still, if there are excessive commits, then LGWR has to perform many writes/semctl calls and this can increase ‘log file sync’ waits. This can also result in sharp increase in redo wastage’ statistics’.

This event is also the major contributor for overall CPU usage.

In the above scenario, we cannot avoid the COMMITS as the application requires it. What i have done is reduced the CPU contention so that LGWR does not wait for the lack of CPU.

The other aspect is to see where these online redo log files are created. We have made raw devices for these files so that the writing happened fast.

Also there is a misconception that increasing the redo log buffer will solve the problem , but this is not the case as if you the see fundamentals of log buffer flush is for every commit or when 1/3 is full or 1 MB of redo is generated. In this case for every commit there is redo buffer flush happenes so no need to have the redo buffer more than 3MB.

Also try to place the major hot tables and indexes separately on to multiple disks to increase the disk i/o.

Team ORAKHOJ
Get free AWR / Statspack Analysis by sending mail to tuning@orakhoj.com

AWR / Statspack Analysis

Problem : Performance and CPU





Solution : The 3rd TOP 5 Wait Event 'log file switch' indicates there is a problem with Checkpoint and Switching. It directly relates to a problem with the size of the online redo log files. According to the statpack we found that there are 70 + switches perfoemd by the database per hour and also a Delete Statement was found as the TOP CPU Consuming Statement, It clearly states that there is a bottleneck during the delete operation as much redo will be generated and because of small online redo log files, switched are happening frequently which in turn making the delete statement wait for log switch to perform. Also the delete statement is using the Index which is the cuase for the first 'db file sequential read ' wait event.

Happy Tuning
Team ORAKHOJ -- Get Free AWR / STATSPACK Analysis from ORAKHOJ tuning@orakhoj.com

Statspack/ AWR Analysis

Oracle Release 10.2.0.3
Problem CPU and Performance

Top 5 Wait Events

You can find 'db file sequential read' and 'read by other session' apart from CPU as TOP 5Events. Based on the 'db file sequential read' it is evident that the system is using most of the indexes.

Once we go to the other wait event 'read by other session' which is nothing but buffer busy waits where information is requested from the database, Oracle will first read the data from disk into the database buffer cache. If two or more sessions request the same information, the first session will read the data into the buffer cache while other sessions wait; e.g. many sessions scanning the same index or performing full-table scans on the same table. Tuning this issue is a matter of finding and eliminating this contention.

So to figure which object is causing the contention, you can look at which tablespace/datafile has got buffer waits and which object is falling under the Segments by Physical Reads and Logical Reads. Based out which we have identified an index which is the cause for this contention. The same has been rebuild on to a different tablespace to reduce the contention.

You can get free tuning advices from our DB experts by simply sending mail to tuning@orakhoj.com
-------------------------------------------------------------------------------

Statspack/ AWR Analysis

Database Version = 10.2.0.3



Problem : High CPU consumption.

Solution : Based on the above top 5 events one can cleary see that db file scattered read is high event after the CPU, which means that the application is performing full tablescans. Based on this one has to look at SQL ordered by CPU session of the report to find out which queries on performing full table scan. Based on the index recommendations the above problem has been sorted out.

Team ORAKHOJ

You can also get free tuning analysis from our database experts. Please send a mail to tuning@orakhoj.com

Free Statspack/AWR Analysis

Dear Oracle Professional,
Free Tuning Advisory Service has been started by ORAKHOJ- Exclusive Search Engine for Oracle Professionals. You can get free analysis & tuning recommendations from our tuning experts. Mail your AWR (html format) or Statspack reports to tuning@orakhoj.com. AWR / Statspack can be of RAC / Non RAC databases.
For best results, please collect AWR or Statspack report for 30 to 60 minutes duration.The information shared will be kept confidential including the mailer information.

Thank you for your patronage and keep using ORAKHOJ. - The ORAKHOJ Team"
www.orakhoj.com