Performance Implications With Complete Table/Mview Refresh

The issue
Some databases have middle size dimension (lookup) tables/mviews regularly updated. Some updates are complete or near complete table/mview refresh. Some queries need access those tables in full (full table scan, for example). The timing of the table complete refresh and the queries may cause very long running time for the queries and higher IO usages than usual. Here the middle size table usually has more than 1M records and more than hundreds of thousands of blocks.

Table/Mview Complete Refresh

There are several scenarios for table/mview complete refresh:
Using MERGE, but almost all existing rows are updated and some new rows are added (TAOEXT). TRUNCATE then INSERT so the content will be totally new (UAD EDWARD2 for POW tables) MVIEW complete refresh DELETE then INSERT (GENTCL)

Timing
The query starts after the complete refresh starts, but before the complete refresh commits. The complete refresh starts after the query starts, but some changes on the concerned table have been done before the query retrieves all the required data blocks. Oracle consistent read requires the query to reconstruct the table data before the changes.

Where to Read The Data?

If the refresh generates UNDO during MERGE or DELETE/INSERT, Oracle has to read UNDO blocks to reconstruct the original data. When Oracle reads a block from disk or buffer cache, it finds one or more rows might have been changed. For some cases, a row might have been changed multiple times (for example, a query last for several hours and the table is refreshed hourly). Oracle has to look for UNDO blocks to restore original row values. In worst case, each changed row points to a different UNDO block. Reading UNDO is slow:
Single block reads, while FTS on the target table can use multi block reads such as direct path reads or db file scattered reads. Need read much more blocks: possibly one block per row, or even worse because of the UNDO chains from multiple changes. Without UNDO in concern, one block read could fetch tens even hundreds of rows.

How about TRUNCATE Then INSERT?

No UNDO will be generated from TRUNCATE. Data has to be reconstructed from REDO logs. Reconstruct data from REDO is very expensive.

The Symptom
How can we know the slow query response time is caused by table complete refresh?
v$session or ASH shows significant waits on db file sequential read with row_wait_obj# or current_obj# as 0, and P1 (file number) points to UNDO table space. v$session or ASH shows significant waits on buffer busy waits or read by another session and the block class is UNDO segment (P3), when multiple queries runs at the same time. v$session or ASH shows significant waits on log file sequential read. Note log file sequential read is rarely seen on regular user session. So if you see such event from a query session, Oracle has to reconstruct the data from REDO because the data is not available from both buffer cache and data file.

Using v$sql_plan_monitor
Check column output_rows, physical_read_requests and physical_read_bytes, for the plan line related to reading of the concerned table. out_put_rows grows slowly, while physical_read_requests and physical_read_bytes grow much bigger than the size of the table self. For example, we should not really expect physical_read_requests grows bigger than the total blocks of the concerned table. (One exception is when we have chained or migrated rows for table with very long columns and rows, such as PRDYCRM S_ORG_EXT).

Other Indicates
Session Stats: v$sesstat
data blocks consistent reads - undo records applied redo k-bytes read total

Example
UAD job 3120, table POW.ORDER_LINE
POW.ORDER_LINE is refreshed by TRUNCATE then INSERT on hourly base. The table size is around 1.2M records and 34K blocks. Occasionally job 3120 runs for hours with log file sequential read. At one check with v$sql_plan_monitor, Oracle used 2.8 hours and more than 9M read requests to read less than 700K rows (10101 sec, 9,365,676 IO requests, 6,652,598MB of physical reads, for 687,352 rows). Session stats check at the same time: redo k-bytes read total 6,772,589,456

Usually log file sequential read is not seen on user sessions.

IO read request count is 8,455,341, and the size is 5,950,125MB

No row retrieved yet.

Example
GENTCL, MVIEW JOE.LINE_ALL complete refresh. MVIEW size: around 12M rows, 400K blocks. It also has 11 indexes. One SQL using JOE.LINE_ALL three times with one execution with FTS. The same query is executed from 7 colos with each colo sends two queries at the same time. The average execution time usually is 6 minutes. During the mview complete refresh time, average execution time is 102 minutes. Top waits: read by other session, db file sequential read and gc buffer busy acquire. We expected db file scattered read or direct path read. An interesting case here is, because there were 14 concurrent executions, and data mainly from UNDO, the total IO was much smaller than usual when all executions use direct path reads. Session Stats: majority of LIO is from UNDO.
consistent gets 11,032,957 data blocks consistent reads - undo records applied 9,768,377

Note object id is 0

More On MVIEW Refresh With Indexes Presented

Lesson learned from GENTCL mview JOE.ALL_LINE: 12M rows, 400K blocks, 11 indexes, from dblink (ruby). Complete refresh took more than 15 hours and failed (timed out at remote DB with session no activity). Complete mview refresh is DELETE then INSERT. With 11 indexes enabled, both DELETE and INSERT are very IO costly and slow. While DELETE of table data can be done in parallel, the index deletion is done by query coordinator (QC). With 11 indexes on, the majority of IO activity is db file sequential read. For example, after nearly 3 hours (10721 sec), with 11,914,475 rows read, the IO consumed by the table self is 373,982 requests with 5,969MB, with the indexes on, the deletion plan step used 1,948,652 requests and 15,223MB. The INSERT step is even worse. After more than 3 hours (12353 seconds), it handled 3,809,374 rows, with 5,737,116 read requests and 44821MB of data read. Recreate the MVIEW and add index later should be much faster.