Session id: 1137

Oracle Flashback: Techniques For Recovering From Human Error

Timothy Chien Senior Product Manager Oracle Corporation Wayne Cho Senior Product Manager Oracle Corporation Greg Penk VP of Data Administration Banknorth Group, Inc. Joshua Ort Oracle Database Administrator ReserveAmerica, Inc.

Agenda
Human Errors Flashback Technologies

Pinpoint Error Quick Repair Banknorth Group, Inc. ReserveAmerica, Inc. Oracle Collaboration Suite

Customer Profiles

Q&A

Human Error
Estimated to be the biggest single cause of downtime Need to quickly determine what happened and fix it

Localized damage Needs surgical detection and repair Example removed wrong person named Smith Widespread damage Requires drastic action to avoid long downtime Example batch job deletes this months orders

Human Errors Other Downtime

Analysis and correction using traditional recovery is slow and complex

Restore database to point in time and extract data

Oracle Database 10g is a breakthrough release for human error correction

Flashback Query

A Time Machine for Your Data

Flashback Query allows viewing data as it was before a mistake

Query data at a time of your choosing Standard SQL interface simplifies deployment Self-service means faster, cheaper, and easier Flashback Query is a fast operation to enable self service

Mistake
Delete from Emp where Ename=Smith;

Correction
Insert into Emp select * from Emp AS OF yesterday where Ename=Smith;

Flashback Time Navigation

Flashback Query Query all data at point in time
Select * from Emp AS OF 2:00 P.M. where

Tx 3 Tx 2

Flashback Versions Query See all versions of a row between two times See transactions that changed the row
Select * from Emp VERSIONS BETWEEN 2:00 PM and 3:00 PM where

Tx 1

Flashback Transaction Query See all changes made by a transaction

Select * from FLASHBACK_TRANSACTION_ QUERY where xid = 000200030000002D;

Flashback Database
A new strategy for point in time recovery
Eliminate the need to restore a whole database backup Integrated seamlessly with RMAN Think of it as a continuous backup Restores just changed blocks

Disk Write New Block Version

Old Block Version

Replay log to restore DB to time Its fast - recover in minutes, not hours Its easy - single command restore Flashback Database to 2:05 PM

Data Files

Flashback Log

Rewind button for the Database

Flashback Database versus Classic Point-In-Time Recovery

Recovery is 100 times faster with Flashback
700
627

600
Time (minutes)

500 400 300 200 100 0 10

114 51 2 3 4 6 250

Restore Recovery Flashback

100

1,000

10,000

Database Size (GB)

Flashback Error Correction

Database
Customer

Recovery at all levels Database Level

Flashback Database restores the whole database to time Uses Flashback Logs Flashback Table restores rows in a set of tables to time Uses UNDO in database Flashback Drop restores a dropped table or a index Recycle bin for DROPs Flashback Query restores rows to time

Table Level

Order

Row Level

Banknorth
Capitalizing on Flashback Technologies for Troubleshooting and Reporting
Greg Penk VP of Data Administration Banknorth Group, Inc.

Banknorth Group, Inc.

Bank and financial holding company, with consolidated assets of over $29 billion In 2004,voted Best Managed Bank by Forbes Magazine Over 360 branch locations throughout northern New England and upstate New York

Banknorth Oracle Database 10g Infrastructure

OLTP Database providing back-end for Oncore Teller System Over 1500 users will access the database (Over 360 bank branches) Database resides on a 3-node HP-UX RAC Recovery Manager (RMAN) disk-based backups to the Flash Recovery Area Planning Data Guard environment

Requirements at Banknorth
100% Accuracy of data and reporting

Any irregularities must be identified quickly and precisely ---- or no one leaves the building Outages are unacceptable Flashback Database enabled for 24 hours in Data Guard environment

Recovery Time Objectives (RTO) = NOW

Guaranteed Data Availability

Benefits of Flashback Technologies

Provides continuous snapshot capabilities

Online troubleshooting of any data trend irregularities Finer granularity for reports Compare and correct data while database remains online Same benefits in Data Guard environment Flashback logging enabled for 24 hours

Fast recovery from logical errors

Easily undo batch job, schema, upgrade changes Creates no adverse performance overhead Maintains same security as database

Flashback Query Key for Problem Diagnosis

Leverages current month-end reports providing as of time capabilities

Transaction Volume Report Network Outages Impact Report Saves hours of time reconciling transactions, replacing manual method

Granular troubleshooting ability

Undo Retention configured for 48 hours

Summary
Flashback Technologies enable quick reporting and troubleshooting

Flashback Query for generating mid-day transaction reports Data correction ability at the transaction, row, and table level

Flashback Database enables quick point-in-time recovery for last 24 hours

Protects against user error and logical corruption on primary and standby databases

Snapshot capabilities without having to purchase additional software

ReserveAmerica
Protecting Data and Minimizing Recovery Effort

Joshua Ort Oracle Database Administrator ReserveAmerica, Inc.

ReserveAmerica
ReserveAmerica "The #1 Access Point for Outdoor Recreation"
North America's leading Camping Reservation, Campground Management and Tour/Ticketing Solutions Provider Over 3.5 million camping reservations per year Services over 2.4 million members Inventory includes over 150,000 campsites in 44 states and 1 province

ReserveAmerica Infrastructure
Oracle Database 10g in production since April 2004. Sun V880 on SPARC III with 8x1200 MHz CPUs and 16 GB RAM 4 TB SAN Storage 1000+ users Heavy read OLTP system

Easy to Enable Flashback Database

Identify Location Identify Size Enable Flashback Identify Retention

Click Apply

Quick Recovery/ Extended Data Protection

Ability to recover data without resorting to a backup Highly granular point-in-time or transactionbased recovery

More Troubleshooting / QA Environment Options

Use of Flashback Version Query by Product Support and Development
Troubleshoot bad data

Use of Flashback Database by QA

Failed deployments Scrub, clean QA database

ReserveAmerica : Flashback in Action

Custom ddl_log table audits all DDL activity Table is inadvertently dropped Next DDL attempt reports error Flashback Drop used to recover data No need to involve tapes or backups Quick and accurate recovery

Summary
Fast Recovery with Flashback Database

Lessens the need to restore from backups Flashback Query & Versions Query Product development can target bad data QA can undo results of old testing to prepare for new testing

Detailed row-by-row recovery

Flashback useful to various groups

Customer Profile:

Oracle Email -Flashback Email Recovery

Email server built on Oracle Database Incorporates end-user correction capabilities Flashback Email Recovery

Utilizes Automatic Undo Management Utilizes Flashback Query

Integrated with Microsoft Outlook

Enable Flashback Mail Recovery Oracle Email Admin GUI

11:59am Delete messages

12:01pm Messages expunged

12:10pm Recover messages

Enter Flashback Query Criteria

Messages Recovered

Recovered Messages in Inbox

Collaboration Suite Using Built in Database Functionality

Flashback Query empowers application developers to build self-service, error correction functionality

Past versions of data can be viewed and recreated by just using AS OF clause

Oracle Email empowers end-users to recover deleted messages

Leverages Flashback Query to restore deleted messages Deleted messages kept for UNDO_RETENTION target period Integrated with Outlook Client via Outlook Connector

New Oracle Database 10g Release 2 Features

Flashback Database through RESETLOGS

Allows a physical standby database that is temporarily open readwrite to later flashback the changes and be converted back to a physical standby database. Allows flashback of the primary and standby databases to an SCN or a point in time prior to the switchover operation. Provide a user defined name that is associated with a database point in time used in conjunction with Flashback Database, Flashback Table and Recovery Manager (RMAN) Two types of Restore Points, based on recovery needs: Named Restore Point - User defined alias related to a database point in time Guaranteed Restore Point - Ensures flashback logs are kept until the restore point is used or deleted

Restore Point

Flashback Technologies Summary

Comprehensive error correction capabilities

Accessed via simple SQL Easily troubleshoot row, table, transaction Recover dropped tables Quickly repair errors in row, table, database-wide

Time to correct error equals time to make it! Correction Time = Error Time + f(DB_SIZE)

Flashback Technologies Summary

Customer Successes

Banknorth Easily extends branch reporting solution Protect against error & corruption with little downtime ReserveAmerica Troubleshoot bad data Reuse QA instances by rewinding changes Oracle Email Leverages Flashback Query for recovering deleted messages Empowers end-users, integrated with Outlook Client

QUESTIONS ANSWERS