Analysing Disk Performance Using Disk Watcher

Analyzing Disk Performance Using
Disk Watcher
Dawn May
dmmay@us.ibm.com
2010 IBM Corporation

IBM Power Systems
Agenda
What is Disk Watcher?
How do I run Disk Watcher?
What data is collected by Disk Watcher?
How do I analyze Disk Watcher data?
What documentation is available?
2 2010 IBM Corporation

IBM Power Systems
What is Disk Watcher?
New performance data collector

Available via PTF for releases V5R3 and 5.4
Included with 6.1 and 7.1
Near real-time diagnosis of disk performance issues
Data beyond that available in tools such as Work with Disk Status
(WRKDSKSTS), Work with System Status (WRKSYSSTS), and
Work with System Activity (WRKSYSACT)
These tools do provide information concerning: disk I/O, paging
rates, CPU usage, and temporary storage usage
Disk Watcher goes further by providing information related to the

Task Dispatching Elements (TDEs), objects, and programs
involved in the I/O operation

IBM Power Systems
Disk Watcher positioning
Collection Disk Performance

Services Watcher Explorer (PEX)
Sample data Statistics as well as Trace data
Collected every 5 Trace data Information
minutes (default) Simple interface to collected for every
No information collect I/O event
concerning specific Offers a balance Collection and
I/O operations between other tools analysis complex
less more
detail detail

IBM Power Systems
High level function overview

STRDW
Applications
QAPYDW
Disk Watcher Database
I/O request files
Machine Interface
Storage SLIC Disk Watcher

Management One Session per ASP
I/O Recorder
Disk 1 Per Configured Unit

IBM Power Systems
Potential uses of Disk Watcher
Evaluating the I/O performance of programs or workloads

Evaluating the performance of I/O operations on multi-path
disk units
Evaluating the performance of I/O queuing
Determining how performance may be improved by re-
spreading data across units
Determining the optimal placement
of devices, IOAs, or buses

IBM Power Systems
IBM Systems Director Navigator for i

Disk Watcher
Disk Watcher GUI features included with IBM i

Performance Tools, Manager Feature
From GO LICPGM:
5761PT1 *COMPATIBLE IBM Performance Tools for i5/OS

5761PT1 *COMPATIBLE Performance Tools - Manager Feature
5761PT1 *COMPATIBLE Performance Tools - Agent Feature
5761PT1 *COMPATIBLE Performance Tools - Job Watcher

IBM Power Systems
IBM Systems Director Navigator for i
IBM Systems Director Navigator for i is the Web console for managing IBM i
Much of the function that exists in System i Navigator, but with a browser user interface
Simply point your browser to http://systemname:2001

IBM Power Systems
Prerequisites
Users need to be authorized to use the Investigate Data and Collections

performance tasks
Include users on the QPMCCDATA authorization list
Edit Authorization List
Object . . . . . . . : QPMCCDATA Owner . . . . . . . : QSYS

Library . . . . . : QSYS Primary group . . . : *NONE
Type changes to current authorities, press Enter.
Object List
User Authority Mgt
*PUBLIC *EXCLUDE
QSYS *ALL X
PDI01 *USE
PDI02 *USE
PDI03 *USE
PDI04 *USE
PDI05 *USE
PDI06 *USE
PDI07 *USE
PDI08 *USE
PDI09 *USE
More...

IBM Power Systems
Prerequisites, continued
To see the Disk Watcher definitions through the GUI, users need authority to the
QAPYDWDFN file in QUSRSYS
File is shipped with *PUBLIC *EXCLUDE

IBM Power Systems
How do I run Disk Watcher on 6.1 or 7.1?

IBM Power Systems
IBM-Supplied Disk Watcher Definitions
Several pre-defined Disk Watcher definitions are available

Start by looking a statistics and use a trace only if very detailed analysis is required
QSTATS collects statistical data (summarized into counts and times)
QTRC collects trace data (very detailed I/O information)
QFULL collects both statistical and trace data

IBM Power Systems
Adding a Disk Watcher Definition

IBM Power Systems
Start Disk Watcher

IBM Power Systems
How Do I Run Disk Watcher with the Commands?
Start Disk Watcher (STRDW) command on V5R3 and 5.4
CL Commands also available in 6.1 and 7.1

Add Disk Watcher Definition (ADDDWDFN) to define the collection
Identifies the performance data that is to be collected
Remove Disk Watcher Definition (RMVDWDFN) to remove a

definition
Start Disk Watcher (STRDW) to start the collection
End Disk Watcher (ENDDW) to end the collection (optionally)

IBM Power Systems
Disk Watcher Definition Parameters

ASP number
ASP number for which data will be collected
ASP device name
Device name of the independent ASP for which data will be collected
Disk unit identifier
Limits data collection to a list of up to ten disk units in the selected ASP(s)
Storage pool identifier
Limits data collection to a specific storage pool
Type
Type of Disk Watcher collection to run
Collection interval
Interval between retrieval of disk I/O data
Considerations to avoid missing data

IBM Power Systems
Disk Watcher Definition

Conditional trace data parameters
These parameters only limit trace data

I/O type
Limits trace data collection to a particular type of I/O operation
Collection condition & Collection range
Limits trace data collected to I/Os which meet a specified condition
Condition may be based on Total response time, Queue time, or
Service time
Values are specified in microseconds

IBM Power Systems
Disk Watcher Definition

Additional parameters
Object information
Determines whether object and program information will be collected
Performance considerations
Force record write
Determines when disk I/O data records will be written to the database files
To file ASP threshold
Refers to the ASP where the Disk Watcher database files are located
The maximum percent of this ASP that can be consumed
System ASP threshold
The maximum percent of the system ASP that can be consumed

IBM Power Systems
Start Disk Watcher Parameters

Member / Collection
Name of the member which will be created in the QAPYDW database files
Library
Library where the database files will be created
Text description
Description associated with the database file members
Replace data
Whether to overwrite existing database file members with the specified name
Collection Interval
Can override what is specified on the definition
End collection
Determines when data collection will end
Multiple end options may be specified
Whichever occurs first will end the collection
Hardware File
Whether to collect the storage device resource information
This is necessary for viewing trace data via Investigate Data

IBM Power Systems
Managing Disk Watcher Collections

IBM Power Systems
What data is collected by Disk Watcher?
Data collected into a Created for

every
set of database files QAPYDWRUNI QAPYDWINTI
collection
All files have

QAPYDW prefix Created for
QAPYDWSTAT
*STATS
collection
Created for
n 1
QAPYDWTRC QAPYDWTDER
*TRACE
collection
n n
1 1
Optional for
QAPYDWPGMR QAPYDWOBJR
*TRACE
collection

IBM Power Systems
Files documented in infoCenter 5.4

IBM Power Systems
Disk Watcher Documentation in 6.1 or 7.1 Info Centers

IBM Power Systems
Format of Info Center data

IBM Power Systems
How do I analyze Disk Watcher data?

In PTF releases querying files is easiest way to analyze data
Can use the Investigate Data feature with prior release
disk watcher data
Look for trends in the data
Are a large number of I/O operations associated with a certain TDE,
object, or program/procedure?
Are long I/O operations associated with a certain hardware resource?
Is there a high level of page faulting is it associated with a certain
TDE, object, or program/procedure?

IBM Power Systems
Viewing V5R3 or 5.4 Disk Watcher data with the 6.1

Performance Data Investigator
Disk Watcher data collected on V5R3 or 5.4 releases can be

viewed with the 6.1 Performance Data Investigator
Use SAVOBJ to save the collected disk watcher database files into a save file
FTP the save file to the 6.1 system
Restore Collection capability on 6.1 Performance Tasks GUI
Or
Use the Restore Performance Collection (RSTPFRCOL) command to restore the data
on the V6R1 system
The type of collection will be *DWFILE
Data Collected on V5R3/5.4 must:

Include the QAPYDWHDWR
Set up an environment variable before starting Disk Watcher
ADDENVVAR QPY_DW_HDWR_OPT 1
Include Object Information of *RESIDENT or *ALL
OBJINF(*RESIDENT) or OBJINF(*ALL) on the STRDW command

IBM Power Systems
Investigating Disk Watcher Data

IBM Power Systems
Statistical Overviews
Disk Statistical Overview
Average response times and total I/Os for the entire collection
Good starting point to get an overview of the entire collection
Can view overviews by
Disk Pools
Disk Units
Disk Paths

IBM Power Systems
Disk Statistical Overview by Disk Unit

IBM Power Systems
Statistical Details
Statistical Details allow more precise analysis of the details of the statistical
Disk Watcher data
Table views only
Sort to find largest contributors
Can view details by
Disk Pools
Disk Units
Disk Paths

IBM Power Systems
Disk Statistical Details by Disk Unit

Response Time Groups

IBM Power Systems
Trace
Trace requires the hardware
table and object information to
be collected
Viewing of detailed Disk Watcher
trace data in table form
Metrics for every I/O operation in
the collection
Get into the details .
What object the I/O is for
The program/procedure
that did the I/O
and more
Use filtering / sorting to
focus in on interesting data

IBM Power Systems
Disk Watcher Database Files

You can display the data in any of the Disk Watcher
database files
All fields in the file are displayed

IBM Power Systems
Special considerations
Cannot run Disk Watcher when another Storage

Management function such as ASP Balance or Disk
Reorganization is active
Message CPFB511 will be issued which will indicate the active
function which is preventing Disk Watcher from starting
Storage management trace buffers use storage in the
machine pool
If not enough storage is allocated, message CPFB515 will be issued
which will indicate the amount of additional storage required
message includes ways to change the size of the machine pool

IBM Power Systems
Performance considerations
Specifying *NONE or *RESIDENT on the Object information

(OBJINF) parameter will reduce resource consumption
Resource consumption front-loaded when performing any type of
resolution (*RESIDENT or *ALL)
Balancing need for data and the amount of resources you are
willing to consume
On very large systems limiting the collection to a subset of ASPs or
Disk units will produce a more manageable amount of data and
consume fewer system resources (memory, CPU, etc.)
However, limiting collection may reduce ability to identify problem
areas
Avoid using very short interval time unless disk units are very
busy
60 seconds is a good starting point

IBM Power Systems
How do I get Disk Watcher?

Function available through a combination of i5/OS code, i5/OS MRI, and SLIC
PTFs
One i5/OS PTF in each release will bring in other PTFs as requisites; complete
help in the PTF cover letters
V5R4M0 SI24919
V5R3M0 SI24918
PTFs available for Disk Watcher fixes:

V5R3M0 SI29605
V5R4M0 SI29606
Disk Watcher data collector Included in the base operating system with 6.1 and
later
CL Commands
SI30482 should be loaded to ensure necessary hardware information is collected to view
trace data (6.1 release)
SI30723 should be loaded to ensure necessary object information is collected to view trace
data (6.1 release)
GUI is part of Performance Tools product, Manager Feature

IBM Power Systems
What documentation is available?
V5R3 and 5.4

PTF coverletters
Contains full command documentation for V5R3 / 5.4
Files documented in Information Center with Collection Services Files
6.1 and 7.1

Complete command help for the CL commands
Disk Watcher documentation in the Information Center
Disk Watcher files now documented in their own section

IBM Power Systems
Back Up and Reference Information

IBM Power Systems
QAPYDWRUNI
One record per collection, updated every interval
Collection information System information

Interval number Number of physical & virtual
Start and end times processors
File level System name
Collection size System serial number
End reason System type
Status of collection System model
Trace data condition flag Operating system version
Name of calling job
Current user of calling job
STRDW command string

IBM Power Systems
QAPYDWINTI
One record per interval
Interval information
Interval number
Interval start time
Interval end time
ASP count
Disk unit path count
Data missed
If set, then consider using smaller collection interval
Condition status

IBM Power Systems
QAPYSTATS
One record per device resource name per interval
Disk I/O statistics data
Interval number
Device resource name / ASP / Disk unit number
Data missed
Sector size
First and last I/O times
Minimum and maximum operation and queue times
I/O buckets
Sort I/O operations by total response time
Operation and queue times reported for each bucket
Bucket 1 = less than 100 microsecs
Bucket 2 = 100 to 199 microsecs
Bucket 13 = 204800 microsecs and greater

IBM Power Systems
QAPYDWTRC
One record per I/O operation
collected
Disk I/O trace data
Device resource name / ASP / Disk
unit number
Segment key
Key to join with QAPYDWOBJR file
Program or procedure key
Key to join with QAPYDWPGMR file
Task count
Key to join with QAPYDWTDER file
Deferred queue depth Pool number
I/O flags (multi-path, synchronous) Deferred queue time
I/O type (read / write) Start time of I/O operation
Disk subunit (if part of mirrored pair) End time of I/O operation

IBM Power Systems
QAPYDWTDER
One record per TDE which caused an I/O operation
Task Dispatching Element (TDE) resolution data
Task count
Key to join with QAPYDWTRC file
Unique identifier for the TDE, cannot be reused prior to IPL
Thread ID
Primary thread task count
TDE type
I = Initial (primary) thread
S = Secondary thread
T = Task
L = Lisenced Internal Code (LIC) thread
Qualified job name or task name
Current user

IBM Power Systems
QAPYDWPGMR
One record per program which caused an I/O operation
Program resolution data
Procedure key
Key to join with QAPYDWTRC
Object type
Object subtype
Library name
Program name
Module name
Procedure name

IBM Power Systems
QAPYDWOBJR
One record per object on which an I/O operation was performed
Object resolution data
Segment key
Key to join with QAPYDWTRC
Segment type
Object type
Object subtype
Object name
Context name (library)
Object ASP
IFS path name
Object type/subtype details available here:
http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/rbam6/rbam6objecttypes.htm
Common type/subtype combos not documented on the webpage above:
Hex 0B90 = Database member
Hex 0C90 = Index member

IBM Power Systems
QAPYDWHDWR
Contains output from Display Hardware Resources
(DSPHDWRSC) command
Hardware data captured at beginning of collection
V5R3/V5R4: Triggered by setting an environment variable
ADDENVVAR QPY_DW_HDWR_OPT 1
6.1: STRDW parameter Hardware file HDWF
PTF SI30482 to change default to *YES

IBM Power Systems
Recent articles on understanding disk performance
A New Way to Look at Disk Performance

http://www.ibmsystemsmag.com/i5/may07/administrator/15631p1.aspx
Using Wait State Accounting to Determine Disk Performance

http://www.systeminetwork.com/artarchive/20700/index.html
Understanding Disk Performance, Part 2: Disk Operation on i5/OS

http://www.systeminetwork.com/artarchive/20870/Understanding_Disk_Performance__Part_
A Look at System I Integrated DASD Configuration and Performance under i5/OS

Redpaper REDP-3919-00
End-to-End Performance Management on IBM i a Redbooks publication

http://www.redbooks.ibm.com/redbooks/pdfs/sg247808.pdf

IBM Power Systems
Special notices
This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in
other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM
offerings available in your area.
Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions
on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give
you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY
10504-1785 USA.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives
only.
The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or
guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the
results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations
and conditions.
IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions
worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment
type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal
without notice.
IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.
All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this
document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-
available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document
should verify the applicable data for their specific environment.
Revised September 26, 2006

IBM Power Systems
Special notices (cont.)

IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 6 (logo), AS/400, Active Memory, BladeCenter, Blue Gene, CacheFlow, ClusterProven, DB2, ESCON, i5/OS, i5/OS
(logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC,
pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, AIX 5L, Chiphopper,
Chipkill, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System, , GPFS, HACMP,
HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA,
Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software
(logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER7, pureScale, System i, System p, System p5, System Storage, System z,
Tivoli Enterprise, TME 10, TurboCore, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a
trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark
information" at www.ibm.com/legal/copytrade.shtml
The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.
UNIX is a registered trademark of The Open Group in the United States, other countries or both.
Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.
Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.
Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.
AMD Opteron is a trademark of Advanced Micro Devices, Inc.
Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both.
TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).
SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are
trademarks of the Standard Performance Evaluation Corp (SPEC).
NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.
AltiVec is a trademark of Freescale Semiconductor, Inc.
Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc.
InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.
Other company, product and service names may be trademarks or service marks of others.
Revised February 9, 2010

Analysing Disk Performance Using Disk Watcher

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Analysing Disk Performance Using Disk Watcher

Uploaded by

Copyright:

Available Formats

Analyzing Disk Performance Using

2010 IBM Corporation

2 2010 IBM Corporation

What is Disk Watcher?

New performance data collector

Disk Watcher goes further by providing information related to the

3 2010 IBM Corporation

Disk Watcher positioning

Collection Disk Performance

4 2010 IBM Corporation

High level function overview

Storage SLIC Disk Watcher

5 2010 IBM Corporation

Potential uses of Disk Watcher

Evaluating the I/O performance of programs or workloads

6 2010 IBM Corporation

IBM Systems Director Navigator for i

Disk Watcher GUI features included with IBM i

5761PT1 *COMPATIBLE IBM Performance Tools for i5/OS

7 2010 IBM Corporation

IBM Systems Director Navigator for i

Simply point your browser to http://systemname:2001

8 2010 IBM Corporation

Users need to be authorized to use the Investigate Data and Collections

Include users on the QPMCCDATA authorization list

Edit Authorization List

Object . . . . . . . : QPMCCDATA Owner . . . . . . . : QSYS

Type changes to current authorities, press Enter.

9 2010 IBM Corporation

10 2010 IBM Corporation

How do I run Disk Watcher on 6.1 or 7.1?

11 2010 IBM Corporation

IBM-Supplied Disk Watcher Definitions

Several pre-defined Disk Watcher definitions are available

12 2010 IBM Corporation

Adding a Disk Watcher Definition

13 2010 IBM Corporation

Start Disk Watcher

14 2010 IBM Corporation

How Do I Run Disk Watcher with the Commands?

Start Disk Watcher (STRDW) command on V5R3 and 5.4

CL Commands also available in 6.1 and 7.1

Remove Disk Watcher Definition (RMVDWDFN) to remove a

Start Disk Watcher (STRDW) to start the collection

End Disk Watcher (ENDDW) to end the collection (optionally)

15 2010 IBM Corporation

Disk Watcher Definition Parameters

16 2010 IBM Corporation

Disk Watcher Definition

These parameters only limit trace data

17 2010 IBM Corporation

Disk Watcher Definition

18 2010 IBM Corporation

Start Disk Watcher Parameters

19 2010 IBM Corporation

Managing Disk Watcher Collections

20 2010 IBM Corporation

What data is collected by Disk Watcher?

Data collected into a Created for

All files have

21 2010 IBM Corporation

Files documented in infoCenter 5.4

22 2010 IBM Corporation

Disk Watcher Documentation in 6.1 or 7.1 Info Centers

Specifying NONE or RESIDENT on the Object information