How to Install and configure OSWatcher Black

Box (OSWbb)
The Oracle OSWatcher Black Box (OSWbb) product is a collection of shell scripts intended to
collect and archive operating system and network metrics to aid in diagnosing performance
issues. OSWbb operates as a set of background processes on the server and gathers data on a
regular basis, invoking such UNIX utilities as vmstat, netstat, iostat, top, and others.

Beginning with release 4.0.0, OSWbb includes a built-in analyzer called OSWbba, which analyzes
the data that OSWbb collects. It provides information on system slowdowns, hangs, and other
performance problems. It also provides the ability to graph vmstat and iostat data.

OSWbb is particularly useful for Oracle Real Application Clusters (RAC) and Oracle Grid
Infrastructure configurations. OSWbb is included in the RAC-DDT (Diagnostic Data Tool) script file,
but is not installed by RAC-DDT.

OSWatcher is certified to run on the following platforms:

 AIX
 Solaris
 HP-UX
 Linux

Installing OSWbb

You must install OSWbb on each node where data is to be collected. For RAC or shared disk
systems, each node requires an OSWbb installation into a unique directory (for example,
/oswbb_node1 and /oswbb_node2). OSWatcher can be downloade through MOS Doc ID
301137.1 as a tar file. Although the installation and configuration procedure of oswbb for all the
Operating systems is almost the same. The post uses RHEL 7 system for the purpose of
demonstration.

After downloading the TAR file, copy the file to the directory where OSWbb is to be installed and
run the following command:

# tar xvf oswbb.tar

A directory named oswbb is created which houses all the files associated with oswbb. OSWatcher
is now installed. If you list the files/directories inside the oswbb directory :

# ls
analysis call_uptime.sh Exampleprivate.net ltop.sh OSWatcherFM.sh oswib.sh oswsub.sh src
tarupfiles.sh xtop.sh
call_du.sh docs ifconfigsub.sh mpsub.sh OSWatcher.sh oswnet.sh profile
startOSWbb.sh tar_up_partial_archive.sh
call_sar.sh Example_extras.txt iosub.sh nfssub.sh oswbba.jar oswrds.sh psmemsub.sh
stopOSWbb.sh vmsub.sh
Starting OSWbb

To start the OSWbb utility, execute the startOSWbb.sh shell script. The startOSWbb.sh script
accepts two optional arguments that control the frequency (in seconds) that data is collected and
the number of hours worth of data to archive. If you do not enter any arguments, the script runs
with default values of 30 and 48, meaning collect data every 30 seconds and store the last 48
hours of data in archive files.

The following example starts the tool and collects data at 60-second intervals and logs the last 10
hours of data to archive files. Some of the output produced when starting the tool is shown:

# ./startOSWbb.sh 60 10
Setting the archive log directory to/data/oswbb/archive

Testing for discovery of OS Utilities...

VMSTAT found on your system.
IOSTAT found on your system.
MPSTAT found on your system.
IFCONFIG found on your system.
NETSTAT found on your system.
TOP found on your system.

......

Starting OSWatcher v7.3.3 on Thu Nov 10 17:48:51 IST 2016

With SnapshotInterval = 60
With ArchiveInterval = 10

.....

Data is stored in directory: /data/oswbb/archive

Starting Data Collection...

oswbb heartbeat:Thu Nov 10 17:48:56 IST 2016

oswbb heartbeat:Thu Nov 10 17:49:56 IST 2016
....
OSWbb started successfully, the discovery process completed, and data collection begins. You can
exit out of the session, once the data collection starts.

Stopping OSWbb

To stop OSWbb, execute the stopOSWbb.sh shell script. This terminates all processes associated
with OSWbb and is the normal, graceful mechanism for stopping the tool.
# ./stopOSWbb.sh
Viewing the data collected

OSWatcher invokes these distinct operating system utilities, each as a distinct background
process, as data collectors. These utilities will be supported, or their equivalents, as available for
each supported target platform.

 ps
 top
 ifconfig
 mpstat
 iostat
 netstat
 traceroute
 vmstat
 meminfo (Linux Only)
 slabinfo (Linux Only)

Use the cd command to change to the archive directory, and then use ls to view the contents of
the directory. The archive directory is created when OSWbb is started for the first time. The
directory contains 10 subdirectories, one for each data collector.

# cd archive/
# ls -l
total 40
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswifconfig
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswiostat
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswmeminfo
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswmpstat
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswnetstat
drwxr-xr-x. 2 root root 4096 Nov 10 17:48 oswprvtnet
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswps
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswslabinfo
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswtop
drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswvmstat
To check a sample, we will cd into a directory oswiostat and view the content of one of the files :

# cd oswiostat/
# ls -l
total 20
-rw-r--r--. 1 root root 12067 Nov 10 17:59 localhost.localdomain_iostat_16.11.10.1700.dat
-rw-r--r--. 1 root root 7047 Nov 10 18:06 localhost.localdomain_iostat_16.11.10.1800.dat
# cat localhost.localdomain_iostat_16.11.10.1700.dat
Linux OSWbb v7.3.3
zzz ***Thu Nov 10 17:48:56 IST 2016
avg-cpu: %user %nice %system %iowait %steal %idle
2.04 0.00 2.04 2.04 0.00 93.88

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await
svctm %util
sda 0.00 0.00 1.02 0.00 16.33 0.00 32.00 0.03 28.00 28.00 0.00 28.00 2.86
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.02 0.00 16.33 0.00 32.00 0.03 28.00 28.00 0.00 28.00 2.86
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

zzz ***Thu Nov 10 17:49:56 IST 2016

avg-cpu: %user %nice %system %iowait %steal %idle
2.00 0.00 2.00 0.00 0.00 96.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await
svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
As you can see iostat command output is collected at predefined interval. The historical archive
data comes handy when you want to gauge the load on the server for a period of time.

Uninstalling oswbb

To de-install OSWatcher issue the following command on the oswbb directory.

# rm -rf oswbb

How does it work ?

OSWatcher invoke OS utilities like vmstat, netstat, iostat, etc. by creating a “Data Collectors” for each of
them available on the system. The “Data Collectors” works as background processes to collect
periodically the data provided by these different OS utilities.
Once collected, all the statistics are stored inside a common destination (archive directory).

Below is the content of the archive directory. As you can see there is a dedicated folder for each type of
OS statistics collected :

oracle@srvtestoel7:/u01/app/oracle/product/oswbb/archive/ [JOCDB1] ll
total 36
-rw-r--r-- 1 oracle oinstall 1835 28 sept. 16:55 heartbeat
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswifconfig
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswiostat
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswmeminfo
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswmpstat
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswnetstat
drwxr-xr-x 2 oracle oinstall 6 23 sept. 09:18 oswprvtnet
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswps
drwxr-xr-x 2 oracle oinstall 6 23 sept. 09:18 oswslabinfo
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswtop
drwxr-xr-x 2 oracle oinstall 4096 28 sept. 16:52 oswvmstat

Downloading
You can download OSWatcher from My Oracle Support – Doc ID 301137.1 (.tar file – 6Mb)

Installing
To install OSWatcher, you simply have to untar the downloaded file :
$ tar -xvf oswbb733.tar
All necessary files are stored in the oswbb folder.

Uninstalling
To remove OSWatcher from your server, you only have to :
– Stop all OSWatcher running processes
– Delete the oswbb folder

Starting
$ nohup ./OSWatcher.sh P1 P2 P3 P4

Parameters
– P1 = snapshot interval in seconds (default : 30 seconds)
– P2 = number of hours of archive data to store (default : 48 hours)
– P3 = name of a compress utility to compress each file automatically (default : none)
– P4 = alternate location to store the archive directory (default : oswbb/archive)
You can also set the UNIX environment variable oswbb_ARCHIVE_DEST to specify a non-default location.

Startup steps
Starting OSWatcher involve 4 steps :
1. Check parameters
$ ./OSWatcher.sh 60 24 gzip /tmp/oswbb/archive
Info...Zip option IS specified.
Info...OSW will use gzip to compress files.
...
2. Discover OS utilities
Testing for discovery of OS Utilities...
VMSTAT found on your system.
IOSTAT found on your system.
MPSTAT found on your system.
...
...
Discovery completed.
3. Discover CPU count
Testing for discovery of OS CPU COUNT
oswbb is looking for the CPU COUNT on your system
CPU COUNT will be used by oswbba to automatically look for cpu problems
CPU COUNT found on your system.
CPU COUNT = 1
 4. Data collection
Data is stored in directory: /tmp/oswbb/archive
Starting Data Collection...
oswbb heartbeat:mar. sept. 13 22:03:33 CEST 2016
oswbb heartbeat:mar. sept. 13 22:04:33 CEST 2016
oswbb heartbeat:mar. sept. 13 22:05:33 CEST 2016

Check if OSWBB is running

$ ps -ef | grep OSWatcher | grep -v grep
oracle 8130 1 0 13:47 pts/0 00:00:33 /bin/sh ./OSWatcher.sh 5 48
oracle 8188 8130 0 13:47 pts/0 00:00:00 /bin/sh ./OSWatcherFM.sh 48 /u01/app/oracle/product/oswbb/archive

The OSWatcherFM.sh process is the file manager who delete collected statitstics once they have reached
their retention.

Stopping
Run the stopOSWbb.sh to stop all OSWatcher processes

$ ./stopOSWbb.sh

Configure automatic startup

Oracle provide a RPM package to configure auto-start of OSWatcher when the system starts.
You can download it here : My Oracle Support – Doc ID 580513.1

Once downloaded, install the package (as root) :

$ rpm -ihv oswbb-service-7.2.0-1.noarch.rpm

Preparing... ######################################### [100%]
1:oswbb-service ######################################### [100%]

You can adapt the following values in /usr/libexec/oswbb-service/oswbb-helper to define the parameters
with which OSWatcher will auto-starts :

OSW_HOME='/u01/app/oracle/product/oswbb/'
OSW_INTERVAL='10'
OSW_RETENTION='48'
OSW_USER='oracle'
OSW_COMPRESSION='gzip'
OSW_ARCHIVE='archive'

Start the service :

$ service oswbb start

Starting oswbb (via systemctl): [ OK ]

Check the service :

$ service oswbb status

OSWatcher is running.

Stop the service :

$ service oswbb stop

Stopping oswbb (via systemctl): Warning: Unit file of oswbb.service changed on disk, 'systemctl daemon-reload'
recommended.
[ OK ]
Enable the service when the system start :

$/sbin/chkconfig oswbb on

Systemd commands (Linux 7) :

$ systemctl stop oswbb.service

$ systemctl start oswbb.service
$ systemctl status oswbb.service
$ systemctl enable oswbb.service

Inside the archive directory, one dedicated folder is created by type of collected statistics :

oracle@srvtestoel7:/u01/app/oracle/product/oswbb/archive/ [JOCDB1] ll
total 0
drwxr-xr-x 2 oracle oinstall 136 23 sept. 10:00 oswifconfig
drwxr-xr-x 2 oracle oinstall 132 23 sept. 10:00 oswiostat
drwxr-xr-x 2 oracle oinstall 134 23 sept. 10:00 oswmeminfo
drwxr-xr-x 2 oracle oinstall 132 23 sept. 10:00 oswmpstat
drwxr-xr-x 2 oracle oinstall 134 23 sept. 10:00 oswnetstat
drwxr-xr-x 2 oracle oinstall 6 23 sept. 09:18 oswprvtnet
drwxr-xr-x 2 oracle oinstall 124 23 sept. 10:00 oswps
drwxr-xr-x 2 oracle oinstall 6 23 sept. 09:18 oswslabinfo
drwxr-xr-x 2 oracle oinstall 126 23 sept. 10:00 oswtop
drwxr-xr-x 2 oracle oinstall 132 23 sept. 10:00 oswvmstat

This tool is a Java utility and exists since OSWatcher version 4.0.0. It permits to create graphs and
omplete HTML reports containing collected OS statistics.

OSWBBA require no installation. It is embedded in the OSWatcher home directory.

To start the Analyser, run oswbba.jar :

$ java -jar oswbba.jar -i ./archive

Starting OSW Analyzer V7.3.3
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c) 2014 by Oracle Corporation
Parsing Data. Please Wait...
Scanning file headers for version and platform info...
Parsing file srvtestoel7.it.dbi-services.com_iostat_16.07.25.2000.dat
...

The ”–i” parameter indicates the OSWatcher archive directory and is mandatory.

Once launched, the main menu is displayed :

Enter 1 to Display CPU Process Queue Graphs

Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter 6 to Generate All CPU Gif Files
Enter 7 to Generate All Memory Gif Files
Enter 8 to Generate All Disk Gif Files
Enter L to Specify Alternate Location of Gif Directory
Enter T to Alter Graph Time Scale Only (Does not change analysis dataset)
Enter D to Return to Default Graph Time Scale
Enter R to Remove Currently Displayed Graphs
Enter A to Analyze Data
Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)
Enter P to Generate A Profile
Enter X to Export Parsed Data to File
Enter Q to Quit Program

You must enable a X-Windows environment to display graphs.

If you don’t want to go through this menu every time you want to display graph or generate report, you
can pass all of the above options to OSWBBA from the command line, for example :

$ java -jar oswbba.jar -i ./archive -4 -P last_crash

 -i : Specify the archive directory
 -4 : Create memory graphs
 -P : Create a profile called “last_crash”

Other options :
 -6..8 : Same options as in the menu
 -L : User specified location to place gif files
 -A : Create a report
 -B : Specify the start time to analyze (format Mon DD HH:MM:SS YYYY)
 -E : Specify the end time to analyze (format Mon DD HH:MM:SS YYYY)
 -F : Specify a filename of a text file containing a list of options
(all others options are ignored if –F is used)

Example :
$ java -jar oswbba.jar -i ./archive -6 -B Sep 23 09:25:00 2016 -E Sep 23 09:30:00 2016

Will start OSWatcher Analyzer with the following parameters :

 Archive directory : $OSWatcher_HOME/archive
 Generate all CPU GIF files
 Time slot : 23 of Septembre 2016 – 09:25:00 to 09:30:00
Generated file :
It’s also possible to specify in a text file all options you want to use and then run OSWBBA with the “-f”
parameter :

$ cat input.txt
-P today_crash -B Sep 23 09:00:00 2016 -E Sep 23 11:00:00 2016
$ java -jar oswbba.jar -i ./archive -F input.txt

This will generate a complete HTML report (called “today_crash”) with all available graphs based on the
statistics stored in the archive directory.