What is Exadata ?

Architecture, Components and Four Main

Features of Exadata Storage Server

How about if Storage Server only return 1 Million rows

which were requested in where clause of the query ?
This is what Exadata Software does i.e. Filtering at Storage Level. Of course this is one of the mail
feature of Exadata Software. A DBA can easily estimate the impact of this change. Technically this term is
called as Exadata Smart Scan.
So, Here is

What is Exadata ?

1. A bundle of Hardware and Software from Oracle Itself and ready to use immediately.
2. An Intelligent Storage Server to optimize Database Performance. In order to achieve this intelligent a
software called Exadata is installed on all storage servers.

In Exadata Terms, A single storage server is called Cell or Cell server. So I will be referring storage server
and cell server interchangeably in this article. I might be referring DB Server, DB Node for Database
Server.

Architecture of Exadata Database Machine:

In above diagram, there are three DB nodes, which are connect with two cell servers. Each storage
server has it components.

Component of Exadata Storage Server:

It's another storage device, which has CUP, Memory, Disks, network cards, an Operating System Oracle
Linux and Most important past Exdata Storage Server Software. There are mainly three services which
run on cell server for processing, CellSRV (Cell Service), MS (Management Server) and RS (Restart
Server).

Cell Service (CellSRV): This is the primary component of Exdata Cell which provides
all Exadata Storage services. This process communicate with Database server for providing
databaseblocks.

Management Server (MS): MS

service

provide

an interface

to

DBA for

communicating or Managing Cell Server. CellCLI is the Command Line tool which Exadata DBA used for
performingExadata Administration services.

Restart Server (RS): This service is to make sure the functioning of Exadata Server. RS
monitors CellSRV and MS for liveliness and restart them whenever required.

Four Main Features of Exadata Storage Server.

1. Smart Flash Cache Intelligent Caching: Storage servers usually have
two kind of storage Hard disk and Flash Cache. Flash Cache are fast storage device used to keep most
frequently accessed data for faster access, whenever there is a requirement of a block it is first looked
into Flash Cache and if available returned to the server without going to Hard disk for reading the block.
This is a common feature in Storage servers.
Now with Exadata Smart Flash Cache Inetlligence, Flash Disks are optimized to work according to
Oracle Database requirements. Exadata Software Directs Flash Cache to keep following objects into it.
A. Frequently access data and index blocks
B. Control Files read and write.
C. File headers read and write.
Following are not kept in Flash Cache.
a. Backup Related I/O
b. Data Pump I/O
c. Full Table Scan.
Database Administrator can also influence caching of Database Objects by pinning them into Flash
Cache by setting CELL_FLASH_CACHE parameter. Here is a better way to know Exadata Smart Flash
Cache.

2. Hybrid Columnar Compression (HCC): Compression

is used for

reducing storage consumption for large databases. In this technique a logical unit called Compression

unit is used to store Hybrid columnar Compressed Rows. At time of data loading Column values are
detached from rows, grouped and then compressed. After compression this is fit into compression unit.
Database Smart scan processing is also supported at cell level i.e. Data blocks need not to movefrom cell
server to database server for processing. The amount of saving storage space using HCC is around 10 to
15 times. Suppose you have 100 Tera Bytes of data, by using HCC DBA can storethis Data in to 10 TB of
Storage space.

3. Smart Scan Processing: In Exadata storage server has got intelligence to filter
data at storage level, rather than transferring it to database server.
Exadata Smart Scan works filtering for following kind of queries:
1. Predicate Filter: In a query with where clause only blocks with satisfy where clause condition will only
go to database server not the entire table.
2. Column Filtering: Suppose in a select query like "select name, age from employee" is execute on DB
server by DBA, then only columns name and age data will send to Database Server not all columns of
tables.
3. Join Processing: All join processing are done at Storage level itself, so only filtered data is sent to DB
server.

4. I/O Resource Management: In traditional Database environment, If you have

more than one database running on a Shared storage server. Large queries from one database can use
more resource and cause performance issue for other databases. Another case a batch job has started in
one DB by a DBA which case performance trouble into OLTP Database.
To avoid all these challenges Exadata has introduced Exadata Storage Server I/O Resource
Management (IORM). Till now we have Database Resource Manager (DBRM) to manage database
resource within the database but now with help of IORM DBA can manager resource across databases.

Let's take an example: You have two database Production and testing running on same Cell
Server. Now Database Administrator can configure Resource plan which gives priority to Production
Database over testing DB. Here, Whenever Testing Database load will affect Production Database IORM
will manage rerource request such that it doesn't impact production Database performance. This
means Test Database requests has to wait until they can execute without disturbing production database
performance.

What is Exadata Smart Flash Cache ?

Now, Exadata use Flash Cache more intelligently for utilization of storing Oracle Database, This feature is
called as Exadata Smart Flash Cache. Exadata Smart Flash Cache understand different type of
Database I/O and take cache decision according to I/O type. If an I/O operation is meant to be in Flash

Cache it will go otherwise it will go to Hard Disk. In Conclusion, Exadata Smart Flash Cache is an
intelligent Flash Cache which works according to I/O types.
Following I/O operations are stored in Smart Flash Cache.
1. Frequently Access data and Index blocks.
2. Control files reads and writes.
3. File Header read and writes.
Following I/O operations are not stored in Smart Flash Cache.
1. I/O operations for data block mirror copies.
2. Backup related I/O.
3. Data pump I/O.
4. Full Table scan.
To achieve, This intelligence each I/O is tagged with a metadata about I/O type, Exadata Flash Cache use
this information to make decision about keeping database block into flash cache or not.
For Example: DBA is taking an expdp backup, in traditional storage server with flash cache all
dbblocks are loaded into flash cache and can be a performance bottleneck in the system even we know
that, we are not going to use these blocks in future.
But in Exadata Smart Flash Cache, Exadata Storage Server knows that these data blocks should not go
into flash cache and makes a big difference in database performance. There are two types of Database
operation performed DB read or DB write. So let's see

How Exadata Smart Flash Cache works for Database

Read and Write Operations ?
Here we will discussion Database read and write operations with different -2 scenarios

1. Database Write Operations: Suppose DBA issue an insert statement which is

going to write data in data file at hard disk. In Exadata CellSRV is the service which perform read and
write
operations.Here
is
the
sequence
of
steps:
1.DB
request
for
2.
CellSRV
will
write
block
3.
Sent
an
Acknowledgement
4. Place a copy of data block into smart flash case as well.

to

data
into
hard
Database

write.
disk.
Node.

To optimize next read operation in Exadata. Now, next read request for same block will only go to flash
cache and will give a boost in performance by avoiding disk read operation.

2. Database Read Operation: There could be two situations for Database read
operations.
A. Data Block is Cached into Flash Cache: In this case DBA will get fast response, because the block
will be read from flash cache so no hard disk read I/O. Here are the steps:
1. Database sent a read request.
2. Data block is already in Smart Flash Cache.
3.This will supply data block to database node.

This whole process has skip data block reading from hard disk so boost the performance.
B. Data Block is not Cached into Flash Cache: In this case, when Database Administrator issue a
select query to read some data and data is not present in Smart flash cache. Below are the action
sequence:
1. DB request for Data block.
2. CellSRV will read blocks from Hard Disk.
3. CellSRV provide block to database node.
4. CellSRV place a copy of block into Exadata Smart Flash Cache.

Now next time when ever same block request will come from DBA the situation will be like 2A.

How to Change Objects Behavior for Smart Flash

Cache ?
Database Administrator can influence Objects default behavior i.e. To keep them in Flash Cache or not by
setting an object parameter "Cell_Flash_Cache". Which has three values DEFAULT,KEEP and NONE.

Default: This shows Exadata Smart Flash Cache is used normally. DBA can use belowcommand to
set CELL_FLASH_CACHE as DEFAULT.
ALTER TABLE {table_name} STORAGE(CELL_FLASH_CACHE DEFAULT)

Keep: Using CELL_FLASH_CACHE as KEEP means Exadata will try to keep this object more
aggressively in to Flash Cache.
ALTER TABLE {table_name} STORAGE(CELL_FLASH_CACHE KEEP)

None: Setting CELL_FLASH_CACHE as NONE by Remote DBA for an object shows Flash Cache
will not use at all for this object.

ALTER TABLE {table_name} STORAGE(CELL_FLASH_CACHE NONE)

DBA can consider these option for Database Objects according to their importance. like a frequent
accessed partition of a table can reside into Smart Flash Cache with help of keep parameter.
Each I/O operation supplied to Cell server can have one of the cache hints, which decide this I/O should
be cached or not. This process is managed by Exadata Cell Software So,Remote DBA has no role or
intervention into this process.
Cache: This hint indicated I/O will be cached in Flash Cache e.g. I/O for an Index lookup.
Nocache: Shows I/O operation will not cache in Exadata Smart Flash Cache like RMAN Backup I/O.
Evict: This means I/O should be removed from Flash Cache like ASM Re balance operation.
Please share if this post help you to understand Exadata Smart Flash Cache in a better way.

Exadata Administration CellCLI :

http://www.unixarena.com/2014/11/exadata-storage-cell-commands-cheatsheet.html

One of the big differences between Exadata and Non-Exadata systems is the necessity to administer
the Exadata Storage Server.
The first time you have to configure the Server side, it has to be done through KVM (Keyboard, Video,
Mouse), meaning you will need to be physically near your server. Once the initial configuration steps
have been performed, we shall be able to administer the Exadata Storage Servers over the network
(i.e. SSH protocol or redirect the KVM console to your desktop using the Sun Integrated Lights Out
Management ILOM remote client).
Once you are in the server, some tools will be available to us, including Cell Command Line Interface
(CellCLI).

CellCLI is only available on the Storage Server. If we need to execute it from outside the storage server
(or on more than one storage server), we will need to use the Distributed Command Line Interface
(dcli).
CellCLI will allow us to perform administration task like startup, shut down, monitoring activities as well
as maintenance.
The syntax to follow is:
[admin command / object command] [options];
Being:
1.- Admin Commands: Administration actions like START, QUIT, HELP, SPOOL.
2.- Object Command: Administration action to be performed on the cell objects like ALTER, CREATE,
LIST.
3.- Options: Will allow us to specify additional parameters to the command.

---------------------------------------------------------------------------------------------------------------[celladmin@ssssss ~]$ cellcli

CellCLI: Release 11.2.2.2.0 - Production on Sat Jul 16 17:28:08 BST 2011
Copyright (c) 2007, 2009, Oracle. All rights reserved.
Cell Efficiency Ratio: 17
CellCLI> list cell
ssss_admin online
CellCLI> exit
quitting

If you prefer to execute the command from command-line, we will have to use -e:

1
2
3
Or

[celladmin@ssssss ~]$ cellcli -e list cell

ssss_admin online
[celladmin@ssssss ~]$

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

[celladmin@sssss ~]$ cellcli -e help

HELP [topic]
Available Topics:
ALTER
ALTER ALERTHISTORY
ALTER CELL
ALTER CELLDISK
ALTER GRIDDISK
ALTER IORMPLAN
ALTER LUN
ALTER PHYSICALDISK
ALTER QUARANTINE
ALTER THRESHOLD
ASSIGN KEY
CALIBRATE
CREATE
CREATE CELL
CREATE CELLDISK
CREATE FLASHCACHE
CREATE GRIDDISK
CREATE KEY
CREATE QUARANTINE
CREATE THRESHOLD
DESCRIBE
DROP
DROP ALERTHISTORY
DROP CELL
DROP CELLDISK
DROP FLASHCACHE
DROP GRIDDISK
DROP QUARANTINE
DROP THRESHOLD
EXPORT CELLDISK
IMPORT CELLDISK
LIST
LIST ACTIVEREQUEST
LIST ALERTDEFINITION
LIST ALERTHISTORY
LIST CELL
LIST CELLDISK
LIST FLASHCACHE
LIST FLASHCACHECONTENT
LIST GRIDDISK
LIST IORMPLAN
LIST KEY
LIST LUN
LIST METRICCURRENT
LIST METRICDEFINITION
LIST METRICHISTORY
LIST PHYSICALDISK
LIST QUARANTINE
LIST THRESHOLD
SET
SPOOL
START

so, what about security?, CellCLI does rely on OS authentication.

There are three predefined users on the Exadata Storage Server:

1.- cellmonitor: This user can perform monitoring tasks.
2.- cellamin: This user can perform most of the administration task such CREATE,
ALTER, MODIFY cell objects (cannot perform CALIBRATE).
3.- root: This user has super-user privileges.

---------------------------------------------------------------------------------------How to check smart scan is enabled :

PROD> show parameter cell_offload_processing

NAME
TYPE
VALUE

----------------------cell_offload_processing

-------- -----------------------------boolean TRUE

The problem:
We were using DBFS to store our exports. This was the perfect solution as the business could see the
files on the destination folder, but it did not meet our requirements performance wise on our Exadata.
We have decided to mount NFS and performance did improve, but we had a different problem. NFS is
mounted on the database server and business do not have access for security reasons and segregation
of duties.
Since then, the export jobs run, but business could not see what files were created, so the question
was asked.
The solution:
After some research I came across with the following package:
SYS.DBMS_BACKUP_RESTORE.searchFiles
I did have to play a little bit, and I finished with the following script:
1.- Create an Oracle type

1
2

create type file_array as table of varchar2(100)

/

2.- Create the function as SYS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

CREATE OR REPLACE FUNCTION LIST_FILES (lp_string IN VARCHAR2

default null)
RETURN file_array pipelined AS
lv_pattern VARCHAR2(1024);
lv_ns VARCHAR2(1024);
BEGIN
SELECT directory_path
INTO lv_pattern
FROM dba_directories
WHERE directory_name = 'NFS_DIR';
SYS.DBMS_BACKUP_RESTORE.SEARCHFILES(lv_pattern, lv_ns);
FOR file_list IN (SELECT FNAME_KRBMSFT AS file_name
FROM X$KRBMSFT

16
17
18
19
20
21
22
23

WHERE FNAME_KRBMSFT LIKE '%'|| NVL(lp_string, FNAME_KRBMSFT)||'%'

) LOOP
PIPE ROW(file_list.file_name);
END LOOP;
END;
/

3.- Grant necessary permissions:

1
2

grant execute on LIST_FILES to public;

create public synonym list_files for sys.LIST_FILES;

4.- Test without WHERE clause:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

TESTDB> select * from table(list_files);

COLUMN_VALUE
--------------------------------------------------------------------------------------------------/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB.log
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece1.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece2.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece3.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece4.dmp
/nfs/oracle/TESTDB/pump_dir/imdp_piece_TESTDB_09092011.log
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece1.dmp_old
/nfs/oracle/TESTDB/pump_dir/imdp_piece_TESTDB_12092011.log
8 rows selected.
Elapsed: 00:00:00.10

5.- Test with WHERE clause:

1
2
3
4
5
6
7
8
9
10

TESTDB> select * from table(list_files) where column_value like '%dmp%';

COLUMN_VALUE
--------------------------------------------------------------------------------------------------/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece1.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece2.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece3.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece4.dmp
/nfs/oracle/TESTDB/pump_dir/expdp_TESTDB_piece1.dmp_old
Elapsed: 00:00:00.12

-------------------------------------------------------------------------------------------

problem: Scenario
From time to time, I have to run scripts or single commands on all nodes for Exadata. This can take
some time.
The problem:
We have a request from our developers to flush the shared pool on all nodes on our UAT Exadata. This
is due to a bug we are still experiencing.
The solution:
This is a typical request for my team, were we have to run something on all our nodes. Flushing shared
pool can be one of them.
Connecting and executing the same command 8 times, if you have a full rack, can be time-consuming
and it is, probably, not the most exciting task to do.
For this reason I came up with the following script. It is something to help me with, so please, test it
before using on production.
The sql file to be executed can be anything you would normally run from sqlplus.
------------------------------------------------------------------------------------

#!/bin/ksh
###############################
##
## Script to execute sql commands on all Exadata nodes
##
## Owner: David Alejo-Marcos
##
## usage: Exadata_sql_exec.ksh
## i.e.: Exadata_sql_exec.ksh MYTESTDB flush_shared_pool.sql
##
## Example of test.sql:

##
## oracle@hostname1 (MYTESTDB1)$ cat test.sql
## select sysdate from dual;
## select 'this is my test' from dual;
##
## alter system switch logfile;
##
###############################
NO_OF_ARGS=$#
typeset -u DB_SRC=$1
typeset -l DB_SCRIPT1=$2
typeset -l DB_SCRIPT=$2_tmp
if [[ "$NO_OF_ARGS" = "2" ]]; then
echo "Parameters passed: OK"
else
echo "usage: Exadata_sql_exec.ksh"
echo "i.e.: Exadata_sql_exec.ksh MYTESTDB flush_shared_pool.sql"
exit 1
fi
echo "sqlplus -s '/as sysdba'" >/tmp/${DB_SCRIPT}
echo "set linesize 500" >>/tmp/${DB_SCRIPT}
echo "set serveroutput on" >>/tmp/${DB_SCRIPT}
echo "select host_name, instance_name from v\\\$instance;" >>/tmp/${DB_SCRIPT}
echo "@/tmp/${DB_SCRIPT1}" >>/tmp/${DB_SCRIPT}
echo "exit;" >>/tmp/${DB_SCRIPT}
echo "EOF" >>/tmp/${DB_SCRIPT}
# Copy both scripts to all Exadata DB Servers.
dcli -l oracle -g ~/dbs_group -f /tmp/${DB_SCRIPT} -d /tmp
dcli -l oracle -g ~/dbs_group -f ${DB_SCRIPT1} -d /tmp
# Execute command on all Exadata DB
dcli -l oracle -g ~/dbs_group " (chmod 700 /tmp/${DB_SCRIPT} ; /tmp/$
{DB_SCRIPT} ${DB_SRC} ; rm /tmp/${DB_SCRIPT} ; rm /tmp/${DB_SCRIPT1})"