Professional Documents
Culture Documents
Table of contents
Introduction ......................................................................................................................................................3
Testing scenario and architecture ....................................................................................................................3
Introduction to testing .....................................................................................................................................10
Proof Point move a PDB between CDBs ....................................................................................................10
Proof Point relocate a datafile to an alternate diskgroup .............................................................................14
Proof Point create a clone from PDB ..........................................................................................................17
Conclusion .....................................................................................................................................................19
For more information ......................................................................................................................................20
Introduction
This whitepaper is a study of the new Multitenant features of Oracle Database 12c and how a
shared disk architecture using Fibre Channel infrastructure can be used to facilitate dramatically
simplified database consolidation compared to previous releases. In this paper we will explore
the new multitenant functionality by building a test database cluster, within which we relocate
databases between nodes, instances and storage tiers. The new functionality in Oracle 12c
makes these operations extremely straightforward and provides a compelling case for
consolidation using a shared disk infrastructure. Although shared disk storage may be achieved
through other means, this paper focuses on the proven combination of Fibre Channel and
Oracles Automatic Storage Management functionality. All testing performed for this whitepaper
was carried out using the latest Gen 5 (16GFC) Fibre Channel components from Emulex, which
ensures that maximum bandwidth is available in the Storage Area Network for any operations
that require data copying, and provides low latency access for all other database I/O operations.
Access to the high bandwidth throughput provided by Gen 5 parts will become increasingly
important for data mobility in consolidated cloud environments such as the one demonstrated
here.
Disclosure:
Scale Abilities was commissioned commercially by
Emulex Corporation to conduct this study. To ensure
impartial commentary under this arrangement, Scale
Abilities was awarded sole editorial control and Emulex
granted sole veto rights for publication.
From a storage perspective, three different tiers of storage were defined to reflect the common
reality of multiple classes of storage being present in the data center. These were defined as
Small Storage Array, to represent a mid-tier traditional storage array, Big Storage Array, to
represent a top-tier traditional storage array and Fast Storage Array, to represent the growing
reality of solid-state storage. All the storage is shared across all cluster nodes, making the
relocation of databases to different servers a very straightforward process, especially with the
Multitenant Option in Oracle 12c. Given that all the data is available to all of the cluster nodes, a
migration of PDBs between CDBs only requires a logical unplug from the source CDB and a
plug into the recipient CDB No data copying is required, only shared access.
Figure 2. Migration of PDBs between CDBs
The inherent shared architecture of the Multitenant Option makes the consolidation of
databases much simpler than in previous releases. The following diagram shows how a similar
implementation of two databases would have worked in the 11g release:
In this 11g example, each database would exist as an entity in its own right, with a full copy of
the data dictionary, and its own set of background processes and memory allocations (ie: the
instance) on each node in the cluster. Each of those instances would more or less compete for
resource, and had no direct control over the resource consumption by the other instances.
Certain measures could be taken, such as Instance Caging and Preferred Nodes, but these
were really just workarounds prior to the 12c solution. In the 12c Multitenant world, the
resources of all the databases can be effectively resource managed, and there is no overhead
for hosting multiple databases other than the resource actually required to host those
databases.
The testing focus was primarily on functionality and architecture, rather than raw performance.
Accordingly, both PDB1 and PDB2 were kept relatively simple, with just a few large tables
created in each. Each database was approximately 1TB in size to demonstrate a relatively large
data transport requirement. Any performance observations noted in this whitepaper are included
Each server node was an HP DL360 Generation 8 server with 96GB of memory and two CPU
sockets populated with 8 core Intel E5-2690 processors running at 2.9GHz. Each server was
equipped with two port Emulex LPe16002B Gen5 Fibre Channel HBAs, with each port
connected to independent Fibre Channel zones via Brocade 6510 Gen 5 Fibre Channel
switches. By using Emulex Onecommand Manager we verified Emulex Gen 5 Fibre Channel
adapters connectivity to the FC targets. In addition, updating the firmware was a simple process
during the initial setup. By using Emulex OneCommand Manager we were able to update online
the latest firmware version for Emulex Lightpulse LPe16002B adapter.
oracle1.emulex.com
oracle2.emulex.com
oracle3.emulex.com
oracle4.emulex.com
oracle5.emulex.com
The cluster was split into two server pools using Clusterware server pools. The low-criticality
container database (CDB1) is hosted by a server pool named lesscritical, which includes the
oracle4 and oracle5 hosts. The mission-critical container (CDB2) is hosted by a server pool
named missioncritical, which includes the oracle1, oracle2 and oracle 3 hosts.
[oracle@Oracle1 ~]$ srvctl config srvpool
Server pool name: Free
Importance: 0, Min: 0, Max: -1
Category:
Candidate server names:
Server pool name: Generic
Importance: 0, Min: 0, Max: -1
Category:
Candidate server names:
Server pool name: lesscritical
Importance: 0, Min: 0, Max: -1
Category:
We can also view the CRS resources to ensure we have instances of our CDBs running on the
nodes we expect:
[oracle@Oracle1 ~]$ crsctl stat res -t
----------------------------------------------------------------------------Name
Target State
Server
State details
----------------------------------------------------------------------------ora.cdb1.db
1
ONLINE OFFLINE
STABLE
2
ONLINE ONLINE
oracle4
Open,STABLE
3
ONLINE OFFLINE
STABLE
4
ONLINE ONLINE
oracle5
Open,STABLE
5
ONLINE OFFLINE
STABLE
ora.cdb2.db
1
ONLINE ONLINE
oracle1
Open,STABLE
2
ONLINE OFFLINE
STABLE
3
ONLINE ONLINE
oracle2
Open,STABLE
4
ONLINE OFFLINE
STABLE
5
ONLINE ONLINE
oracle3
Open,STABLE
ASM is configured using the new FlexASM feature of Oracle12c. This is a very useful new
feature that moves away from the former dependency to have one ASM instance on every
cluster node. From Oracle 12c, it is possible to host full instances of ASM on a subset of the
cluster nodes, and for other cluster nodes to collect ASM metadata information via a local proxy
instance, which in turn connects to the full ASM instances. For this testing, the full ASM
instances where local on nodes 1, 2, 3 and remote via the Proxy instance on nodes 4 and 5.
[oracle@Oracle1 ~]$ crsctl stat res -t
----------------------------------------------------------------------------Cluster Resources
----------------------------------------------------------------------------ora.asm
1
ONLINE ONLINE
oracle1
STABLE
2
ONLINE ONLINE
oracle3
STABLE
3
ONLINE ONLINE
oracle2
STABLE
Although FlexASM was used in this case, it is also possible to perform all of the testing in this
whitepaper using the traditional ASM model of having one instance of ASM on each cluster
node.
Introduction to testing
In pre-12c releases, it was necessary to host multiple full instances of each database on each
node required, resulting in wasted resources and rudimentary resource management
capabilities. For example, if one wanted to consolidate databases named DB1 and DB2 onto a
shared cluster using pre-12c technology, it would be necessary to have instances for both DB1
and DB2 running on at least two nodes of the cluster. Each of these instances would have to be
configured to accommodate the full workload of its application on each node, including any
passive nodes if running in active/passive mode. This resulted in gross memory and kernel
resource waste on each server. The results worsened when instances would failover to other
nodes, as workloads would very often end up hosted on an inappropriate node. Resource
management was even more restricted, as each instance had no visibility or control of the
resource requirements of instances on the shared server.
The concept of Container Databases (CDBs) and Pluggable Databases (PDBs) is new in Oracle
12c, and is fundamental to providing the missing link in database consolidation for Oracle
databases. A CDB is the container for one or more PDB, and a PDB is the actual database
from the viewpoint of the application. PDBs can be plugged into and unplugged from CDBs
using simple commands, and they can be cloned and moved to other CDBs. All the PDBs that
are plugged into a CDB share a single Oracle instance (per node, in the case of RAC), and can
be resource-managed by a single set of controls within the CDB.
The philosophy behind this testing was to perform key operations on PDBs that would be
frequently required in a true consolidated environment and to discover what the utility-value was
from running large shared clusters of multiple CDBs.
In addition to the Multitenant option, 12c also now supports fully online datafile moves. It is
believed this functionality goes hand in hand with the management of PDBs, as it offers the
ability to relocate databases onto different tiers of storage, as well as movement between CDBs.
Next PDB1 is unplugged. This changes the state of the PDB in the current CDB (CDB1) to be
unplugged and prevents all access to PDB1. It produces an XML file in the specified location
that contains all the metadata required to plug the database back into a CDB.
SQL> alter pluggable database pdb1 unplug into
'/home/oracle/pdb1_unplug.xml';
Pluggable database altered.
One aspect of this operation not immediately obvious is the XML file created by the foreground
process on the database server to which the user is connected. In a RAC environment (and in a
client/server environment), this is not necessarily the same server which is running SQL*Plus;
therefore, care must be taken to understand where the database connection is made.
The resulting XML contains a variety of interesting information, such as a list of files, location,
size, container ID, database ID, options installed, version and so on. This XML file is the master
description of the PDB and is used by the next CDB into which the PDB is plugged to determine
the steps required.
Looking back at CDB1, it is evident that PDB1 is now unplugged and cannot be opened up
again:
11
Now PDB1 can be plugged into CDB2. Start by connecting to the CDB2 root using the TNS
alias created by dbca:
[oracle@Oracle1 ~]$ sqlplus sys@cdb2 as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Fri Sep 6 08:13:08 2013
Copyright (c) 1982, 2013, Oracle.
Enter password:
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> select pdb_id,pdb_name,guid,status from CDB_PDBS;
PDB_ID PDB_NAME
GUID
STATUS
---------- ---------- -------------------------------- ------------3 PDB2
E5A3120148FF19CDE043975BC00AAF39 NORMAL
2 PDB$SEED
E5A30153B3FB11AFE043975BC00AC732 NORMAL
SQL> create pluggable database pdb1 using '/home/oracle/pdb1_unplug.xml'
nocopy;
Pluggable database created.
12
Next, lets check the status and open up PDB1 in its new container database:
SQL> select con_id,inst_id,name,open_mode from gv$PDBs order by 1,2
2 /
CON_ID
INST_ID
---------- ---------2
1
2
3
2
5
3
1
3
3
3
5
4
1
4
3
4
5
13
NAME
-----------------------------PDB$SEED
PDB$SEED
PDB$SEED
PDB2
PDB2
PDB2
PDB1
PDB1
PDB1
OPEN_MODE
---------READ ONLY
READ ONLY
READ ONLY
READ WRITE
READ WRITE
READ WRITE
MOUNTED
MOUNTED
MOUNTED
Note
The attempt to move a datafile for a PDB while connected to the CDB$ROOT was unsuccessful, and the
error message was not entirely helpful:
14
Although the move was successful, it was very slow (more on that shortly). During the move, Scale
Abilities was able to read and update a table within that tablespace:
SQL> select sum(object_id) from bigtable where rownum<10
SUM(OBJECT_ID)
-------------324
SQL> update bigtable set object_id=object_id+10 where rownum<10;
9 rows updated.
SQL> commit;
Commit complete.
SQL> select sum(object_id) from bigtable where rownum<10;
SUM(OBJECT_ID)
-------------414
Now, considering the long runtime for the move operation. As mentioned earlier, this paper isnt
focused on performance; however, that was a very long runtime and worthy of some
explanation. The Scale Abilities lab will perform a more technical investigation in the future and
write a blog post about it. In the meantime heres what a trace of the session executing the
move looked like:
WAIT #140641124141656: nam='control file sequential read' ela= 971 file#=0
block#=59 blocks=1 obj#=-1 tim=393150810594
WAIT #140641124141656: nam='db file sequential read' ela= 1494 file#=45
block#=119070465 blocks=128 obj#=-1 tim=393150812119
WAIT #140641124141656: nam='db file single write' ela= 1759 file#=45
block#=119070465 blocks=128 obj#=-1 tim=393150813980
WAIT #140641124141656: nam='DFS lock handle' ela= 193 type|mode=1128857605
id1=110 id2=1 obj#=-1 tim=393150814264
WAIT #140641124141656: nam='DFS lock handle' ela= 218 type|mode=1128857605
id1=110 id2=3 obj#=-1 tim=393150814537
WAIT #140641124141656: nam='DFS lock handle' ela= 11668 type|mode=1128857605
id1=110 id2=2 obj#=-1 tim=393150826256
WAIT #140641124141656: nam='control file sequential read' ela= 249 file#=0
block#=1 blocks=1 obj#=-1 tim=393150826570
WAIT #140641124141656: nam='control file sequential read' ela= 982 file#=0
block#=48 blocks=1 obj#=-1 tim=393150827577
15
An interesting observation for aficionados of trace files is that the file number of the original and
copy datafile appear to be registered as the same (45), which is likely a type of anomaly with the
wait interface abstraction. This can be observed in the respective waits for db file sequential
read and db file single write, where it is reported as the file# parameter. It seems the move
operation is taking place 128 blocks at a time (128*8KB=1MB in this case); however, there are a
series of coordinating measures taking place in the RAC tier which slow down the progress. In
particular, note the 11.6ms wait for DFS lock handle highlighted in green, which (when
decoded) shows a wait for the CI enqueue. The CI enqueue is a cross-instance invocation most
likely associated with coordinating the DBWR processes on other instances to ensure dirty
blocks are written out to both the original datafile and the new copy.
The total elapsed time for one iteration (1MB moved) in this trace file fragment is 18.984ms,
which implies a disk throughput of 52MB/s, very little of which is spent actually transferring data
because of the time spent waiting for the DFS lock handle. Scale Abilities also tried a datafile
move operation with the PDB explicitly closed on all but one instance, but it did not improve the
throughput and requires further investigation.
Shutting down all the instances except the one where the move takes place dramatically
reduces the overhead, but does not remove it altogether:
WAIT #140653075283640: nam='DFS lock handle' ela= 4542 type|mode=1128857605
id1=110 id2=2 obj#=-1 tim=2388675442209
WAIT #140653075283640: nam='control file sequential read' ela= 806 file#=0
block#=1 blocks=1 obj#=-1 tim=2388675443083
WAIT #140653075283640: nam='control file sequential read' ela= 921 file#=0
block#=48 blocks=1 obj#=-1 tim=2388675444060
WAIT #140653075283640: nam='control file sequential read' ela= 965 file#=0
block#=50 blocks=1 obj#=-1 tim=2388675445060
WAIT #140653075283640: nam='control file sequential read' ela= 967 file#=0
block#=60 blocks=1 obj#=-1 tim=2388675446062
WAIT #140653075283640: nam='db file sequential read' ela= 1748 file#=45
block#=20943489 blocks=128 obj#=-1 tim=2388675447849
WAIT #140653075283640: nam='db file single write' ela= 2891 file#=45
block#=20943489 blocks=128 obj#=-1 tim=2388675450857
16
SID
SERIAL#
---------- ---------392
6745
OPNAME
PCT
----------------------------------------- ---------Online data file move
77.42
This is much more indicative of the performance we should be seeing, considering the
specification of this system: A copy time of 43:16 represents a decent average write for a
bandwidth of 406MB/s. The ability to achieve a much higher number than this with a little careful
tuning of the I/O subsystem seems promising, but these results are a good start for a nonperformance focused exercise. The observed 406MB/s would saturate a 4GFC fibre channel
17
It is likely that the block size used for the copy is equal to one ASM allocation unit (default 1MB),
considering the trace file indicates that this session is coordinating the copy directly with the
ASM instance.
18
Conclusion
The new Multitenant functionality seems a natural fit for consolidation using server pooled
clusters and shared disk solutions using Gen 5 Fibre Channel. The easy access to data using a
shared storage infrastructure complements the ease of use afforded by the PDB concept and
effectively eliminates data copies for many use cases. The ability to pool multiple pluggable
databases into a small number of container databases dramatically improves both the system
management burden and the ability to effectively resource-manage databases in a shared
environment.
Multitenancy can also be implemented without shared disk and with locally attached SSD
storage. Doing so removes many of the advantages of consolidation, and perhaps indicates the
systems in question were not ideal candidates for consolidation. Management of a consolidated
environment requires, by its very nature, the ability to move workloads between servers and
storage tiers as the demands and growth of the business dictate.
Although some issues were observed with the throughput of the online datafile move
functionality, it should be stressed that there is nothing architecturally wrong in the method
Oracle has elected to use. Scale Abilities fully expects the throughput of such moves to be
significantly higher once the root cause is found, and this will become a frequently used
operation for DBAs, in both consolidated environments and more traditional ones.
One possible implication of the increased mobility of databases and their storage is that the
bottleneck will move from the labor intensive manual processes formerly required, and put more
pressure on the storage tier. In particular, it is evident that the bandwidth provision in the SAN
infrastructure will become increasingly important. This will be especially true for the server
HBAs, because the data mobility is a server-side action -- potentially migrating large data sets
from many storage arrays at any point in time. The availability of Gen 5 a Fibre Channel HBAs
will be instrumental in providing this uplift in data bandwidth.
19
20
World Headquarters 3333 Susan Street, Costa Mesa, California 92626 +1 714 662 5600
Bangalore, India +91 80 40156789 | Beijing, China +86 10 68499547
Dublin, Ireland+35 3 (0)1 652 1700 | Munich, Germany +49 (0) 89 97007 177
Paris, France +33 (0) 158 580 022 | Tokyo, Japan +81 3 5322 1348
Wokingham, United Kingdom +44 (0) 118 977 2929
21