WHITE PAPER

3PAR Thin Provisioning Architecture Overview and Best Practices

Authored by: Ashok Singhal, CTO & Founder, 3PARdata, Inc.

3PAR Thin Provisioning Architecture Overview and Best Practices

THIN PR OVISI ON ING ARCHIT ECTUR E

1 2 3

PREFACE ..................................................................................................... 3 INTRODUCTION........................................................................................... 4 THIN PROVISIONING ARCHITECTURE ...................................................... 5

3.1 3PAR Traditional Virtual Volume Management ................................................. 5 3.2 3PAR Thin Provisioning Volume Management.................................................. 6
3.2.1 3.2.2 3.2.3 3.2.4 Common Provisioning Groups ............................................................................... 6 Thin Provisioned Virtual Volumes (TPVVs)............................................................. 7 Thin Provisioning Capacity Allocation Scheme ....................................................... 8 Control Mechanisms and alerts.............................................................................. 9

BEST PRACTICES................................................................................................ 11 4.1 When not to use CPGs ................................................................................... 11 4.2 When to use different CPGs for different VVs ................................................. 11
4.2.1 4.2.2 Use different CPGs for VVs that require different LD characteristics...................... 11 Use different CPGs when there is a large number of VVs ..................................... 11

4.3 When to use the same CPG for different VVs ................................................. 12 4.4 Avoid creating VVs with explicit SD and SA space .......................................... 12 4.5 Avoid changing LD characteristics within a CPG ............................................. 12 4.6 Managing Free Space .................................................................................... 12 4.7 Use Allocation and Warning Limits for CPGs .................................................. 13 4.8 Use the allocation warning and allocation limit for TPVVs ............................... 13 5 6 SUMMARY ............................................................................................................ 14 ABOUT 3PAR........................................................................................................ 15

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

PREFACE
This white paper covers the advanced topic of 3PAR Thin Provisioning. It is recommended the readers have a good familiarity with 3PAR InSpire Architecture and basic understanding of Thin Provisioning before reading on. A 3PAR Account Executive or Systems Engineer can provide a copy of these primers. Alternatively, a copy of Thin Provisioning white paper can be obtained from 3PARs website (www.3par.com/products/prodlib.html) and a copy of the Architecture white paper can be requested by clicking on send more info.

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

I NT RO DUCT IO N
3PAR Thin Provisioning offers a simple solution to the problem of unused allocated capacity. conceivably needed over its lifetime. accordance with application writes. This white paper provides a technical overview of 3PAR Thin Provisioning including a discussion on the capacity allocation scheme and Thin Provisioning control mechanisms and alerts. recommendations on best practices for taking maximum advantage of Thin Provisioning. It concludes with Thin

Provisioning allows IT departments to safely allocate to an application as much virtual logical capacity as Meanwhile, physical capacity is drawn from a common pool of purchased storage on an as-needed basis. That is, physical capacity is drawn from the pool in direct

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

T HI N PRO VI SIO NI NG ARCHI T ECT URE

To understand how 3PAR Thin Provisioning works, it is useful to first understand 3PARs approach to virtualization.

3.1 3PAR Traditional Virtual Volume Management

The 3PAR InForm Operating System employs memory mapping a 3-level of mapping server disk (256 methodology similar to the virtual architectures virtualizes operating systems. The first level of physical drives of any size into a pool of uniform-sized chunklets megabytes (MB) each) and manages the redundant pathing to each chunklet and disk drive. The fine-grained nature of these chunklets of eliminates assets. underutilization storage

Volumes can be sized precisely and not according to large arbitrary increments. The chunklets fine-grained nature also enhances performance for all applications, regardless of their capacity requirements, by allowing distribution of chunklets across scores, or even hundreds of disks. Complete system access to every chunklet means no pockets of inaccessible storage. The second level of mapping associates chunklets with Logical Disks (LDs). This association allows logical devices to be created with template properties based upon RAID characteristics and the location of chunklets across the system. LDs can be tailored to meet a variety of cost, capacity, performance, and availability characteristics, depending upon the service levels required. In addition, the first and second level mappings taken together serve to parallelize work massively across disks and their Fibre Channel connections. The third level of mapping associates traditional Virtual Volumes (VVs) with multiple LDs. VVs are virtual capacity representations that are ultimately exported to hosts and applications. A traditional base VV consists of three separate spaces that are mapped to LDs: User Space represents the size of the Virtual Volume seen by a host to which the Virtual Volume has been exported as a Virtual LUN. Snapshot Data (SD) Space used for copy-on-write data space created by all the snapshots Snapshot Administration (SA) Space used for the mapping structure for all the snapshots

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

For traditional VVs, the user space is 100% pre-mapped to the underlying LDs and ultimately to chunklets (i.e. a 10GB base VV requires a dedicated 10GB user space). Though changes may be made subsequent to creation (for example, to optimize the chunklet location), a full mapping exists at all times for all underlying LDs of a base VV. Underlying chunklets and LDs (or portion of LDs) are thus pre-dedicated to a given VV, regardless of how much has been written to that VV. 3.2 3PAR Thin Provisioning Volume Management
3PAR Thin Provisioning Volume Management removes the pre-dedication of physical capacity. It builds on the 3PAR traditional volume management, but adds an intermediate function, known as a Common Provisioning Group. and the LD layer. where The CPG Unlike VVs are function sits between the VV layer traditional 3PAR virtual volume management backed 100% by LDs, a CPG effectively creates LDs from free chunklets on an as-needed basis. A CPG then maintains mappings from its owned LDs to VVs that have been associated with it. A base VV that depends on a CPG is known as a Thin Provisioned Virtual Volume (TPVV). Note that just as with traditional VV management, there is still massive parallelism of work over system resources, with chunklets distributed across scores, or even hundreds of disks. 3.2.1 Common Provisioning Groups

As discussed above, a Common Provisioning Group (CPG) is a group of LDs with an associated mechanism to automatically provision logical capacity to the Snapshot Administration and Snapshot Data spaces for associated Thin Provisioned Virtual Volumes or traditional base Virtual Volumes. It consists of two disjoint sets of LDs one set of LDs for SA spaces and another set of LDs for SD spaces. Key attributes of CPGs include: Auto Growth. By default, a CPG is configured to create new LDs when the available LD space falls below the configured grow size. The initial buffer pool of LDs starts off at a fraction of the exported capacity of mapped VVs and grows over time as needed. Effectively logical capacity growth tracks the growth in actual used capacity, allowing customers to purchase and install physical capacity as needed over time. Alternatively, auto-growth of LDs can be disabled and unused LDs can be manually added to a CPG to limit available logical capacity to a fixed pool.

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

Capacity Pooling. CPGs enable shared access to pooled logical capacity. Instead of prededicating LDs, however small or large, to traditional VVs, a CPG allows multiple TPVVs and snapshot space of normal VVs to share a buffer pool of LDs. When the SA or SD space for an associated TPVV or a normal VV are close to running out of space, the SA or SD space grows automatically by mapping new regions from LDs in the CPG (on each node). So, while a CPG contains dedicated LDs, pooled resources remain undedicated to TPVVs or normal VVs until needed.

3.2.2

Thin Provisioned Virtual Volumes (TPVVs) Unlike

Virtual capacity is represented to host servers and applications using Virtual Volumes (VVs).

traditional VVs for which the user space is 100% pre-mapped to the underlying LDs, a Thin Provisioned VV (TPVV) has no user space. Instead, the user space of base TPVVs is mapped on demand to the SD space on a copy-on-write basis (just like snapshot data). Thus, the physical storage space allocated to a TPVV may be much less than its virtual size (the size of the LUN as seen from the host). The SA and SD spaces of a TPVV are always associated with a CPG and automatically grow from space on LDs within the CPG. TPVVs behave like traditional VVs the differences are transparent to the host. TPVVs can serve as base volumes for Virtual Copies, Full Copies, or Remote Copies. As with traditional VVs, application-tailored TPVVs are created and exported to hosts in a just a few seconds.

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

3.2.3

Thin Provisioning Capacity Allocation Scheme

Allocation Algorithm - summary

TPVVs (Exported Capacity)
1) Host writes mapped to allocated TPVV LD space in 16KB pages optimize capacity usage

TPVVs (SD Capacity)

1) Initial Allocation 4 GB mapped from CPG LD pool upon TPVV creation 2) Subsequent Allocation(s) - Minimum of 4 GB
mapped from CPG LD pool as the TPVV SD capacity runs low

Common Provisioning Groups

1) Auto-grow LDs by configurable growth increment: a) Upon CPG creation or b) When unallocated LD space falls below the configured growth increment (min. 8GB) 2) Allocation Limit prevents overconsumption by disabling auto-growth.

Physical Disks
broken into Chunklets (256MB each)

Host Write Growth Increment

Logical Disk Mapped LD Space Mapped LD Space Written data less than allocated TPVV LD capacity

Logical Disk

4 GB
Logical Disk Logical Disk

16 KB pages

TPVV LD capacity less than CPG LD capacity

Note beginning in InForm OS Release 2.1.1, the allocation scheme will change as follows: Minimum TPVV SD space allocation will change from 4GB to 256 MB (per node) CPGs will auto-grow LD capacity when unallocated LD space falls below three-quarter the configured growth increment

Capacity allocations occur at multiple levels:

CPG LD Pool CPGs auto-grow LDs by configurable grow-size. In InForm OS Release 2.1.0, the default grow-size is 16GB and the minimum (non-zero) grow-size is 8GB. Upon CPG creation, an initial set of LDs is created as configured by grow-size. Subsequently, as the amount of unallocated LD space in a CPG falls below the configured grow size, the CPG automatically grows new LDs (again by the configured grow-size). LDs continue to auto-grow within a CPG as required over time until the CPG allocation limit is reached. Thereafter, LD auto-growth is disabled. Beginning in InForm OS Release 2.1.1 (Q4 03), the default grow-size for CPG LD pool will be 32 GB. Moreover, CPGs will automatically grow new LDs when the unallocated LD space falls below three-quarter of the configured grow-size (versus falling below the full grow-size in InForm OS Release 2.1.0). TPVV LD Allocation A CPG automatically provisions LD regions to the SA and SD spaces of associated VVs. In InForm OS Release 2.1.0, the minimum SD space allocated is 4 GB. Upon TPVV creation, an initial LD allocation of 4 GB is automatically mapped from the CPG to the TPVV SD space. (Note SA space allocation is a small fraction of SD space allocation.) As the application writes consume space and available TPVV SD space falls below a minimum threshold, additional 4 GB in LD regions is automatically mapped from the CPG LD pool as required. Beginning in InForm OS Release 2.1.1 (Q4 03), TPVV LD allocations will occur at a finer-grained, minimum increment of 256 MB (per node the volume is exported from) versus 4 GB today. Thus, an initial LD allocation of 256 MB (per node) will be mapped from CPG to the TPVV SD space

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

upon TPVV creation. Subsequent allocations driven by actual usage of allocated capacity will be adaptive the allocation amount will vary based on the rate of capacity consumption. By default, an allocation of 256 MB (per node) is mapped from the CPG unless the last LD allocation amount is consumed within one minute. Anticipating a quick rate of storage consumption, subsequent LD space allocations will grow progressively by 256 MB (per node) up to the maximum of 1 GB (per node). As the TPVV exported size or allocation limit is reached, the last allocation will map 256 MB (per node) over TPVV exported size to ensure that full exported TPVV address space is usable. Base TPVV Space Mapping Unlike a standard VV, a TPVV has no user space that is premapped to LDs and ultimately to chunklets. Instead, the user space of a TPVV is segmented into 16 KB pages, which are mapped on host writes to the TPVV SD space. The fine granularity of this mapping optimizes capacity usage.

3.2.4

Control Mechanisms and alerts

Multiple control mechanisms are available and configurable to enable alerts that notify storage administrators of important events.
3.2.4.1 Allocation Warning

Allocation warning provides a mechanism for warning storage administrators when a specified logical capacity threshold is reached. An allocation warning can be specified independently for CPGs and TPVVs: CPG Allocation Warning For a CPG there are two different warning limits depending on whether CPG auto-grows or whether auto-grow is disabled: Allocation Warning Limit (using the aw option on createcpg or setcpg commands) If the used SD space exceeds this limit (as a percentage of total SD space), an alert is generated. This limit is useful mainly for CPGs for which auto-grow is disabled. SD Grow Warning Limit (using the sdgw option on createcpg or setcpg commands) Under default auto-grow configuration, an alert is generated if the aggregate size of the SD LDs exceeds this limit.

3.2.4.2

TPVV Allocation Warning An alert is generated if the SD space of the TPVV exceeds the allocation warning point (specified as percent of TPVV size).
Allocation Limit

Allocation Limit provides a control mechanism to pre-determine the maximum consumption of logical capacity. An allocation limit can be specified independently for CPGs and TPVVs: CPG Allocation Limit Once reached, the set of SD LDs in the CPG is not allowed to grow beyond the configured limit. New application writes to TPVVs mapped to the CPG receive a write failure once the existing SD LDs in the CPG become filled up. The write failures persist until the CPG allocation limit is raised. TPVV Allocation Limit - Once reached, the SD space of a TPVV is not allowed to exceed the allocation limit (specified as a percent of TPVV size) and new application writes fail.

FALL 2003

3PAR PROPRIETARY

THIN PR OVISI ON ING ARCHIT ECTUR E

3.2.4.3

Available Free Space Alerts

Beginning with InForm OS Release 2.1.1, a history of used and free CPG space will be available.1 The InForm OS samples CPGs and the space available to the CPGs for growth once per day. This history is stored in a table in the Admin VV and can be displayed using the hist option in the showspace and showcpg commands. Up to 365 entries of history are stored. An alert is automatically generated if the sampled available free space for a CPG falls below the point at which the CPG cannot grow to either the CPG warning limit or the CPG allocation limit.
3.2.4.4 Used Physical Capacity Alerts

As available physical capacity across the 3PAR InServ Storage Server is utilized by traditional VVs and/or TPVVs, multiple pre-configured alerts are generated that provide information about used physical capacity as a percent of total system capacity. These alerts in conjunction with available free space alerts serve as an advance warning to the storage administrator to plan for and add necessary physical capacity. In the unlikely scenario that effective physical capacity becomes used, the 3PAR InServ Storage Server naturally prevents new writes from occurring until more capacity is added. These control mechanisms and alerts combined with various alert delivery mechanisms (3PAR InForm CLI, 3PAR InForm GUI, SNMPv2 Traps, and email/page from 3PAR Central) enable storage administrators to manage 3PAR Thin Provisioning in the way that suites them.

1 This feature is not available in InForm OS Release 2.1.0

FALL 2003 3PAR PROPRIETARY 10

THIN PR OVISI ON ING ARCHIT ECTUR E

BEST PRACT I CES

While 3PARs virtualization provides a powerful, yet simple, process of provisioning, a certain level of upfront and on-going planning is required to take maximum advantage of Thin Provisioning. The best practices discussed below highlight critical management practices for consideration.

4.1 When not to use CPGs

CPGs work well for larger VVs. In InForm OS Release 2.1.0, the minimum TPVV SD space allocation is 4GB. So unless it is expected that a significant portion of the SD space will be used by snapshots, TPVV sizes of 4GB or smaller do not benefit from Thin Provisioning. TPVV sizes should preferably be 16GB or bigger. This does not apply if you expect to have a large number of snapshots that use substantial SD space. Starting in InForm OS Release 2.1.1, the minimum SD space allocation will occur at a finer increment of 256 MB per node, making CPGs attractive even for smaller VVs.

4.2 When to use different CPGs for different VVs

4.2.1 Use different CPGs for VVs that require different LD characteristics.

All the VVs associated with a CPG allocate space from a shared pool of LDs. There is no way for a storage administrator to explicitly specify the use of a specific LD within a CPG to a particular space allocation. When auto-grow is used, the LDs are created using the uniform LD characteristics (configurable at CPG creation or once CPG already exists). So VVs associated with a single CPG should require similar LD characteristics. For VVs that require different LD characteristics use different CPGs. For example: 4.2.2 Use a different CPG for RAID 1 VVs and RAID 5 VVs. Use a different CPG for VVs restricted to nodes 0,1 and for VVs restricted to nodes 2,3. Use different CPGs for VVs for which it is desirable to use different disks to minimize interaction with each other for performance reasons.

Use different CPGs when there is a large number of VVs

VVs in the same CPG can share the same LD. It is not desirable to have too many VVs sharing a single LD for two reasons: In the unlikely event that the LD is damaged (due to multiple simultaneous disk failures for example) all the VVs associated with the LD will be unavailable. Performance of the VVs will suffer from too much interleaving between VVs.

In consideration of these factors, it is recommended that no more than 32 VVs be associated with a single CPG.

FALL 2003

3PAR PROPRIETARY

11

THIN PR OVISI ON ING ARCHIT ECTUR E

4.3 When to use the same CPG for different VVs

CPGs allocate new LD space in certain granularity of LDs (minimum of 8GB, default of 16GB (InForm OS Release 2.1.0) or 32 GB (starting in InForm OS Release 2.1.1)). If there are too few VVs associated with the same CPG, there could be a sub-optimal amount of unused space in CPGs. A balanced association of multiple VVs to a CPG takes maximum advantage of capacity pooling benefits of CPGs.

4.4 Avoid creating VVs with explicit SD and SA space

In addition to TPVVs, SD space of traditional base VVs can be associated with a CPG. This association eliminates the planning required to estimate and dedicate logical capacity required for copy-on-write snapshots of a VV. Instead, the snapshot space can grow on-demand as needed. However, for the SA and SD space of a traditional VV to be eligible for association with a CPG, it is a pre-requisite that the VV not have any pre-existing SA or SD space. Thus, when creating a traditional VV, avoid explicitly specifying SD and SA space if it is desirable to associate these spaces with a CPG. An existing SA or SD space can never be removed from a VV. Similarly, once a VV is associated with a CPG, the SA and SD spaces of the VV cannot be explicitly grown.

4.5 Avoid changing LD characteristics within a CPG

LD space within a CPG is automatically allocated to VVs as needed. The storage administrator has no direct control over which LDs get used for a particular allocation. Therefore, in general, it makes little sense to have LDs with different characteristics in the same CPG. Although the capability to modify the LD characteristics for new LDs within a CPG is available, its use is advisable only in an emergency where different LD characteristics enable the CPG to utilize the remaining available physical capacity.

4.6 Managing Free Space

Each VV associated with a CPG can allocate space from the following sources (in order): Unused SD space. Though LDs from the CPG are mapped to the SD space of an associated VV and its snapshot descendents as needed, this space can grow to be quite large since the LD capacity of the SD space used by removed snapshots is marked free but left allocated to the SD space, i.e. SD space of a base VV does not shrink. Therefore, the only way that free SD space can be released for use by other base VVs is to remove the base VV. Unused SD space can be displayed using the showvv command. Unallocated CPG LD space. This space is shared by all the VVs associated with the CPG. Unallocated LD space can grow to be of considerable size since the SA and SD space of a base VV associated with the CPG is returned to the CPG as unallocated LD space upon VV removal. CPGs do not automatically remove unused LDs, however, the returned unallocated LDs get remapped to the SD space of associated TPVVs or traditional VVs as needed over time. To free this space for other uses (for example other CPGs), the unused LDs must be manually removed from a CPG. (Note, that only LDs that are entirely unused can be removed.) Unallocated LD space for a CPG can be displayed using the command: showld cpg cpgname. Free chunklets. Free chunklets available to a CPG for creating new LDs may be limited by the LD creation parameters for the CPG. It is important to understand that if different CPGs can draw from the same pool of chunklets, chunklets allocated to one CPG can reduce the pool of storage

FALL 2003

3PAR PROPRIETARY

12

THIN PR OVISI ON ING ARCHIT ECTUR E

available to other CPGs. It is recommended that storage administrators implement the following strategies to stay abreast of available free space: 1) Set the CPG allocation and warning limits; 2) Monitor free space reduction rate using the command: showspace cpg cpgname hist; 3) Monitor available free space alerts; and 4) Maintain a buffer stock of physical capacity.

4.7 Use Allocation and Warning Limits for CPGs

When the pool of chunklets available to a CPG for growing new LDs is also used for other purposes (for example: to grow LDs for another CPG, to create new VVs not associated with a CPG or as spares), it is important to ensure that the other uses of the chunklets are not starved by unchecked growth of the CPG. Storage administrators should set the CPG allocation limit to limit the CPGs growth to a point that still leaves sufficient free chunklets for the other uses. It is important to note that the allocation limit is a hard limit and the CPG will not grow beyond it. Any VVs that require more space from the CPG once the hard limit is reached will not be able to grow and will eventually present write errors to hosts unless the CPG allocation limit is raised. The CPG warning limit should be set sufficiently below the CPG allocation limit so that it alerts the storage administrator with ample time to react before the CPG allocation limit is reached. The CPG allocation limit and CPG warning limit also provide thresholds for alerts on available free space for a CPG.

4.8 Use the allocation warning and allocation limit for TPVVs
When multiple TPVVs share a CPG, unexpected growth of one VV can limit the ability of other VVs to grow. The allocation warning and allocation limit for the VVs can be used to limit the growth of individual VVs.

FALL 2003

3PAR PROPRIETARY

13

THIN PR OVISI ON ING ARCHIT ECTUR E

SUMMARY
3PAR Thin Provisioning offers Dedicate-on-Write as opposed to Dedicate-on-Allocation capacity provisioning, allowing logical capacity growth to track the growth in actual used capacity. Dedicate-on-Write provisioning is enabled by two key provisioning features - Common Provisioning Groups (CPGs) and Thin Provisioned Virtual Volumes (TPVVs). CPGs automatically create Logical Disks (LDs) and provision capacity on a metered basis to associated TPVVs and snapshot space of traditional Virtual Volumes (VVs). TPVVs export logical capacity to hosts and map application writes in fine-grained increments to allocated logical capacity. While the 3PAR InForm OS presents a simple, virtualized interface for provisioning, some upfront and on going planning and management are required. Available control mechanisms and alerts allow storage Moreover, recommended best administrators to manage Thin Provisioning in a way that suites them. demand.

practices allow storage administrators to take maximum advantage of capacity pooling and allocation on

FALL 2003

3PAR PROPRIETARY

14

THIN PR OVISI ON ING ARCHIT ECTUR E

ABO UT 3PAR
3PAR is the leading provider of a new category of enterprise storage arrays called Utility Storage that offers a highly scalable, efficient, and controllable information infrastructure. 3PARs tightly tuned system of software, hardware, and mission-critical service allows customers to securely consolidate storage, centralize information, and reduce total cost of ownershipsimply. been serving enterprise customers since 2002. The companys corporate headquarters is located at 4209 Technology Drive, Fremont, CA 94538. Phone: 510-413-5999, fax: 510-354-3070, email: info@3pardata.com <mailto:info@3pardata.com>, Web site: www.3par.com <http://www.3par.com>. Well positioned with key strategic investors including Oracle, Sun, and VERITAS, 3PAR has been operating successfully for the past four years and has

2002 2003 3PARdata, Inc. All rights reserved. 3PARdata, the 3PAR logo, 3PAR, and Serving Information are all trademarks of 3PARdata, Inc.

tp-arch-wp-01.0

FALL 2003

3PAR PROPRIETARY

15