4/4/12

CRX Clustering

Adobe (www.adobe.comhttp://www.adobe.com/go/gnav_adobe_logo_en_us) Products (www.adobe.comhttp://www.adobe.com/go/gnav_products_en_us) Solutions (www.adobe.comhttp://www.adobe.com/go/gnav_solutions_en_us) Learning (www.adobe.comhttp://www.adobe.com/go/gnav_learning_en_us) Help (www.adobe.comhttp://www.adobe.com/go/gnav_help_en_us) Downloads (www.adobe.comhttp://www.adobe.com/go/gnav_downloads_en_us) Company (www.adobe.comhttp://www.adobe.com/go/gnav_company_en_us) Store (www.adobe.comhttp://www.adobe.com/go/gnav_store_en_us) Adobe Store (www.adobe.comhttp://www.adobe.com/go/gnavtray_store_en_us) for home and home office Education Store (www.adobe.comhttp://www.adobe.com/go/gnavtray_store_education_en_us) for students, educators, and staff Business Store (www.adobe.comhttp://www.adobe.com/go/gnavtray_store_business_en_us) for small and medium businesses Other ways to buy (www.adobe.comhttp://www.adobe.com/go/gnavtray_store_otherwaystobuy_en_us)

Overview (www.adob e.com/docs/en.html) / CRX 2.1 (www.adob e.com/docs/en/crx/2-1.html) / Administering (www.adob e.com/docs/en/crx/2-1/administering.html) /

CRX Clustering
Clustering allows you to link two or more CRX or CQ instances together so that they share the same data storage. Typically, each instance runs on a distinct physical machine and is bound to a distinct IP address (in development many instances may be run on a single machine, but each will have a distinct port). A load balancer is used to distribute requests across the set of cluster instances while presenting a single address to the network, making the entire cluster appear as a single CRX/CQ instance. This arrangement eliminates the danger of single point of failure while improving performance and availability. You can easily scale your server environment when demand increases by adding additional machines seamlessly. For the applications running on CRX (including CQ), clustering is entirely transparent, so no changes to application code are needed in order to take advantage of the additional capability. By default, CRX (and therefore CQ) has clustering enabled, meaning that a standalone instance runs as a singleinstance cluster. Adding additional instances is easily done through the CRX administration graphical interface. If specific customization of the clustering setup is required, this can be done relatively easily through manual editing of configuration files. NOTE Standard industry terminology refers to each instance in a cluster as a node. You may encounter this terminology in other documentation. In this documentation, the term instance is used to avoid confusion with the unrelated concept of a JCR node (see JCR Data Model).

CRX Data Storage Overview

To understand how clustering works, a brief review of CRX data storage may be helpful. CRX data storage is composed of three elements:
Persistence Manager Store: This store holds all repository content except large binaries (these reside in the data store). The implementation of this store depends on the particular persistence manager (PM) being used. The default PM is the Tar PM w hich stores data in the file system as conventional Unix-style t rfiles. Other PMs, such as the MySQL PM and the Oracle a PM, store data in databases. Data Store: This store holds large binaries. On w rite, these are streamed directly to the data store and only an identifier referencing the binary is w ritten to the PM store. By providing this level of indirection, the data store ensures that large binaries are only stored once, even is they appear in multiple locations w ithin the content in the PM store. In effect the data store is an implementation detail of the PM store. Like the PM, the data store can be configured to store its data in a file system (the default) or in a database.
dev.day.com/docs/en/crx/2-1/administering/cluster.html 1/10

4/4/12

CRX Clustering

Journal: Whenever CRX w rites data it first records the intended change in the journal. Maintaining the journal helps ensure data consistency and helps the system to recover quickly from crashes. As w ith the PM and data stores, the journal can be stored in a file system (the default) or in a database.

In theory, each element in the CRX storage system can be independently configured to store its data in a file system or in one of a number of supported databases. In practice, mixing and matching storage techniques is rarely done. In most cases the default configuration consisting of Tar PM, File Data Store and File Journal, is acceptable. In cases where a customer has a reason to prefer database storage, all three mechanisms are usually configured to write to the same database (see Configuring Database PMs, Configuring the Data Store and Configuring the Journal).

How CRX Clustering Works

CRX clustering works as follows:
A CRX cluster consists of one master instance and some number of slave instances. A single standalone CRX instance is simply a master instance w ith zero slave instances. Typically, each instance runs on a distinct physical machine. The instances must all be able to communicate directly w ith each other over TCP/IP. All the instances share a common storage facility, either a shared file system directory mounted as a local drive on each of the instance machines, or a database to w hich all instances have access. In the rare case w here one of the storage elements is hosted on a database w hile another uses a file system (see above) the instances w ill need access to both the common file directory and the database. If the Tar PM is used, in addition, each instance also has a local copy of the PM store (but not of the journal or data store). The local copy is called the local cache, w hile the shared PM store is referred to as the master storage. The local copy is not used w hen using a database persistence manager. The shared file system or database is usually located on a separate, dedicated machine (i.e., not one of the CRX instance machines) This shared storage w ill typically be some type of high-reliability/high-availability system. The master instance can both read and w rite data to the shared storage and the journal. Slave instances can only read from the shared Tar PM storage and journal but never w rite to them directly. NOTE: All instances have independent read/w rite access to the data store, but since this access is alw ays mediated through identifiers stored in the shared storage and since data store records, once w ritten, are immutable, contention problems betw een clusters never occur at the data store level. In order to w rite, a slave instance sends the requested changes to the master instance, w hich w ill w rite the changes to both the journal and to the shared storage on behalf of the slave. In this w ay contention over the shared storage and journal is avoided and all w rites are serialized properly.

Default CRX Clustering Configuration

By default CRX (and therefore CQ) is configured to use exclusively file-based storage. In particular:
The master storage use the T r Mpersistence manager. aP The data store uses the F l D t S o eimplementation. ieaatr The journal uses the F l J u n limplementation. ieora

Example Cluster
This diagram depicts a cluster of two CRX instances running under the default (Tar PM, file-based) configuration:

dev.day.com/docs/en/crx/2-1/administering/cluster.html

2/10

4/4/12

CRX Clustering

In this setup the various write scenarios are managed as follows: 1. A (the master instance) wishes to write data. A writes the intended change to the Journal. For example A d X It then writes the change to its local d . cache (in the local directory: c x q i k t r / e o i o y w r s a e / w r s a e n m > c p / r-ucsatrpstr/okpcs<okpc-ae/oy) and to the master storage (in the shared directory: s a e / o k p c s < o k p c - a e / Large h r d w r s a e / w r s a e n m > ). binaries are streamed directly to the data store and only an identifier is written to cache and master storage. From time to time (by default once every 5 seconds, or if B wants to write something), B asks A over TCP/IP if there is new data. It does this by consulting As journal. Each instance remembers the position of the last entry it read in the journal. One precondition of writing an entry to the journal is that the instance must first read all entries that it might have missed. If there is new data, the journal information recording those changes is sent from A to B (also over TCP/IP). If the changes sent indicate that Bs intended write operation is no longer valid (for example, if the intended target has been modified or deleted) this is detected and the operation is aborted. Otherwise, the changes are applied to Bs local cache. 2. B (the slave instance) wishes to write data. B tells A (over TCP/IP) that it wants to write something. A ensures that only one slave can write data at the same time. If nobody else is currently writing then:
dev.day.com/docs/en/crx/2-1/administering/cluster.html 3/10

4/4/12

CRX Clustering

If A has new data, A will send it to B and B will write it to its cache (as mentioned above). B then sends the data it wishes to write to A over TCP/IP. If B is writing a large binary it writes the binary data directly and sends only the identifier to A via TCP/IP. A writes these changes to the journal, its local cache and to the the master storage. B tells A the the task is finished. B writes the changes to its local cache. 3. A and B both wish to write data. Since all writes are routed through A (the master instance), A ensures that all writes are serialized. Writing then proceeds as described above. 4. Instance A is down, B wishes to write data. When B detects that A is not responding, it automatically becomes the master. It then synchronizes its local cache with the shared master storage by consulting the journal. If A comes back up, it either connects to B as a slave and updates itself by consulting the journal or, if so configured (see Setting the Preferred Master Instance), it tells B that it wants to become master.

Tar File Optimization

The Tar PM stores its data in standard Unix-style tar files. Occassionally, you may want to optimize these storage files to increase the speed of data access. Optimizing a Tar PM clustered system is essentially identical to optimizing a stand-alone Tar PM instance (see Optimizing Tar Files (www.adob e.com/docs/en/crx/21/administering/persistence_managers.html#Optimizing%20Tar%20Files) ). The only difference is that to optimize a cluster, you must run the optimization process on the master instance. If the optimization is started on the master instance, the shared data as well as the local cache of the other cluster instances is optimized automatically. There is a small delay before the changes are propagated (a few seconds). If one cluster instance is not running while optimizing, the tar files of that cluster instance are automatically optimized the next time the instance is started. NOTE Additionally, in clusters, you need to ensure that the Tar optimization times are set to the same value on all cluster nodes. For example, < a a n m = a t O t m z A " v l e " : 0 prm ae"uopiiet au=1040"> :0/

Clustering Requirements
CRX clustering by itself does not guarantee improved performance or availability. In order for these to be realized one has to ensure a few other things are in place:
Each CRX instance should be on its ow n dedicated machine. During development and testing one can install multiple instances on a single machine, but for a production environment this is not recommended. The master storage, the data store and the journal (w hether file-based or database-based) should be hosted on a highreliability, high-availability storage system. For file storage the recommended technologies include enterprise-class SAN systems, NFS servers and CIFS servers. For database storage a high-availability setup running either Oracle or MySQL is recommended. Note that since the data is ultimately stored on this shared system the reliability and availability of the cluster in general depends greatly on the reliability and availability of the shared storage. The netw ork connecting the cluster instances to each other and to the shared storage should be of high-availability and low latency. A load balancer must be set up to distribute requests across the cluster instances. When running CQ, further gains can be realized by using the Dispatcher module w ithin an Apache Web Server at the front-end to maximize the caching of content as static HTML, only retrieving dynamically generated data from CQ w hen necessary. The Dispatcher is also useful w hen users (surfers or authors) of CQ are located in disparate geographical locations.

When configured in this way, CRX and CQ clustering will provide high reliability, availability and performance.
dev.day.com/docs/en/crx/2-1/administering/cluster.html 4/10

4/4/12

CRX Clustering

Types of Clustering
Active/Active
The clustering system built in to CRX is a form of active/active clustering. This means that multiple cluster instances run simultaneously, sharing the job of reading and writing to a common data storage. For CQ installations, this type of clustering can be employed both in author and publish environments and provides high-availability and high performance in both.

Active/Passive
Active/active clustering can be contrasted with active/passive clustering, where a single instance reads and writes from a common data storage until it fails, at which point requests are redirected to a secondary instance which takes over but uses the same common data storage. This type of clustering provides high-availability, but without the higher performance achieved by load balancing, since only one instance is running at any one time. CRX is compatible with this form of clustering as well, though it requires manual configuration and in most cases is inferior to standard CRX clustering.

Read-Only
For CQ installations, a related set up, called read-only clustering, is often used. In this arrangement, multiple publish instances each hold identical copies of the same content and share the job of servicing read requests. Writing however, is done on a single author instance which then periodically syndicates content to the publish instances. Syndication means sending duplicates of the same data to all recipients, thus keeping them all synchronized. In this arrangement there is no central data storage for the cluster as a whole. This type of clustering provides both highavailability and high-performance, but only for publishing, not authoring.

Installing and Configuring a CRX Cluster

This section provides detailed information on installing and configuring a CRX cluster, including how to install for large repositories, how to manually configure the clusters, and how to configure database storage.

Simple Cluster Installation

Every CRX installation comes pre-configured to run within a cluster. Under these default settings, the cluster can be configured easily through the CRX console GUI. NOTE If the repository is large, it is faster to create a new cluster node using the Cluster Installation for Large Repositories (www.adob e.com#Cluster Installation for Large Repositories) . Otherwise, the search index needs to be rebuilt.

1. Install two or more CRX instances. In a production environment, each would be installed on a dedicated server. For development and testing, you may install multiple instances on the same machine. 2. Ensure that the instance machines are networked and visible to each other over TCP/IP. 3. Create a shared directory that is visible to all the instance machines. Mount this directory on each machine as, for example, /crx/shared.
dev.day.com/docs/en/crx/2-1/administering/cluster.html 5/10

4/4/12

CRX Clustering

In production environments, the shared directory is usually located on a dedicated machine. During development and testing, you may choose to put the shared directory on one of the instance machines. 4. After CRX is installed, networked, and a shared directory is created, do the following on each CRX instance: 5. Click Repository Management on the startup page. The CRX console appears. 6. Click Log in and log in as an administrator (The default adminstrator account is login: admin, password: admin, for example). 7. Click Repository Configuration. The Repository configuration page opens. 8. Click Cluster. The Clustering page opens.

9. Assuming that the shared directory has been mounted on each machine at /crx/shared, enter that path in the Shared Path field and click Join.

10. The console asks you to log in again. After you have done so, that instance is part of the cluster.
dev.day.com/docs/en/crx/2-1/administering/cluster.html 6/10

4/4/12

CRX Clustering

You can also watch a screencast of the cluster installation process (www.adob e.comhttp://www.day.com/content/day/en/products/web _content_management/screencastinstallcluster.html) . When clustering is set up in this way, the system takes care of configuring many of the required parameters. If you would like more control over these parameters, you can manually configure them.

Cluster Installation for Large Repositories

If your repository is large, using the CRX GUI (Join Cluster functionality) may take a long time because the search index needs to be rebuilt. The quickest way to create a new cluster node in that case is to do the following: 1. Clone the CRX instance by doing the following: a. Stop the repository. b. Delete all files named listener.properties (if existing) in: crx-quickstart/repository/shared/journal/control crx-quickstart/repository/shared/version/control crx-quickstart/repository/shared/workspaces/<workspaceName>/control c. Delete all .lock files (if existing). d. Delete all optimize*.tar files (if existing). e. Delete all lock.properties files (if existing). f. Delete all files below crx-quickstart/repository/shared/journal. g. Delete all crx-quickstart/repository/revision.log files. h. Move the folder crx-quickstart/repository/shared to a shared filesystem. i. Copy all files (except the shared folder) to the target machine where the new instance will be running. j. Delete the file crx-quickstart/repository/cluster_node.id on the new instance. k. If running on the same machine, change the port of the cloned instance. 2. For each instance, replace the local crx-quickstart/repository/shared folder with a symlink pointing to the shared folder on the file share. 3. Reconfigure the journal settings in all repository.xml files (if needed) as described in the following sections for non-clustered and clustered environments: In a non-clustered environment where CRX is running standalone, configure the maximum size of a Journal log file to 100 MB and limit the number of allowed files to 1.

<ora cas"o.a.r.oejunlFlJunl> Junl ls=cmdycxcr.ora.ieora" <aa nm=saePt"vle"{e.oe/hrd/ prm ae"hrdah au=$rphm}sae"> <aa nm=mxmmie vle"0870"/ prm ae"aiuSz" au=14560 > <aa nm=mxmmie"vle""/ prm ae"aiuFls au=1 > <Junl /ora>

If CRX is running in a cluster with other CRX instances, adjust the number of rotated Journal log files. The Journal basically keeps track on every save operation of the repository, so depending on how long a slave cluster-node can be offline before being able to catch up with the latest revision of the master cluster-node instance, the following two parameters can be configured to match requirements: m x m m g : maximum age of a Journal log-file before it gets removed aiuAe
dev.day.com/docs/en/crx/2-1/administering/cluster.html 7/10

4/4/12

CRX Clustering

m x m m i e : maximum number of allowed log-rotation aiuFls

<ora cas"o.a.r.oejunlFlJunl> Junl ls=cmdycxcr.ora.ieora" <aa nm=saePt"vle"{e.oe/hrd/ prm ae"hrdah au=$rphm}sae"> <aa nm=mxmmie vle"0870"/ prm ae"aiuSz" au=14560 > <aa nm=mxmmg"vle"1"/ prm ae"aiuAe au=PM > <aa nm=mxmmie"vle"0 / prm ae"aiuFls au=1" > <Junl /ora>

After this procedure is finished, check the following: Start all cluster instances. On the new instance, check the configurations of the following and adapt if needed: replication agents admin passwords custom application-specific configuration settings

Manual Configuration
The cluster-specific parameter configuration is found in the file <r-ntl>cxqiktr/eoioyrpstr.m cxisal/r-ucsatrpstr/eoioyxl in the section < l s e > Cutr. In a fresh, default installation, it should look like this:

<lse> Cutr <ora cas"o.a.r.oejunlFlJunl> Junl ls=cmdycxcr.ora.ieora" <aa nm=saePt"vle"{e.oe/hrd/ prm ae"hrdah au=$rphm}sae"> <Junl /ora> <Cutr /lse>

By default only the s a e P t parameter is defined. This is this parameter that is changed when you add an hrdah instance to a cluster through the GUI. Additional parameters can be defined to fine tune the behavior of the cluster. The < l s e >section with all Cutr permitted paremeters present would look like this:

<lse snDly"00> Cutr ycea=20" <ora cas"o.a.r.oejunlFlJunl> Junl ls=cmdycxcr.ora.ieora" <aa nm=saePt"vle"c/r-ucsatrpstr/hrd / prm ae"hrdah au=/qcxqiktr/eoioysae" > <aa nm=mxmmie vle"0870"/ prm ae"aiuSz" au=14560 > <aa nm=mxmmg"vle" / prm ae"aiuAe au=" > <aa nm=mxmmie"vle""/ prm ae"aiuFls au=0 >
dev.day.com/docs/en/crx/2-1/administering/cluster.html 8/10

4/4/12

CRX Clustering

<aa nm=peerdatr vle"as"/ prm ae"rfreMse" au=fle > <aa nm=prLs"vle" / prm ae"otit au=" > <aa nm=bnAdes vle" / prm ae"iddrs" au=" > <aa nm=cnetieu"vle"00 / prm ae"oncTmot au=20" > <aa nm=sceTmot vle"00"/ prm ae"oktieu" au=600 > <aa nm=oeainieu"vle"00"/ prm ae"prtoTmot au=300 > <Junl /ora> <Cutr /lse>

The following table describes the function of each parameter. Parameter Description By default, a cluster instance reads the journal and updates its state (including its cache) every 5000 milliseconds (5 seconds). To use a different value, set the attribute syncDelay in the <Cluster> tag. Note: This attribute belongs directly in the <Cluster> tag. It is not a <param> within the contained <Journal> tag. sharedPath maximumSize Mandatory argument specifying the shared directory. Maximum size of a single journal file before it will get rotated. Default: 104857600 bytes (100 MB). Age specified as duration in ISO 8601 or plain format. Journal files exceeding this age will get removed. If this parameter is not specified, this option is disabled (age is not checked). If the parameter is set to an empty string (""), only one Journal log file is kept. The default in CRX is "P1M", which means files older than one month are deleted. Number of rotated journal files kept around. Files exceeding this limit will be removed. A value of 0 disables this option. Default: 0. Flag indicating whether this cluster node should be a preferred master. If this flag is set, the node will become the master upon startup. Default: false. The list of ports to use in a master node. By default, any free port is used. When using a firewall, open ports must be listed. A list of ports or ranges is supported, for example: 9100-9110 or 9100-9110,9210-9220. Default: 0 (any port). Set this parameter if synchronizing among cluster nodes should be done over a specific network interface. By default, all network interfaces are used. Default: empty (use all interfaces). Timeout in milliseconds that a client will wait when initially connecting to a server instance before aborting. Default: 2000. Socket timeout in milliseconds for both reading and writing over a connection from a client to a server. After this timeout period has elapsed with no activity, the connection is dropped. Default: 60000 (1 minute). If you have very long transactions, use 600000 (10 minutes). Operation timeout in milliseconds. After a client or server has locked the journal and this timeout has passed with no record being appended, the lock to the journal is automatically released. Default: 30000 (30 seconds). If you have very long transactions, use 600000 (10 minutes).
9/10

syncDelay

maximumAge

maximumFiles preferredMaster

portList

bindAddress

connectTimeout

socketTimeout

operationTimeout

dev.day.com/docs/en/crx/2-1/administering/cluster.html

4/4/12

CRX Clustering

Configuring Database Storage

The file-based T r e s s e c M n g ralongside the default file-based storage for journal and data store aPritneaae compare favorably to most commercial databases in terms of performance. Nonetheless, you can substitute database storage for this default arrangement. Changing the persistence manager, data store, and journal implementions in a clustered system is performed in the same way as in a single-instance installation (of course, the change has to be made on all CRX instances in the cluster). A number of alternative persistence managers, data store implementions and journal implementations are available, both from Day Software and from other sources such as the Apache Jackrab b it project. Configuration of these mechanisms is covered in Persistence Managers (www.adob e.com/docs/en/crx/21/administering/persistence_managers.html) .

Maintaining your Cluster

After installing and configuring CRX, you may need to perform some maintenance tasks in certain situations:
If you make configuration changes in CRX, you need to apply those configuration changes to all cluster nodes. If one cluster-node becomes unavailable, for example hardw are issues or out-of-memory problems, you need to restore the cluster by either restoring the backup of that cluster or cloning the master node.

(www.adobe.comhttp://www.adobe.com/go/gfooter_truste_logo) Copyright 2012 Adobe Systems Incorporated. All rights reserved. Use of this website signifies your agreement to the Terms of Use (www.adobe.comhttp://www.adobe.com/go/gfooter_terms_of_use) and Online Privacy Policy (www.adobe.comhttp://www.adobe.com/go/gfooter_privacy_policy) .

dev.day.com/docs/en/crx/2-1/administering/cluster.html

10/10