Professional Documents
Culture Documents
Agenda
My Background Documentum xPlore Context and History Overview of Documentum xPlore Tips and Observations on IO and Host Virtualization
My Background
Ed Buech Information Intelligence Group within EMC EMC Distinguished Engineer & xPlore Architect Areas of expertise
Content Management (especially performance & scalability) Database (SQL and XML) and Full text search Previous experience: Sybase and Bell Labs
Built over EMC xDB, Lucene, and leading content extraction and linguistic analysis software
1996
2005
2010
DCTM client
Without Full Text in a Documentum deployment a DQL query will be directed to the RDBMS
DQL is translated into SQL
search
Documentum client
Content Server
DQL for search can be directed to the full text engine instead of RDBMS (FTDQL) This allows query to be serviced by xPlore In this case DQL is translated into xQuery (the query language of xPlore / xDB)
xQuery
Metadata + content
Applications need fine control over not only search criteria, but also result sets
10
Users want the power of Information Retrieval on their structured queries Data Management, HA, DR shouldn t be an after-thought When possible, operate within standards Lucene is not a database. Most Lucene applications deploy with databases.
11
Lessons Learned
Fit to use-case
Fit to use-case
Relational DB technology
JOINs
Fit to use-case
Transactions
Fit to use-case
Relational DB technology
Documentum xPlore
Bring
best-of-breed
XML
Database
with
powerful
Apache
Lucene
Fulltext
Engine
Provides
structured
and
unstructured
search
leveraging
XML
and
XQuery
standards
Designed
with
Enterprise
readiness,
scalability
and
ingesCon
Advanced
Data
Management
funcConality
necessary
for
large
scale
systems
Industry
leading
linguisCc
technology
and
comprehensive
format
lters
Metrics
and
AnalyCcs
xPlore API
Indexing Services Content Processing Services Analytics Search Services Node & Data Management Services Admin Services
xDB API xDB Query Processing& Optimization xDB Transaction, Index & Page Management
Full Transactional Database Query Language: XQuery with full text extensions
= xDB Index = xDB xml file (dftxml, tracking xml, status, metrics, audit)
= xDB segment
C
B
Lucene Integration
Transactional
Non-committed index updates in separate (typically in memory) lucene indexes Recently committed (but dirty) indexes backed by xDB log Query to index leverages Lucene multi-searcher with filter to apply update/delete blacklisting
Lucene indexes managed to fit into xDB s ARIES-based recovery mechanism No changes to Lucene
Goal: no obstacles to be as current as possible
19
22
Recommendations:
Size correctly, don t accept insufficient resources Test pre-production environments
When mapped directly to physical disks then this could concentrate I/ O to fewer than a desired set of drives. High-end SAN s like Symmetrix can handle this situation with virtual LUN s
25
Fast, efficient mobility Maintains replication and quality of service during relocations Supports up to thousands of concurrent VP LUN migrations Recommendation: work with storage technicians to ensure backend storage has sufficient I/O
Fibre Channel
600 GB 15K RAID 1
Tier 2
V L U N
SATA
2 TB RAID 6
Recommendations:
Track actual capacity vs. planned
Vmware: track number of times your VM is denied CPU SANs: track % I/O utilization vs. number of I/O s
For Vmware leverage guaranteed minimum resource allocations and/or allocate to nonoverloaded HW
Pages-in, Pages-out
Can indicate over subscription of memory
28
Sample %Ready for a production VM with xPlore deployment for an entire week
16% 14%
12%
10% 8% 6%
In this case Avg resp time doubled and max resp time grew by 5x
4%
2% 0%
29
30
20 sec interval
31
Sharing I/O capacity If Multiple VM s (or servers) are sharing the same underlying physical volumes and the capacity is not managed properly
then the available I/O capacity of the volume could be less than the theoretical capacity
This can be seen if the OS tools show that the disk is very busy (high utilization) while the number of I/Os is lower than expected
Volume for other application Volume for Lucene application
Both volumes spread over the same set of drives and effectively sharing the I/O capacity
Also, High SAR I/O wait time might be an indication of slow disks
On Windows
Leverage the Windows Performance Monitor Objects: Processor, Physical Disk, Memory
-s 1024 means that 2 GB files will be created -o_direct means that direct I/O (by-passing buffer cache) will be done -v 10 means that 10 different 2GB files will be created. -p 10 means that 10 different threads will query those files Bonnie is an open source disk I/O driver tool for Linux that can be useful for pretesting Linux disk environments prior to an xPlore/Lucene install.
This output means that the random read test saw 735 random I/ O s per sec at 15% CPU busy
Notice that at 200+ I/Os per sec the underlying volume is 80% busy. Although there could be multiple causes, one could be that some other VM is consuming the remaining I/O capacity (735 209 = 500+). tps 206.10 kB_read/s 2402.40 kB_wrtn/s 0.80 kB_read 24024 kB_wrtn 8
SAR u output: 09:29:17 PM 09:29:27 PM 09:29:27 PM 09:29:27 PM 09:29:27 PM 09:29:27 PM CPU all 0 1 2 3 %user 41.37 62.44 30.90 36.35 35.77 %nice 0.00 0.00 0.00 0.00 0.00 %system 5.56 10.56 4.26 3.96 3.46 %iowait 29.86 25.38 35.56 30.76 27.64 %steal 0.00 0.00 0.00 0.00 0.00 High I/O wait %idle 23.21 1.62 29.28 28.93 33.13
Recommendation:
Periodic warmup
non-intrusive
Goal:
Pre-cache portions of the index to improve response time in scenarios Reboot, buffer cache contention, & vm memory contention
Stored fields (facets and node-ids) 1st doc N-th doc xDB XML store (contains text for summary)
Stored fields Xdb node-id plus facet / security info Security lookup (b-tree based) xDB XML store (contains text for summary)
Small structure
Small number for result window Xdb docs with text for summary
FinalFacet calc values over thousands of results Res-1 - sum Res-2 - sum Res-3 - sum : : Res-350-sum
N-th doc
N-th doc
xPlore caches
42
Test Env
32 GB memory Direct attached storage (no SAN) 1.4 million documents Lucene index size = 10 GB Size of internal parts of Lucene CFS file
Stored fields (fdt, fdx): 230 MB (2% of index) Term Dictionary (tis,tii): 537 MB (5% of index) Positions (prx): 8.78 GB (80% of index) Frequencies (frq) : 1.4 GB (13 % of index)
Nothing cached Stored fields cached Term dict cached Positions cached Frequencies cached Entire index cached
Linux buffer cache cleared completely before each run Resp as seen by final user in Documentum Facets not computed in this example. Just a result set returned. With Facets response time difference more pronounced. Mileage will vary depending on a series of factors that include query complexity, compositions of the index, and number of results consumed
44
Other Notes
Caching 2% of index yields a response time that is only 60% greater than if the entire index was cached.
Caching cost only 9 secs on a mirrored drive pair Caching cost 6800 large sequential I/O s vs. potentially 58,000 random I/O s
Contact
Ed Buech
edward.bueche@emc.com http://community.emc.com/people/Ed_Bueche/blog http://community.emc.com/docs/DOC-8945
46