You are on page 1of 33

SAP High-Performance Analytic Appliance 1.

0
(HANA 1.0)

Architecture
Objectives

•Aftercompleting this presentation, you will be able to understand the:


• Architecture of HANA 1.0
• Concept of MVCC
• Concept of Column Store
• Concept of Row Store
• Persistance Layer of HANA 1.0
• Concept of Backup & Recovery

© 2011 SAP AG. All rights reserved. 2


Agenda

Architecture
Multi Version Concurrency Control MVCC
Row Store
Column Store
Persistence Layer
Backup & Recovery

© 2011 SAP AG. All rights reserved. 3


Architecture
Architecture Overview
In Memory Computing Engine (IMCE) and Surroundings

MS Excel Clients (planned, e.g.) BI4 Explorer


IMCE Studio
Administration Modeling Dashboard SAP BI4 universes
BI4 Analysis
Design (WebI,...)

ERP
Replication In-Memory Computing Engine
Agent Session Management

Request Processing / Execution Control Transaction


ERP DB Log
Manager
Replication SQL Parser MDX
Server SQL Script Calc Engine Authorization
SAP Business Objects BI4 Manager
Load Relational Engines
Data SBO BI4 Controller Metadata
Row Store Column Store
Services Information Manager
Designer Design Tool
SBO BI4 Persistence Layer Logger
Page Management
Data servers
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems
SAP
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 5


Architecture Overview
In Memory Computing Engine (IMCE) and Surroundings

MS Excel Clients (planned, e.g.) BI4 Explorer


IMCE Studio In-Memory Computing Engine
Administration Modeling Dashboard SAP BI4 universes
Session Management
Design (WebI,...)
BI4 Analysis

ERP
Replication In-Memory Computing Engine
Request
Processing / Execution
Agent Management Transaction
Control
Session
Replication
SQL Parser MDX Manager
Server Log Request Processing / Execution Control Transaction
ERP DB SQL Script Calc Engine
SQL Parser MDX
Manager
Authorization
Replication
Server SQL Script ManagerAuthorization
Calc Engine
SAP Business Objects BI4 Relational Engines Manager
Load Column
Relational Engines Metadata
Data SBO BI4 Row Store
Controller Metadata
Row StoreStore Column Store
Services Information Manager Manager
Designer Design Tool
SBO BI4 Persistence
Persistence Layer Layer Logger
Data Page Management
servers
Page Management Logger
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems Disk Storage
SAP Data Volumes Log Volumes
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 6


Architecture Overview
Loading Data into HANA

IMCE Studio Modeling MS Excel Clients (planned, e.g.) BI4 Explorer


Administration Dashboard SAP BI4 universes
BI4 Analysis
Design (WebI,...)

Replication
In-Memory Computing Engine
Agent
Session Management

Log Request Processing / Execution Control Transaction


ERP DB
Manager
Replication SQL Parser MDX
Server SQL Script Calc Engine Authorization
Manager
Business Objects Enterprise Load Relational Engines
Controller Metadata
Data SBO Row Store Column Store
Information Manager
Services
Designer Design Tool
SBO BI4 Page Management Persistence Layer Logger
Data servers
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems
SAP
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 7


Architecture Overview
Data Modeling
MS Excel Clients (planned, e.g.) BI4 Explorer
IMCE Studio Dashboard SAP BI4 universes
BI4 Analysis
Administration Modeling Design (WebI,...)

ERP In-Memory Computing Engine


Replication
Agent Session Management

Log Request Processing / Execution Control Transaction


ERP DB
Manager
Replication SQL Parser MDX
Server SQL Script Calc Engine Authorization
Manager
Business Objects Enterprise Load Relational Engines
Controller Metadata
Data SBO Row Store Column Store
Information Manager
Services
Designer Design Tool
SBO BI4 Page Management Persistence Layer Logger
Data servers
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems
SAP
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 8


Architecture Overview
Reporting

IMCE Studio MS Excel Clients (planned, e.g.) BI4 Explorer


Administration Modeling Dashboard SAP BI4 universes
BI4 Analysis
Design (WebI,...)

ERP Replication In-Memory Computing Engine


Agent
Session Management

Log Request Processing / Execution Control Transaction


ERP DB
Manager
Replication SQL Parser MDX
Server SQL Script Calc Engine Authorization
Manager
Business Objects Enterprise Load Relational Engines
Controller Metadata
Data SBO Row Store Column Store
Information Manager
Services
Designer Design Tool
SBO BI4 Page Management Persistence Layer Logger
Data servers
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems
SAP
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 9


Architecture Overview
Administration

IMCE Studio MS Excel Clients (planned, e.g.) BI4 Explorer


Administration Modeling Dashboard SAP BI4 universes
BI4 Analysis
Design (WebI,...)
ERP Replication In-Memory Computing Engine
Agent
Session Management

Log Request Processing / Execution Control Transaction


ERP DB
Manager
Replication SQL Parser MDX
Server SQL Script Calc Engine Authorization
Manager
Business Objects Enterprise Load Relational Engines
Controller Metadata
Data SBO Row Store Column Store
Information Manager
Services
Designer Design Tool
SBO BI4 Page Management Persistence Layer Logger
Data servers
Services ( program
for client)
Disk Storage
Data Volumes Log Volumes
Other Source Systems
SAP
NetWeaver 3rd Party
BW

© 2011 SAP AG. All rights reserved. 10


Multi Version Concurrency
Control MVCC
MVCC: Multi Version Concurrency Control

All transactions want to access the same entry with value D

T1 Read D
Commit
T4 Change D -> A

T5 Read ???
T3 Read D

T2 Read D
T6 Read A

Time

There must be a mechanism which synchronizes theses accesses

© 2011 SAP AG. All rights reserved. 12


MVCC: Multi Version Concurrency Control

T1 update personal commit


set name = ‘Jones'
where userid =
'4711'and name =
'McEwen'
MVCC transaction level Snapshot
select * select *
from personal from personal
where name like 'M%' where name like 'M%'
T2
PNR NAME PNR NAME
Version
4711 McEwen 4711 McEwen
n
5200 Miller 5200 Miller

MVCC statement level Snapshot


select *
from personal select *
where name like 'M%' from personal
where name like 'M%'
PNR NAME
T3 PNR NAME
4711 McEwen
5200 Miller 5200 Miller
© 2011 SAP AG. All rights reserved. 13
MVCC: Multi Version Concurrency Control

Commit ID (CID)

• Reflects the commit sequence of write transactions


• Is equivalent to a commit timestamp
• Maximum CID -> last recent Commit ID
• Is used by ROW store for version consolidation

When a transaction is committed the max. CID is increased and the new value is
assigned to the committed transaction

Transaction ID (TID)
• Identifies the starting point of write transactions
• Is increased whenever a write transaction starts
• Is increased when a read transaction is transformed into a write transaction
• Is used by COLUMN store for version consolidation

The TID is assigned to write transactions as its unique identifier

© 2011 SAP AG. All rights reserved. 14


Row Store
Row Store Architecture
Row Store Block Diagram

• Row Store Block Diagram


• Transactional version memory
•Contains temporary versions
•Needed for Multi-Version
Concurrency Control (MVCC)
• Segments
•Contain the actual data (content of
row-store tables) in pages
• Page Manager
• Memory allocation
• Keeping track of free/used pages
• Version Memory Consolidation
• Think ‘garbage collector for MVCC’
• Persistence Layer
• Invoked in write operations (log)
• And in performing savepoints
checkpoint writer

© 2011 SAP AG. All rights reserved. 16


Row Store Architecture
Row Store Block Diagram

Write Operations Persisted Segment


• Mainly go into “Transactional Version • Contains data that may be seen by any
Memory” ongoing transaction
•“INSERT” also writes to Persisted Segment • Data that has been committed before any
active transaction was started)
Recent Data that
Main Memory may be
versions of
changed Write seen by all
records Operations active
transaction
Version Consolidation Transaction
s
Persisted
• Moves “visible version” al Version Segment Memory Handling
Memory
from Transaction Version
Memory into Persisted • Row store tables are
Segment (based on Read Operations linked list of memory
Commit ID) pages
• Pages are grouped
• Clears “outdated” record Version
versions from Memory in segments
Consolidation
Transactional Version • Page size: 16 kB
Memory
© 2011 SAP AG. All rights reserved. 17
Indexes for Row Store Tables
Primary Index / Row ID / Index Persistence

Each row-store table has a primary index


• Primary index maps the ROW ID and the primary key of a table
• ROW ID:
• Each record has an identifier called ROW ID
• Contains the number of the segment and the slot’s offset in this
segement
• How to find the memory page for a table record?
• Access on primary index delivers the Row ID
• The ROW ID tells about the Segment and the Page offset where the
record is located in memory
• Secondary indexes can be created if needed

Persistence of indexes in row store


• Indexes in row store only exist in memory
• No persistence of index data
• Index definition stored with table metadata
• Indexes filled on-the-fly when system loads tables into memory on
system start-up
© 2011 SAP AG. All rights reserved. 18
Column Store
In-Memory Computing Engine (IMCE)
Column Store Block Diagram

Column store optimizer & OLAP optimizer


• Create physical execution plans
• Column store optimizer is cost based
• OLAP Optimizer is rule based

Column store execution control


• Column store operations like aggregate, joins etc.
can be parallelized

Main storage
• Data structure that contains the main part of the data
• Efficient compressed data

Delta storage
• Exists only in main memory
• Change operations are stored here

History storage
• Contains data versions which are not current
• Normal queries don’t access history data

Load/Unload agent
• Loading data from persistency layer
• Preloaded on startup or loaded on demand

© 2011 SAP AG. All rights reserved. 20


Column Store
Highlights

Storage separation (Main & Delta) Write Operations


•Enables high compression and high • Only in delta storage because write
write performance at the same time optimized.
• The update is performed by inserting a
new entry into the delta storage.
Compresse Main Memory Write
d and Write optimized
Read Operation
optimized s
Read Operations
Data Compression in Main Main Delta • Always have to read from both
Storage main & delta storages and merge
• Compression by creating Read the results.
Operations
dictionary and applying further • IMCE uses multi version
compression methods concurrency control (MVCC) to
• Speed up ensure consistent read operations.
• Data load into CPU cache Delta Merge Operation
• Equality check Search • To move changes in delta storage into the main
• The compression is computed storage because main storage is read optimized...
during delta merge operation. • Happens asynchronously. (Several events to trigger it.)
© 2011 SAP AG. All rights reserved. 21
Row Store vs. Column Store
When to Use Which store

Modeling only possible for column tables


• Thisanswers the frequently asked question:
“where should I put a table – row store or column store”
• Modeller only works with column tables
• Replication server creates tables in column store per default
• Data Services creates tables in column store per default
• SQL to create column table: “CREATE COLUMN TABLE ...”

•(HANA) System tables are created where they fit best


• Administrative tables in Row Store:
• Schema SYS Caches, administrative tables of IMCE...
• Tables from statistics server
• Administrative tables in column store:
• Schema _SYS_BI metadata of created views + Master data for MDX
• Schema _SYS_BIC Some generated tables for MDX
• Schema _SYS_REPO e.g. Lists of active/modified versions of models.
© 2011 SAP AG. All rights reserved. 22
Persistence Layer
Persistence Layer in In-memory Comp. Engine
Purpose and Scope

Why does an in-memory database need a persistence layer


Main Memory is volatile. What happens upon
• Database restart?
• Power outage?
• ...
• Data need to be stored in a non-volatile way

Backup and restore

• IMCE offers one persistence layer which is used by row store and column store
• Regular “savepoints” full persisted image of DB at time of savepoint
• Logs capturing all DB transactions since last savepoint (redo logs and undo logs written)
restore DB from latest savepoint onwards
• Create Snapshots ( backup)

© 2011 SAP AG. All rights reserved. 24


Persistence Layer in In-memory Comp. Engine
Data is Saved to Disk in Intervals

SAP In-Memory Computing Engine

Memory

Data

Information about Regular automatic


data changes savepoints

Log Data
Volume Volumes
Persistent Storage

© 2011 SAP AG. All rights reserved. 25


Persistence Layer in In-memory Comp. Engine
Savepoint – Writing Data in IMCE

Page Buffer

Log queue
Data Cache Other Data pages
Converter Data Pages of virtual files

Savepoint Coordinator

4713
DATA Redo
DATA DATA DATA & Log
4712
& & & Undo
Undo Undo Undo
© 2011 SAP AG. All rights reserved. 26
Persistence Layer in In-memory Comp. Engine
System Restart
Reboot or Power failure deletes in-memory data
• System is normally restarted („lazy“ restart to keep downtime short: tables with preload
flag + subsequently requested tables are loaded first)
• System is restored to the state just before the failure (except non-committed
transactions)
• Used for recovery:
• Last data savepoint
• Log between the last data savepoint and the time of failure
(contains the data changes of all commited transactions up to that point )

Time

1 2 3
Data savepoint Log written Power failure
to persistent storage to persistent storage
(committed transactions)
© 2011 SAP AG. All rights reserved. 27
Backup & Recovery
Backup & Recovery
Save to External Backup Destinations

• Data backup:
• From persistent storage to external backup destinations
• Using database functions (SAP in-memory computing studio)
• Log backup:
• Not supported in SAP HANA 1.0
• Configuration backup
• Manual copy of configuation files to external backup destination
Persistent Storage

Log Data conf


Volume Volumes

D Configuration Backup
DataDBackup
© 2011 SAP AG. All rights reserved. 29
Backup & Recovery
Recovery Scenario – Disk Failure (Data Volume)

Disk failure (data volumes are damaged)


• System is restored to the state just before the failure (except non-committed
transactions)
• Used for recovery:
• Last data backup
• Log since the last data backup
Assumption: log area undamaged, all log entries still available (not yet overwritten)

Time

2 3
1
Log written Disk failure
Data backup
to persistent storage (data volumes)
to external backup
(committed transactions)
destination

© 2011 SAP AG. All rights reserved. 30


Backup & Recovery
Feature Overview

SAP HAnA 1.0

Data Backup

Log Backup

Recovery to last Data Backup

Recovery to status before crash


( )
If log is not damaged

Point in Time Recovery

© 2011 SAP AG. All rights reserved. 31


Summary

In this presentation, you learned about the:


• Architecture of HANA 1.0
• Concept of MVCC
• Concept of Column Store
• Concept of Row Store
• Persistance Layer of HANA 1.0
• Concept of Backup & Recovery

© 2011 SAP AG. All rights reserved. 32


© 2011 SAP AG. All rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal
without the express permission of SAP AG. The information contained herein may be Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services
changed without prior notice. mentioned herein as well as their respective logos are trademarks or registered trademarks
of Business Objects Software Ltd. Business Objects is an SAP company.
Some software products marketed by SAP AG and its distributors contain proprietary
software components of other software vendors. Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase
products and services mentioned herein as well as their respective logos are trademarks or
Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft
registered trademarks of Sybase, Inc. Sybase is an SAP company.
Corporation.
IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, All other product and service names mentioned are the trademarks of their respective
System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, companies. Data contained in this document serves informational purposes only. National
z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, product specifications may vary.
PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, The information in this document is proprietary to SAP. No part of this document may be
OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, reproduced, copied, or transmitted in any form or for any purpose without the express prior
RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, written permission of SAP AG.
Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered
This document is a preliminary version and not subject to your license agreement or any
trademarks of IBM Corporation.
other agreement with SAP. This document contains only intended strategies, developments,
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. and functionalities of the SAP® product and is not intended to be binding upon SAP to any
particular course of business, product strategy, and/or development. Please note that this
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or
document is subject to change and may be changed by SAP at any time without notice.
registered trademarks of Adobe Systems Incorporated in the United States and/or other
countries. SAP assumes no responsibility for errors or omissions in this document. SAP does not
Oracle is a registered trademark of Oracle Corporation. warrant the accuracy or completeness of the information, text, graphics, links, or other items
contained within this material. This document is provided without a warranty of any kind,
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. either express or implied, including but not limited to the implied warranties of
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are merchantability, fitness for a particular purpose, or non-infringement.
trademarks or registered trademarks of Citrix Systems, Inc. SAP shall have no liability for damages of any kind including without limitation direct,
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World special, indirect, or consequential damages that may result from the use of these materials.
Wide Web Consortium, Massachusetts Institute of Technology. This limitation shall not apply in cases of intent or gross negligence.

Java is a registered trademark of Sun Microsystems, Inc. The statutory liability for personal injury and defective products is not affected. SAP has no
control over the information that you may access through the use of hot links contained in
JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for these materials and does not endorse your use of third-party Web pages nor provide any
technology invented and implemented by Netscape. warranty whatsoever relating to third-party Web pages.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer,
StreamWork, and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and other
countries.

© 2011 SAP AG. All rights reserved. 33