Data Warehouse/Data Mart

Components
Concepts
Characteristics

Overview
Operational vs Informational Systems
Data Warehouse components
Data Marts




Basic Data Warehouse
Architecture
Copyright 1997, Enterprise Group, Ltd.
Source OLTP
Systems
Subset Data Marts
Enterprise
Data
Warehouse
One Version
of the Truth
Operational vs. Informational
Systems
Information Access Today
Operational
Systems
Order
Entry

Manf.
Operational vs. Informational
Systems
Information Access Today
Operational
Systems
Informational
Systems
Operational vs. Informational
Systems
Most of the advances in end-user programming have run into
difficulty in actually accessing data that exists in backbone,
operational data bases.
Operational data bases have a very, very long life. Large operational
systems are converted from one technology to a more advanced one
very infrequently (typically every eight to twenty years).
Therefore, why not create specific DBs whose role was to make large
scale end user access easy to isolate the operational DBs, i.e. a Data
Warehouse

Operational vs. Informational
Systems
Operational
Systems
Informational
Systems
Information
Delivery System
Operational vs. Informational
Systems
Operational
Systems
Informational
Systems
Information
Delivery System
Data
Warehouse
Operational vs. Informational
Systems
Operational
Systems
Informational
Systems
Information
Delivery System
Data
Warehouse
Operational vs. Informational
Systems
Operational
Systems
Information
Delivery System
Data
Warehouse
Informational
Systems
Operational vs. Informational
Systems
Operational
Systems
Information
Delivery System
Data
Warehouse
Informational
Systems
Notice that one of the big impacts of
Data Warehousing is to eliminate large
numbers of existing DSS systems!
Y2000 will make this essential!!!
Operational vs. Informational
Systems
Operational
Systems
Information
Delivery System
Data
Warehouse
Informational
Systems
Data
Marts
Data Mart
Layer
Presentation/
Desktop Access
Layer
Meta-data Repository Layer
Warehouse Management Layer
Core DW Layer
Data Staging and
Quality Layer
Data
Access
Layer
Operational Data
Layer
External Data
Layer
Data Feed/
Data Mining/
Indexing Layer
Virtual DW
Coarse DW
Central DW
Distributed DW
Application Messaging (Transport) Layer
Internet/Intranet Layer
direct queries
virtual queries
ad hoc queries
1
2a
2b
2c
3
4
5 6
7
8
9
10
11
Un it e d S t a t e s b y S a le s
$ 1 0 , 3 4 0 t o $ 1 0 , 3 5 0 ( 1 ) $ 8 , 7 3 0 t o $ 1 0 , 3 4 0 ( 2 ) $ 4 , 3 2 0 t o $ 8 , 7 3 0 ( 2 ) $ 1 , 1 0 0 t o $ 4 , 3 2 0 ( 1 ) $ 7 3 0 t o $ 1 , 1 0 0 ( 3 )
Unit ed St at es
$ 1 1 , 0 0 0
Sa le s
North Ameri ca
Non-operational
Data
Layer
Data Marts vs Data Warehouses
Data Mart
Layer
Presentation/
Desktop Access
Layer
Meta-data Repository Layer
Warehouse Management Layer
Core DW Layer
Data Staging and
Quality Layer
Data
Access
Layer
Operational Data
Layer
External Data
Layer
Data Feed/
Data Mining/
Indexing Layer
Central DW
Application Messaging (Transport) Layer
Internet/Intranet Layer
direct queries
virtual queries
ad hoc queries
1
2a
2b
2c
3
4
5 6
7
8
9
10
11
Un it e d S t a t e s b y S a le s
$ 1 0 , 3 4 0 t o $ 1 0 , 3 5 0 ( 1 ) $ 8 , 7 3 0 t o $ 1 0 , 3 4 0 ( 2 ) $ 4 , 3 2 0 t o $ 8 , 7 3 0 ( 2 ) $ 1 , 1 0 0 t o $ 4 , 3 2 0 ( 1 ) $ 7 3 0 t o $ 1 , 1 0 0 ( 3 )
Unit ed St at es
$ 1 1 , 0 0 0
Sa le s
North Ameri ca
Non-operational
Data
Layer
Central Data Warehouse
Tracking DB
Lawson DB
Virtual Date Warehouse
A Virtual Data Warehouse approach is often
chosen when there are infrequent demands for
data and management wants to determine if/how
users will use operational data.
One of the weaknesses of a Virtual Data
Warehouse approach is that user queries a made
against operational DBs.
One way to minimize this problem is to build a
Query Monitor to check the performance
characteristics of a query before executing it.


A Coarse Data Warehouse is often chosen when the
organization has a relatively clean/new operational
system and management wants to make the operational
data more easily available for just that system.
A Central Data Warehouse
is often chosen when the organization has a clear
understanding about it Information Access needs and
wants to provide quality, integrated , information to
its knowledge workers
A Distributed Data Warehouse is similar in most respects
to a Central Data Warehouse, except that the data is
distributed to separate mini-Data Warehouses (Data
Marts )on local or specialized servers

Data Mart
Layer
Presentation/
Desktop Access
Layer
Meta-data Repository Layer
Warehouse Management Layer
Core DW Layer
Data Staging and
Quality Layer
Data
Access
Layer
Operational Data
Layer
External Data
Layer
Data Feed/
Data Mining/
Indexing Layer
Virtual DW
Coarse DW
Central DW
Application Messaging (Transport) Layer
Distributed DW
Internet/Intranet Layer
direct queries
virtual queries
ad hoc queries
1
2a
2b
2c
3
4
5 6
7
8
9
10
11
Un it e d S t a t e s b y S a le s
$ 1 0 , 3 4 0 t o $ 1 0 , 3 5 0 ( 1 ) $ 8 , 7 3 0 t o $ 1 0 , 3 4 0 ( 2 ) $ 4 , 3 2 0 t o $ 8 , 7 3 0 ( 2 ) $ 1 , 1 0 0 t o $ 4 , 3 2 0 ( 1 ) $ 7 3 0 t o $ 1 , 1 0 0 ( 3 )
Unit ed St at es
$ 1 1 , 0 0 0
Sa le s
North Ameri ca
Non-operational
Data
Layer
Central Data Warehouse
Data Mart
Layer
Presentation/
Desktop Access
Layer
Meta-data Repository Layer
Warehouse Management Layer
Core DW Layer
Data Staging and
Quality Layer
Data
Access
Layer
Operational Data
Layer
External Data
Layer
Data Feed/
Data Mining/
Indexing Layer
Virtual DW
Coarse DW
Central DW
Distributed DW
Application Messaging (Transport) Layer
Internet/Intranet Layer
direct queries
virtual queries
ad hoc queries
1
2a
2b
2c
3
4
5 6
7
8
9
10
11
Un it e d S t a t e s b y S a le s
$ 1 0 , 3 4 0 t o $ 1 0 , 3 5 0 ( 1 ) $ 8 , 7 3 0 t o $ 1 0 , 3 4 0 ( 2 ) $ 4 , 3 2 0 t o $ 8 , 7 3 0 ( 2 ) $ 1 , 1 0 0 t o $ 4 , 3 2 0 ( 1 ) $ 7 3 0 t o $ 1 , 1 0 0 ( 3 )
Unit ed St at es
$ 1 1 , 0 0 0
Sa le s
North Ameri ca
Non-operational
Data
Layer
Data Marts Only
Heterogeneity - The Reality
Oracle Financials
Custom
Marketing
Data
Warehouse
Packaged
Oracle
Financial
Data
Warehouse
Packaged
I2 Supply Chain
Non- Architected
Data Mart
Subset
Data Marts
i2 Supply Chain Siebel CRM
3rd Party
Data
Federated BI Architecture
Real Time
ODS
Federated
Financial
Data
Warehouse
Subset
Data Marts
Common
Staging
Area
Oracle Financials i2 Supply Chain Siebel CRM 3rd Party
Federated
Packaged
I2 Supply
Chain
Data Marts
Analytical
Applications
e-commerce
Real Time
Data Mining
and Analytics
Real Time
Segmentation,
Classification,
Qualification,
Offerings, etc.
Federated
Marketing
Data
Warehouse
Benefits of Data Warehouse
Architecture
Provides organizing framework
Gives flexibility for changes and allows
simplified maintenance
Speeds up future development by aiding
understanding of dw
Communication tool for roles and
requirements
Coordinate data marts


Primary Technical Challenge Axis
Easy Hard
Fast
Slow
Parallel
ERP DW
Finance
Custom
ERP DW
Turnkey
ERP DW
VLDB
Near
Real
Time
Marketing
Mid-Size Co.
Large Co.
Single Source
Multi-Source
Monthly
Freq
Small DB
Dirty Data
Clean Data
Prerequisites for Success
Pain driven
Sponsorship at the highest levels
Sustainable political will
Iterative methodology
Manageable scope
User driven design
Service business mindset
Sustainability