LondonR - Simulating An Enterprise Data Science Pipeline On Your Laptop - Ed Bullen - 20161024

Simulating a Data Science Pipe-Line on your Laptop
Ed Bullen, Oracle UK
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 1
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle
Open Source Projects at Oracle
http://openjdk.java.net/projects/graal/
Motivation
Data Science and Engineering
MATHS SCIENCE
PROGRAMMING ENGINEERING
A Simple Data Science Pipe-Line
Engineering a Data Processing Pipe-Line
Source Raw
Pre-Process Summarise Consumers
Data
Hadoop Map Hadoop Reduce HDFS, Hive

UK Crime
Data
Python Python R Studio
Hadoop Streaming API

A Simple Approach – well known (not latest cutting-edge tech) … but …
Stable – effective, easy to implement, static technology components
The Oracle Big Data Lite VM
Free, Simple to Install – Fast Track Access to Hadoop Stack Technologies
Main Download Site:
http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html
Personal Blog – Additional Assistance and Network Configuration Tips:

https://pygot.wordpress.com/2016/07/08/getting-started-with-the-oracle-hadoop-vm/
Map Reduce
A Quick Refresher – Map, Shuffle, Reduce HDFS
Node 1 - MAP Node 2 - MAP
=3 =1 =2 =2
Node 1 - REDUCE Node 2 - REDUCE
Deploy Python and R Straight to Hadoop
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \
-file my_python_mapper.py -mapper "python my_python_mapper.py" \
-file my_python_reducer.py -reducer "python my_python_reducer.py" \
-input /user/hadoopuser/source_HDFS_dir \
-output /user/hadoopuser/dest_HDFS_dir
Mapper Hadoop SORT Reducer

Hadoop Hadoop
STD-IN Executed in STD-OUT and STD-IN Executed in OUT
HDFS HDFS
OS Shell NODE PARTITION OS Shell
Sample Code – Python Map-Reduce for UK Crime Data
https://data.police.uk/data/
Crime ID,Month,Reported by,Falls within,..,LSOA, Crime type...

,2012-01,Avon and Somerset Constabulary,..,E01014399, Anti-social behaviour...
…
…
,2012-02,Avon and Somerset Constabulary,..,E01014400, Burglary...
…
DATE, LSOA , LSOA_Name , crime[0], crime[1], crime[2], ... crime[n]

2012-01,e01014399, LSOA Desc , 1 , 2 , 0 , ... 4
2012-02,e01014400, LSOA Desc , 1 , 2 , 0 , ... 4
Example Code on GitHub

https://github.com/edbullen/py-mapred
Accessing the Data in R Studio
A Simple Approach
# Load Libraries and setup Java ClassPath # Connect to Hive datastore in Hadoop
library("DBI") .jinit(classpath=cp)
library("rJava")
library("RJDBC") drv <- JDBC("org.apache.hive.jdbc.HiveDriver"
, "hive-jdbc.jar")
# Java ClassPath for HIVE Access conn <- dbConnect(drv
cp = c("./hive-jdbc.jar" , "jdbc:hive2://bigdatalite:10000/default"
, "./hadoop-common.jar" , "oracle", "")
, "./libthrift-0.9.2.jar"
, "./hive-service.jar" # Query Data using SQL
, "./httpclient-4.2.5.jar" ukcrimesum <- dbGetQuery(conn
, "./httpcore-4.2.5.jar“ , "select * from ukcrimesum")
, "./hive-jdbc-standalone.jar")
Personal Blog – Connecting R Studio to Hadoop via Hive:

https://pygot.wordpress.com/2016/10/13/connecting-r-studio-to-hadoop-via-hive/
Analysis of the Data-Set
A quick first-pass…
Correlation and Clustering
ukcrimesum <- dbGetQuery(conn

, "select * from ukcrimesum")
#which crimes show correlation?

crimesM <- data.matrix(ukcrimesum[,4:17])
corM <- cor(crimesM)

diag(corM) <- 0
heatmap(corM)
Seasonality
monthagg <- aggregate(cbind(robbery

, burglary
, bicycle_theft
, social) ~ date
, data=monthcrimes
, FUN=sum)
centered <- cbind(monthagg$date

, as.data.frame(apply(monthagg[-1]
, 2
, function(y) y - mean(y))) )
par(mfrow = c(4,1))
attach(centered)
for (name in names(centered)[-1] ) {

barplot(as.vector(centered[name][,1])
, main = paste(name))
}
Mapping
library("rgeos")
library("maptools")
ukshapefileDETAIL <- "./LSOA_2011_EW_BFE_V2.shp"

ukmap <- readShapeSpatial(ukshapefileDETAIL)
lonmap <- ukmap[match(lonLSOA, ukmap@data$LSOA11CD),]
loncrime <- dbGetQuery(conn, "select LSOA,

sum(total_classified) from ukcrimesum
where date in <...>
and lsoa in <...> group by LSOA")
#Combined Map Data (shapeFile) with added data

lonmap.crime <- SpatialPolygonsDataFrame(lonmap
,loncrime ,match.ID=FALSE)
plot(lonmap.crime
, col = countcols[findInterval(counts
, breaks, all.inside = TRUE)]
, axes = FALSE
, border = "transparent“
, main = "2015 Total Crimes" )
Mapping
library("rgeos")
library("maptools")
ukshapefileDETAIL <- "./LSOA_2011_EW_BFE_V2.shp"

ukmap <- readShapeSpatial(ukshapefileDETAIL)
lonmap <- ukmap[match(lonLSOA, ukmap@data$LSOA11CD),]
loncrime <- dbGetQuery(conn, "select LSOA,

sum(bicycle_theft) from ukcrimesum
where date in <...>
and lsoa in <...> group by LSOA")
#Combined Map Data (shapeFile) with added data

lonmap.crime <- SpatialPolygonsDataFrame(lonmap
,loncrime ,match.ID=FALSE)
plot(lonmap.crime
, col = countcols[findInterval(counts
, breaks, all.inside = TRUE)]
, axes = FALSE
, border = "transparent"
, main = "2015 Bicycle Theft" )
Thank You
ed.bullen@oracle.com
Social Media and Blog: ** all personal views, not representing my employer **
@bullened
http://pygot.wordpress.com
http://github.com/edbullen
https://www.meetup.com/Oracle-UK-BigData/

LondonR - Simulating An Enterprise Data Science Pipeline On Your Laptop - Ed Bullen - 20161024

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

LondonR - Simulating An Enterprise Data Science Pipeline On Your Laptop - Ed Bullen - 20161024

Uploaded by

Copyright:

Available Formats

Simulating a Data Science Pipe-Line on your Laptop

Hadoop Map Hadoop Reduce HDFS, Hive

Hadoop Streaming API

Personal Blog – Additional Assistance and Network Configuration Tips:

Node 1 - MAP Node 2 - MAP

Mapper Hadoop SORT Reducer

Crime ID,Month,Reported by,Falls within,..,LSOA, Crime type...

DATE, LSOA , LSOA_Name , crime[0], crime[1], crime[2], ... crime[n]

Example Code on GitHub

Personal Blog – Connecting R Studio to Hadoop via Hive:

ukcrimesum <- dbGetQuery(conn

#which crimes show correlation?

corM <- cor(crimesM)

monthagg <- aggregate(cbind(robbery

centered <- cbind(monthagg$date

for (name in names(centered)[-1] ) {

ukshapefileDETAIL <- "./LSOA_2011_EW_BFE_V2.shp"

loncrime <- dbGetQuery(conn, "select LSOA,

#Combined Map Data (shapeFile) with added data

ukshapefileDETAIL <- "./LSOA_2011_EW_BFE_V2.shp"

loncrime <- dbGetQuery(conn, "select LSOA,

#Combined Map Data (shapeFile) with added data

You might also like