Yahoo!

White Papei

1














RBBNS Neets NoSQL
Bata Replication Between
0iacle anu NongoBB












By
Apun Biian
Y! BBA Team
Yahoo! White Papei

2
!"#$%&'(#)%"

+%",%-. ls a scalable, hlgh-performance, open source noSCL daLabase.
Mongou8 ls parL of Lhe noSCL famlly of daLabase sysLems. lnsLead of sLorlng daLa
ln Lables as done ln a "classlcal" relaLlonal daLabase, Mongou8 sLores sLrucLured
daLa as !SCn-llke documenLs wlLh dynamlc schemas (Mongou8 calls Lhe
formaL 8SCn), maklng Lhe lnLegraLlon of daLa ln cerLaln Lypes of appllcaLlons
easler and fasLer. Maln feaLures of Mongou8 are:

! uynamlc Schema deslgn, whlch decreases appllcaLlon deploymenL.
! 8epllcaLlon and Plgh AvallablllLy, mlrrors across LAns and WAns.
! AuLo Shardlng, Scales horlzonLally wlLhouL compromlslng funcLlonallLy.
! Map/8educe, flexlble aggregaLlon and daLa processlng. LLc.



/$0(12 ls an ob[ecL-relaLlonal daLabase managemenL sysLem. Cracle adheres Lo
AClu (ALomlclLy, ConslsLency, lsolaLlon & uurablllLy) properLles. 8eferenLlal
lnLegrlLy ls really one of Lhe corner sLones of Cracle, and lL ensures loglcal
conslsLency of daLa model. noL only does lL ensure conslsLency wlLhln a cerLaln
loglcal enLlLy (whlch mlghL span mulLlple rows/Lables) buL also more lmporLanLly
malnLalns cross enLlLy conslsLency. Cracle provldes a very powerful query (SCL)
language and programmlng (L\SCL) capablllLles. Cracle 8eal AppllcaLlon ClusLers
are deslgned for horlzonLal scalablllLy, hlgh avallablllLy and exLreme performance
requlremenLs.



Yahoo! White Papei

S
3'$4%52

WlLh boLh Mongou8 and Cracle havlng lLs own beneflLs, Lhere are a loL of
beneflLs ln creaLlng hybrld appllcaLlon sysLems. ?ou can have a parL of Lhe daLa
resldlng on Cracle and a parL of lL Mongou8. A use case would be havlng Lhe
dlmenslon daLa on Cracle (as lL mlghL come from varlous heLerogeneous sources,
oLher daLabases, flaL flles eLc.) and lacL daLa on Mongou8.
Mongou8 has CrldlS and Map/8educe funcLlonallLy Lo process raw daLa fasL.
Mongou8 belng a key value sLore has greaL read performance. 1hough ls my LesL l
found LhaL Lhe wrlLe ls slow ln Mongou8, lf we have repllcaLlon and shardlng
lmplemenLed.

1hls paper dlscusses Lhe varlous meLhods LhaL can be used Lo move daLa beLween
Cracle and Mongou8.
62(7"%1%,8 9#0(:

Cracle LnLerprlse LdlLlon 11.2.0.2
Mongou8 v2.0.3
erl v3.8.6
erl u8u-Cracle
Cracle Advanced Cueulng (leaLure ln Cracle LnLerprlse LdlLlon)
672 3$%(255;

Cracle provldes Lhe opLlon of queulng any lnserL/updaLe/deleLe on a Lable, whlch
can be laLer de-queued and used Lo populaLe oLher daLa sLrucLures. Cracle uLlllzes
Lhls meLhod of dolng Cracle-Lo-Cracle repllcaLlon as well. We used Lhe same
meLhod Lo queue/de-queue daLa from Cracle Lo Mongou8 for lncremenLal daLa
refresh uslng erl as an lnLerface.

lncase of bulk daLa load from Cracle Lo Mongou8, we use erl Lo dump Lhe daLa
Lo a CSv formaL flle (comma dellmlLed formaL) and load lL Lo Mongou8. Mongou8
has Lhe opLlon Lo merge exlsLlng daLa whlle bulk loadlng daLa from Cracle. Pence
Yahoo! White Papei

4
Lhls meLhod can also be used for perlod daLa repllcaLlon from
Cracle Lo Mongou8, lf we wanL Lo avold Lhe overhead on
queue/de-queue aL Cracle level.

LasLly, we can dump Lhe daLa from Mongou8 ln a CSv formaL and load lL Lo Cracle
as well.











!"#$% '()( !*+, -.),../ 01(2*. '()(3(4. (/5 6+/#+'-



BNL
Queue
Peil
BNL Be-
Queue
.CSv Bump
with Peil
NongoImpoit
.CSv using
NongoExpoit
0iacle SQL-
Loauei
0iacle NongoBB
Yahoo! White Papei

S

<052 9#'&)25;

=>? /$0(12 #% +%",%-. @241)(0#)%" A!"($2B2"#01 -0#0 C%0&?;

<%B4%"2"#5 '52&;
Mongou8: Mongou8 shell verslon: 2.0.3
Cracle: 11.2.0.2
Cracle Advanced Cueulng
erl scrlpLlng
u8u:Cracle: perl u8u for Cracle
!SCn perl module.

672 B%&21 D%$ #7)5 52#'4 )5;
8epllcaLe SCC11.LM Lable from Cracle Lo Mongou8.
use Cracle AC Lo queue lnserL, updaLe & deleLe and Lhen use perl scrlpL Lo
dequeue Lhem ln Mongou8.

/$0(12 9)&2 60E12 92#'4 D%$ @241)(0#)%";

9#24 =; 1able SLrucLure:

SCL> desc emp
name uaLa 1ype
-------------------------
LMnC nuM8L8(4)
LnAML vA8CPA82(10)
!C8 vA8CPA82(9)
MC8 nuM8L8(4)
Pl8LuA1L uA1L
SAL nuM8L8(7,2)
CCMM nuM8L8(7,2)
uL1nC nuM8L8(2)

9#24 F; CranL execuLe on u8MS_AC, u8MS_ACAuM Lo SCC11
SCL> granL execuLe on u8MS_AC Lo SCC11,
CranL succeeded.
SCL> granL execuLe on u8MS_ACAuM Lo SCC11,
CranL succeeded.
Yahoo! White Papei

6

9#24 G; SeLup Cueue 1able ln Cracle

C8LA1L 1?L LM_1 AS C8!LC1 (MSC vA8CPA82(4000)),

8LCln
-- 1PlS 8CCLuu8L C8LA1LS A CuLuL 1A8LL lC8 MLSSACLS Cl A 8LuLllnLu 1?L.
u8MS_ACAuM.C8LA1L_CuLuL_1A8LL(
CuLuL_1A8LL => 'LMCuLuL_C1',
CuLuL_A?LCAu_1?L => 'LM_1'
),

-- 1hls procedure creaLes a queue ln Lhe speclfled queue Lable.
u8MS_ACAuM.C8LA1L_CuLuL(
CuLuL_nAML => 'LMCuLuL_C',
CuLuL_1A8LL => 'LMCuLuL_C1'
),

-- 1hls procedure enables Lhe speclfled queue for enqueulng or dequeulng.
u8MS_ACAuM.S1A81_CuLuL(
CuLuL_nAML => 'LMCuLuL_C'
),
Lnu,
/

SCL> SLLLC1 name, queue_Lable,enqueue_enabled,dequeue_enabled,user_commenL l8CM
user_queues,

nAML CuLuL_1A8LL LnCuLuL uLCuLuL uSL8_CCMMLn1
---------------------------------------------------------------------------------------------------------------------
LMCuLuL_C LMCuLuL_C1 ?LS ?LS
AC$_LMCuLuL_C1_L LMCuLuL_C1 nC nC excepLlon queue

9#24 H; CreaLe procedure Lo enqueue messages/uMLs

C8LA1L C8 8LLACL 8CCLuu8L enqueue_emp(payload vA8CPA82) AS
msg1 emp_L := emp_L(nuLL),
msg_ld 8AW(16),
prlorlLy nuM8L8,
enqueue_opLlons u8MS_AC.LnCuLuL_C1lCnS_1,
message_properLles u8MS_AC.MLSSACL_8CL81lLS_1,
8LCln
msg1.msg := payload,
Yahoo! White Papei

7
message_properLles.prlorlLy := 1,
u8MS_AC.LnCuLuL(
queue_name => 'LMCuLuL_C',
enqueue_opLlons => enqueue_opLlons,
message_properLles => message_properLles,
payload => msg1,
msgld => msg_ld),
Lnu,
/

9#24 I; CreaLe procedure Lo dequeue messages/uMs Lo be used Lo load daLa ln Mongou8

C8LA1L 8CCLuu8L dequeue_emp(payload Cu1 vA8CPA82) AS
msg1 emp_L := emp_L(nuLL),
msg_ld 8AW(16),
dequeue_opLlons u8MS_AC.uLCuLuL_C1lCnS_1,
message_properLles u8MS_AC.MLSSACL_8CL81lLS_1,
8LCln
u8MS_AC.uLCuLuL(
queue_name => 'LMCuLuL_C',
dequeue_opLlons => dequeue_opLlons,
message_properLles => message_properLles,
payload => msg1,
msgld => msg_ld
),
payload := msg1.msg,
Lnu,
/

9#24 J; CreaLe 1rlgger Lo queue uMLs on Lable SCC11.LM ln !SCn formaL

C8LA1L C8 8LLACL 18lCCL8 emp_q_Lrlg
Al1L8 lnSL81 C8 uuA1L C8 uLLL1L Cn emp
lC8 LACP 8CW
uLCLA8L
msg vA8CPA82(4000),
8LCln
ll lnSL81lnC C8 uuA1lnC 1PLn
msg := '["empno":' || :new.empno || ',"ename":"' || :new.ename || '"',
msg := msg||',"[ob":"' || :new.[ob || '","mgr":"' || :new.mgr || '"',
msg := msg||',"hlredaLe":"' || :new.Pl8LuA1L || '","sal":"' || :new.sal || '"',
msg := msg||',"comm":"' || :new.CCMM || '","depLno":"' || :new.uL1nC || '"',
msg := msg||',"dml_Lype":"',
msg := msg|| CASL WPLn lnSL81lnC 1PLn 'l' LLSL 'u' Lnu || '"}',
Yahoo! White Papei

8
LLSL
msg := '["empno":' || :old.empno || ',"dml_Lype":"u"}',
Lnu ll,
enqueue_emp(msg),
Lnu,
/

9#24 K; CreaLe a perl scrlpL Lo dequeue Lhe message Lo mongou8 collecLlon (Lable) called LM.
#!/usr/bln/perl
use sLrlcL,
use u8l,
use u8u::Cracle,
use Mongou8,
use Mongou8::Clu,
use 1lme::Local,
use !SCn,


#ConnecL Lo Lhe Cracle u8
my $dbh = u8l->connecL("dbl:Cracle:hosL=hosLname1,porL=1321,sld=LesL","scoLL","Llger")
or dle "Lrror: " . $u8l::errsLr,

#ConnecL Lo Mongo u8
my $conn=Mongou8::ConnecLlon->new(hosL => 'hosLname1:27020'),
prlnL "conn $conn\n",

#ConnecL Lo requlred Mongo u8
my $db=$conn->LesL,
prlnL "db $db\n",

#ConnecL Lo Lhe requlred Mongo CollecLlon
my $coll=$db->emp,
prlnL "CollecLlon: $coll\n",

# CounL Lhe messages ln Cracle Cueue
my $sql1 = "selecL counL(*) from AC\$LMCuLuL_C1",
my $sLm1 = $dbh->prepare($sql1),
$sLm1->execuLe(),
my $ccc = $sLm1->feLchrow_array(),
prlnL "Message CounL: $ccc\n",

#leLch Lhe Cracle AC Cueue uaLa
my $aqdaLa,
my $sql2 = "8LCln dequeue_emp(:aqdaLa), Lnu,",
Yahoo! White Papei

9
my $sLm2 = $dbh->prepare($sql2),
$sLm2->blnd_param_lnouL(":aqdaLa", \$aqdaLa,4000),

#SLarL of Lhe whlle loop
my $counLer="1",

whlle ($counLer <= $ccc) [
$sLm2->execuLe(),
prlnL "CounLer: $counLer\n",
##!SCn ouLpuL ["empno":7369,"ename":"SMl1P","[ob":"CLL8k","mgr":"7902","hlredaLe":"17-
uLC-80","sal":"800","comm":"200","depLno":"20","dml_Lype":"u"}
my $[son = $aqdaLa,
my $decoded_[son = decode_[son($[son),
prlnL $decoded_[son->["dml_Lype"},
prlnL "\n",
prlnL $decoded_[son->["ename"},
prlnL "\n",

lf ($decoded_[son->["dml_Lype"} eq "l")[
$coll->lnserL([empno => $decoded_[son->["empno"},ename => $decoded_[son->["ename"}, [ob
=> $decoded_[son->["[ob"}, mgr => $decoded_[son->["mgr"}, hlredaLe => $decoded_[son-
>["hlredaLe"}, sal => $decoded_[son->["sal"}, comm => $decoded_[son->["comm"}, depLno =>
$decoded_[son->["depLno"}}),
prlnL "lnserL lf \n",
}
lf ($decoded_[son->["dml_Lype"} eq "u")[
$coll->remove(["empno" => $decoded_[son->["empno"}}),
$coll->lnserL([empno => $decoded_[son->["empno"},ename => $decoded_[son->["ename"}, [ob
=> $decoded_[son->["[ob"}, mgr => $decoded_[son->["mgr"}, hlredaLe => $decoded_[son-
>["hlredaLe"}, sal => $decoded_[son->["sal"}, comm => $decoded_[son->["comm"}, depLno =>
$decoded_[son->["depLno"}}),
prlnL "updaLe lf \n",
}
lf ($decoded_[son->["dml_Lype"} eq "u")[
$coll->remove(["empno" => $decoded_[son->["empno"}}),
prlnL "ueleLe lf \n",
}
$counLer ++,
}

9#24 L; 1esLlng Lhe uaLa 8epllcaLlon

updaLlng uaLa ln Cracle:
SCL> updaLe emp seL comm=200 where empno=7369,
Yahoo! White Papei

1u
1 row updaLed.

SCL> commlL,
CommlL compleLe.

Checklng lf Lhe daLa ls ln Cracle Cueue:
SCL> selecL counL(*) from aq$LMCuLuL_C1,

CCun1(*)
----------
1


LxecuLlng Lhe perl scrlpL:

[oracle[hosLname1 perlsLuff]$ perl o2m_emp.pl
conn Mongou8::ConnecLlon=PASP(0x1b43c630)
db Mongou8::uaLabase=PASP(0x1b930380)
CollecLlon: Mongou8::CollecLlon=PASP(0x1c1c3210)
Message CounL: 1
CounLer: 1
u
SMl1P
updaLe lf


Checklng Lhe daLa ln Mongou8:
[oracle[hosLname1 ~]$ mongo --porL 27020
Mongou8 shell verslon: 2.0.3
connecLlng Lo: 127.0.0.1:27020/LesL
> use LesL
swlLched Lo db LesL
> show collecLlons
emp
sysLem.lndexes
> db.emp.flnd()
[ "_ld" : Cb[ecLld("4fc016298b93a68269000000"), "comm" : "200", "sal" : "800", "[ob" :
"CLL8k", "mgr" : "7902", "hlredaLe" : "17-uLC-80", "depLno" : "20", "ename" : "SMl1P",
"empno" : numberLong(7369) }
>

Yahoo! White Papei

11
F>? C%0&)", M'11 60E12 D$%B /$0(12 #% +%",%-.;

7+89+/./)4 :4.5%
Mongou8: Mongou8 shell verslon: 2.0.3
Cracle: 11.2.0.2
erl scrlpLlng
u8u:Cracle: perl u8u for Cracle
MongolmporL

0>? 32$1 5($)4# #% &'B4 %$0(12 #0E12 0"& B2#0&0#0 #% ><N9 D)12

#!/usr/bln/perl
use u8l,
use u8u::Cracle,
use sLrlcL,
my $Lab = $A8Cv[0],

#llle Lo sLore daLa
open(my $ouLpuLllle, '>', $Lab . ".Lsv"),
open(my $ouLpuLllle1, '>', "meLadaLa_" . $Lab . ".Lsv"),

#ConnecL Lo Lhe Cracle u8
my $dbh = u8l->connecL("dbl:Cracle:hosL=snv-ccdldb-
011.corp.yahoo.com,porL=1321,sld=LesL","sysLem","oracle")
or dle "Lrror: " . $u8l::errsLr,
$dbh->[Long8eadLen} = 63333,
$dbh->[rlnLLrror} = 0,

# sql sLaLemenL Lo dump Lable daLa
my $gensql=" selecL * from ". $Lab ,
my $sLh = $dbh->prepare($gensql),
$sLh->execuLe(),

whlle (my $row = $sLh->feLchrow_arrayref) [
prlnL $ouLpuLllle [oln(",", [$row) . "\n",
}

close $ouLpuLllle,
$sLh->flnlsh,

#sql sLaLemenL Lo dump Lable meLadaLa

Yahoo! White Papei

12
my $gensql2="selecL column_name from user_Lab_columns where
lower(Lable_name)='" . $Lab . "'",
#prlnL "$gensql2",
my $sLh2 = $dbh->prepare($gensql2) or dle "Couldn'L prepare sLaLemenL: " . $dbh->errsLr,
$sLh2->execuLe(),
whlle (my $row = $sLh2->feLchrow_arrayref) [
prlnL $ouLpuLllle1 [oln(",", [$row) . "\n",
}
close $ouLpuLllle1,
$sLh2->flnlsh,
prlnL "\n",
prlnL "1o lmporL Lo mongou8 use meLadaLa flle:meLadaLa_$Lab.Lsv\n",
prlnL "1o lmporL Lo mongou8 use daLaflle:$Lab.Lsv\n",

LxecuLe Lhe scrlpL:

[oracle[ snv-ccdldb-011 perlsLuff]$ ./oracsvmongo.pl LesL2

1o lmporL Lo mongou8 use meLadaLa flle:meLadaLa_LesL2.Lsv
1o lmporL Lo mongou8 use daLaflle:LesL2.Lsv

E>? C%0& #72 &'B4 D)12 #% +%",%-. '5)", B%",%)B4%$#

mongolmporL --hosL snv-ccdldb-011 --porL 27020 --db LesL --collecLlon Lables --
fleldllle /home/hadoop/perlsLuff/meLadaLa_LesL2.Lsv --Lype csv --flle
/home/hadoop/perlsLuff/LesL2.Lsv

lncremenLal daLa load, uslng merge (upserL)

mongolmporL --hosL snv-ccdldb-011 --porL 27020 --db LesL --collecLlon users --
fleldllle /home/hadoop/perlsLuff/meLadaLa_LesL2.Lsv --upserLllelds
CWnL8,1A8LL_nAML,1A8LLSACL_nAML,S1A1uS --Lype csv --flle
/home/hadoop/perlsLuff/my1able.Lsv




Yahoo! White Papei

1S
G>? C%0& &0#0 D$%B +%",%-. #% /$0(12 #0E12

7+89+/./)4 :4.5%
Mongou8: Mongou8 shell verslon: 2.0.3
Cracle: 11.2.0.2
MongoexporL
Cracle SCL Loader

(;< '=89 6+/#+'- 5()( )+ ;7>? @"*. =4"/# 8+/#+.A9+1)

mongoexpoit --host snv-ccuiub-u11 --poit 27u2u --ub test --collection tables --csv --
fieluFile homehauooppeilstuffmetauata_test2.tsv --out test2.csv

File contents:

|hauoopsnv-ccuiub-u11 peilstuffj$ moie test.csv
name,quantity,piice
"eggs",1u.u,1.S
"apple","S","1u"

3;< B+(5 )C. 7?> @"*. )+ 01(2*. =4. >DBB+(5.1

SQLLoauei contiolfile:

loau uata
infile 'homehauooppeilstufftest.csv'
into table test
fielus teiminateu by "," optionally encloseu by '"'
( name, quantity, piice )

Now iun sqllui to loau uata:

sqllui systemoiacle contiol=loauei.ctl

Check the uata in 0iacle uatabase:

SQL> select * fiom system.test;
NANE Q0ANTITY PRICE
---------- ---------- ----------
eggs 1u 1.S
apple S 1u

Yahoo! White Papei

14
<%"(1'5)%"

We have successfully LesLed 8epllcaLlon beLween Cracle and Mongou8. WlLh Lhe
lncreaslng use of Cpen-Source key/value daLabases, Lhls proves LhaL we can
seamlessly move daLa across 8u8MS and noSCL uaLabases.