You are on page 1of 20

Microservices are the

Future!
and always will be

Josh Holtzman, Director, Infrastructure Engineering

Xoom, a PayPal Service

31 January 2017
1
Xoom.com
A little history

Digital remittance
Founded in 2001
Acquired bluekite.com in 2014
Acquired by PayPal in 2016

2
Xoom.com
A little history (translated)

Digital remittance Highly regulated environment


Founded in 2001
Acquired bluekite.com in 2014
Acquired by PayPal in 2016

3
Xoom.com
A little history (translated)

Digital remittance Highly regulated environment


Founded in 2001 16 years of code and data
Acquired bluekite.com in 2014
Acquired by PayPal in 2016

4
Xoom.com
A little history (translated)

Digital remittance Highly regulated environment


Founded in 2001 16 years of code and data
Acquired bluekite.com in 2014 Polyglot code and persistence
Acquired by PayPal in 2016

5
Xoom.com
A little history (translated)

Digital remittance Highly regulated environment


Founded in 2001 16 years of code and data
Acquired bluekite.com in 2014 Polyglot code and persistence
Acquired by PayPal in 2016 New rules

6
Throwing down the gauntlet
Break up the monolith(s)

Decouple teams
Reduce time to build and deploy
Understand our resource needs
Scale appropriately

7
Microservices to the rescue
Challenges and risks

Programming paradigms and idioms


Service discovery
Monitoring
Performance
Infrastructure as code
Build and deployment pipeline
Data ownership

8
Programming paradigms and idioms

Network operations
Circuit breakers
Aggressive timeouts
Retries
Throttles
API designs
RPC vs REST
Batch operations
Response code granularity
Contracts
Packaging
Metadata
Management uniform

9
Service discovery
The service-proxy solution
Host
Custom, local, layer seven load balancers App
App B
Zookeeper back-end A
Apache Curator
Registration, health checks, and routing
Service Portal Service-proxy
Integrating with linkerd.io

Zookeeper

10
Monitoring
Grafana and InfluxDB

Define required measurements


persistence operations
remote calls
service endpoints
3rd party service endpoints
Define metric types
gauges
counters
histograms
Standard naming scheme
Self-service dashboards
Time series explosion

11
Performance
Throughput and response latency

Additional network latency has been offset by:


Reduced contention on datastores
Limiting the scope of database transactions
Optimization through observability
Throughput has improved dramatically
Latency distribution is wider
Latency sensitive APIs are deployed nearby

12
Infrastructure as code

TDD isnt just for applications


Terraform and Packer for host provisioning on AWS and Vsphere
Puppet and Ansible acceptance testing using beaker
Network gear
Standardize app packaging
Docker
Contracts for deployment
Application control plane

13
Build and deploy pipeline

Git-flow
Branch per feature
Docker-flow
Container per branch
Seed jobs
Build job per branch
Automated and self service deployments
Dev and QA teams can choose branches to deploy and test
Fidelity of environments
Environment fidelity automation success

14
Build and deploy pipeline

15
Data ownership

Hard problem
Start eliminating cross-domain joins now
Two years on, we are just now migrating the last auth-server client from tables to APIs
Analytics becomes more complicated

16
Current status

~100 distinct microservices across 3 production data centers


Most new features are developed as microservices
Monoliths still exist, but are being chipped away

17
Lessons learned

Measure everything, and be prepared to scale your monitoring system


Application packaging contracts and delivery pipelines are mandatory
Staff a tooling team for build, test, and deployment automation
Enroll your network operations team
The infrastructure and culture we built in order to move to microservices has paid off
Elimination of the monoliths isnt that important

18
Questions

19
Josh Holtzman
jholtzman@paypal.com

20

You might also like