You are on page 1of 37

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

Architectures you
Always
Wondered
About
Lessons learnt from adopting Microservices
at eBay, Google, Gilt, Hailo and nearForm
eMag Issue 31 - August 2015

ARTICLE

INTERVIEW

PRESENTATION SUMMARY

Microservice
trade-offs

Eric Evans on
DDD at 10

Service Architectures
at Scale

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

Martin Fowler on
Microservice tradeoffs
Many development teams have found
the microservices architectural style to
be a superior approach to a monolithic
architecture. But other teams have found
them to be a productivity-sapping burden. Like any architectural style, microservices bring costs and benefits. To make
a sensible choice you have to understand
these and apply them to your specific
context.

Eric Evans on the interplay of


Domain-Driven Design, microservices,
event-sourcing, and CQRS
The interview covers an introduction to DDD, how the communitys understanding of DDD has changed in the last 10 years, strategic design, how to use
DDD to design microservices, and the connection between microservices and
the DDD bounded context.

Service Architectures at Scale:


Lessons from Google and eBay
Randy Shoup discusses modern service architectures at scale, using specific examples from both Google and eBay. He covers some interesting lessons
learned in building and operating these sites. He concludes with a number of
experience-based recommendations for other smaller organizations evolving
to -- and sustaining -- an effective service ecosystem.

Evolutionary Architecture
Randy Shoup talks about designing and building microservices based on his experience of working at large companies, such as Google and eBay. Topics covered include the
real impact of Conways law, how to decide when to move
to a microservice-based architecture, organizing team
structure around microservices, and where to focus on the
standardization of technology and process.

Building a Modern Microservices


Architecture at Gilt

Lessons Learned Adopting


Microservices at Gilt, Hailo
and nearForm
This article contains an extensive interview on the microservices adoption process, the technologies used, the
benefits and difficulties of implementing microservices,
with representatives from Gilt, Hailo and nearForm.

After living with microservices for three years, Gilt can see advantages in team ownership, boundaries defined by APIs, and complex
problems broken down into small ones, Yoni Goldberg explained in
a presentation at the QCon London 2015 conference. Challenges
still exist in tooling, integration environments, and monitoring.

FOLLOW US

CONTACT US
GENERAL FEEDBACK feedback@infoq.com
ADVERTISING sales@infoq.com
EDITORIAL editors@infoq.com

facebook.com
/InfoQ

@InfoQ

google.com
/+InfoQ

linkedin.com
company/infoq

over as head of the


CHARLES took
editorial team at InfoQ.
HUMBLE com in March 2014,

guiding content creation including news,


articles, books, video presentations, and
interviews. Prior to taking on the full-time role
at InfoQ, Charles led InfoQs Java coverage, and
was CTO for PRPi Consulting, a remuneration
research rm that was acquired by PwC in July
2012. For PRPi, he had overall responsibility for
the development of all the custom software
used within the company. He has worked
in enterprise software for around 20 years
as a developer, architect, and development
manager.

A LETTER FROM
THE EDITOR
This eMag has had an unusual history. When we
started to plan it the intent had been to look at
the different architectural styles of a number of
the well known Silicon Valley firms. As we started to work on it though it become apparent that
nearly all of them had, at some level, converged
towards the same architectural style - one based
on microservices, with DevOps and some sort of
agile (in the broadest sense) management approach.
According to ThoughtWorks Chief Scientist
Martin Fowler the term microservice was discussed at a workshop of software architects near
Venice in May 2011, to describe what the participants saw as a common architectural style that
many of them had begun exploring recently. In
May 2012, the same group decided on microservices as the most appropriate name.
When we first started talking about the microservices architectural style at InfoQ in 2013, I
think many of us assumed that its inherent operational complexity would prevent the approach
being widely adopted particularly quickly. Yet a
mere three years on from the term being coined
it has become one of the most commonly cited
approaches for solving large-scale horizontal
scaling problems, and most large web sites including Amazon and eBay have evolved from a

monolithic architecture to a microservices one.


Moreover the style has spread far beyond its Bay
Area roots, seeing widespread adoption in many
organisations.
In this eMag we take a look at the state of
the art in both theory and practice.
Martin Fowler provides a clear and concise
summary of the trade-offs involved when choosing to work with the style.
Eric Evans talks about the interplay of Domain-Driven Design, microservices, event-sourcing, and CQRS.
Randy Shoup describes experiences of
working with microservices from his time at eBay
and Google. He focuses on the common evolutionary path from monoliths to microservices
and paints a picture of a mature services environment at Google. In a follow-up interview he
elaborates on some of the lessons from this experience.
Then Abel Avram speaks to three companies - Gilt, Hailo and nearForm - about their
experiences covering both building a microservices platform from scratch and re-architecting a
monolithic platform by gradually introducing microservices. In the follow-up presentation summary we take a more detailed look at Gilt.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

Read on martinfowler.com

Microservice trade-offs by Martin Fowler

Martin Fowler is an author, speaker, and general loud-mouth on software development. Hes
long been puzzled by the problem of how to componentize software systems, having heard
more vague claims than hes happy with. He hopes that microservices will live up to the early
promise its advocates have found.

Many development teams have found the microservices architectural


style to be a superior approach to a monolithic architecture. But other
teams have found them to be a productivity-sapping burden. Like any
architectural style, microservices bring costs and benefits. To make a
sensible choice you have to understand these and apply them to your
specific context.
Microservices provide benefits
Strong Module Boundaries:

Microservices reinforce modular structure, which is particularly important for larger


teams.

Independent Deployment:

Simple services are easier to


deploy, and since they are
autonomous, are less likely to cause system failures
when they go wrong.

Technology Diversity: With micro-

services you can mix multiple languages, development frameworks


and data-storage technologies.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

but come with costs


Distribution: Distributed sys-

tems are harder to program,


since remote calls are slow
and are always at risk of failure.

Strong Module
Boundaries

The first big benefit of microservices is strong module boundaries. This is an important benefit
yet a strange one, because there
is no reason, in theory, why a microservices should have stronger module boundaries than a
monolith.
So what do I mean by a
strong module boundary? I think
most people would agree that its
good to divide up software into
modules: chunks of software that
are decoupled from each other.
You want your modules to work
so that if I need to change part
of a system, most of the time I
only need to understand a small
part of that system to make the
change, and I can find that small
part pretty easily. Good modular
structure is useful in any program, but becomes exponentially more important as the software grows in size. Perhaps more
importantly, it grows more in importance as the team developing
it grows in size.
Advocates of microservices
are quick to introduce Conways
Law, the notion that the structure of a software system mirrors
the communication structure
of the organization that built it.
With larger teams, particularly if
these teams are based in different locations, its important to
structure the software to recognize that inter-team communications will be less frequent and
more formal than those within a
team. Microservices allow each
team to look after relatively inde6

Eventual Consistency: Maintaining strong consistency is extremely


difficult for a distributed system,
which means everyone has to manage eventual consistency.

pendent units with that kind of


communication pattern.
As I said earlier, theres no
reason why a monolithic system
shouldnt have a good modular
structure. [1] But many people
have observed that it seems rare,
hence theBig Ball of Mudis most
common architectural pattern.
Indeed this frustration with the
common fate of monoliths is
whats driven several teams to
microservices. The decoupling
with modules works because the
module boundaries are a barrier
to references between modules.
The trouble is that, with a monolithic system, its usually pretty
easy to sneak around the barrier.
Doing this can be a useful tactical shortcut to getting features
built quickly, but done widely
they undermine the modular
structure and trash the teams
productivity. Putting the modules into separate services makes
the boundaries firmer, making it
much harder to find these cancerous workarounds.
An important aspect of this
coupling is persistent data. One
of the key characteristics of microservices isDecentralized Data
Management, which says that
each service manages its own
database and any other service
must go through the services
API to get at it. This eliminatesIntegration Databases, which are a
major source of nasty coupling in
larger systems.
Its important to stress that
its perfectly possible to have
firm module boundaries with a
monolith, but it requires disci-

Complexity: You
need a mature operations team to
manage lots of services, which are
being redeployed regularly.

Operational

pline. Similarly you can get a Big


Ball of Microservice Mud, but it
requires more effort to do the
wrong thing. The way I look at,
using microservices increases the
probability that youll get better
modularity. If youre confident in
your teams discipline, then that
probably eliminates that advantage, but as a team grows it gets
progressively harder to keep disciplined, just as it becomes more
important to maintain module
boundaries.
This advantage becomes
a handicap if you dont get your
boundaries right. This is one of
the two main reasons for aMonolith First strategy, and why even
those more inclined to run with
microservices early stress that
you can only do so with a well
understood domain.
But Im not done with caveats on this point yet. You can
only really tell how well a system
has maintained modularity after
time has passed. So we can only
really assess whether microservices lead to better modularity
once we see microservice systems that have been around for
at least a few years. Furthermore,
early adopters tend to be more
talented, so theres a further
delay before we can assess the
modularity advantages of microservice systems written by average teams. Even then, we have to
accept that average teams write
average software, so rather than
compare the results to top teams
we have to compare the resulting software to what it would
have been under a monolithic

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

architecture - which is a tricky


counter-factual to assess.
All I can go on for the moment is the early evidence I have
hear from people I know who
have been using this style. Their
judgement is that it is significantly easier to maintain their modules.
One case study was particularly interesting. The team had
made the wrong choice, using
microservices on a system that
wasnt complex enough to cover theMicroservice Premium.
The project got in trouble and
needed to be rescued, so lots
more people were thrown onto
the project. At this point the microservice architecture became
helpful, because the system was
able to absorb the rapid influx
of developers and the team was
able to leverage the larger team
numbers much more easily than
is typical with a monolith. As a
result the project accelerated
to a productivity greater than
would have been expected with
a monolith, enabling the team
to catch up. The result was still
a net negative, in that the software cost more staff-hours than
it would have done if they had
gone with a monolith, but the
microservices architecture did
support ramp up.

Distribution
So microservices use a distributed system to improve modularity. But distributed software has a
major disadvantage, the fact that
its distributed. As soon as you
play the distribution card, you
incur a whole host of complexities. I dont think the microservice community is as naive about
these costs as the distributed
objects movement was, but the
complexities still remain.
The first of these is performance. You have to be in a really
unusual spot to see in-process
function calls turn into a performance hot spot these days, but

remote calls are slow. If your service calls half-a-dozen remote


services, each which calls another half-a-dozen remote services,
these response times add up to
some horrible latency characteristics.
Of course you can do a
great deal to mitigate this problem. Firstly you can increase the
granularity of your calls, so you
make fewer of them. This complicates your programming model,
you now have to think of how to
batch up your inter-service interactions. It will also only get you
so far, as you are going to have to
call each collaborating service at
least once.
The second mitigation is to
use asynchrony. If make six asynchronous calls in parallel youre
now only as slow as the slowest
call instead of the sum of their latencies. This can be a big performance gain, but comes at another cognitive cost. Asynchronous
programming is hard: hard to get
right, and much harder to debug.
But most microservice stories Ive
heard need asynchrony in order
to get acceptable performance.
Right after speed is reliability. You expect in-process function calls to work, but a remote
call can fail at any time. With lots
of microservices, theres even
more potential failure points.
Wise developers know this
and design for failure. Happily
the kinds of tactics you need for
asynchronous collaboration also
fit well with handling failure and
the result can improve resiliency.
Thats not much compensation
however, you still have the extra
complexity of figuring out the
consequences of failure for every
remote call.
And thats just the top
twoFallacies of Distributed Computing.
There are some caveats
to this problem. Firstly many
of these issues crop up with a
monolith as it grows. Few mono-

liths are truly self-contained, usually there are other systems, often legacy systems, to work with.
Interacting with them involves
going over the network and running into these same problems.
This is why many people are inclined to move more quickly to
microservices to handle the interaction with remote systems.
This issue is also one where experience helps, a more skillful team
will be better able to deal with
the problems of distribution.
But distribution is always a
cost. Im always reluctant to play
the distribution card, and think
too many people go distributed
too quickly because they underestimate the problems.

Eventual
Consistency
Im sure you know websites that
need a little patience. You make
an update to something, it refreshes your screen and the update is missing. You wait a minute
or two, hit refresh, and there it is.
This is a very irritating usability problem, and is almost
certainly due to the perils of
eventual consistency. Your update was received by the pink
node, but your get request was
handled by the green node. Until
the green node gets its update
from pink, youre stuck in an inconsistency window. Eventually it will be consistent, but until
then youre wondering if something has gone wrong.
Inconsistencies like this are
irritating enough, but they can
be much more serious. Business
logic can end up making decisions on inconsistent information, when this happens it can
be extremely hard to diagnose
what went wrong because any
investigation will occur long after the inconsistency window
has closed.
Microservices
introduce
eventual consistency issues because of their laudable insistence

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

Microservices
are the first post
DevOps revolution
architecture
- Neal Ford

on decentralized data management. With a monolith, you can


update a bunch of things together in a single transaction.
Microservices require multiple
resources to update, and distributed transactions are frowned on
(for good reason). So now, developers need to be aware of consistency issues, and figure out how
to detect when things are out of
sync before doing anything the
code will regret.
The monolithic world isnt
free from these problems. As systems grow, theres more of a need
to use caching to improve performance, and cache invalidation
is the other Hard Problem. Most
applications needoffline locksto
avoid long-lived database transactions. External systems need
updates that cannot be coordinated with a transaction manager. Business processes are often
more tolerant of inconsistencies
than you think, because businesses often prize availability
more (business processes have
long had an instinctive understanding of theCAP theorem).
So like with other distributed issues, monoliths dont entirely avoid inconsistency problems,
but they do suffer from them
much less, particularly when
they are smaller.

Independent
Deployment
The trade-offs between modular
boundaries and the complexities
of distributed systems have been
around for my whole career in
this business. But one thing thats
changed noticeably, just in the
last decade, is the role of releasing to production. In the twentieth century production releases
were almost universally a painful
and rare event, with day/night
weekend shifts to get some awkward piece of software to where
it could do something useful. But
these days, skillful teams release
frequently to production, many
8

organizations practicing Continuous Delivery, allowing them


to do production releases many
times a day.
This shift has had a profound effect on the software
industry, and it is deeply intertwined with the microservice
movement. Several microservice efforts were triggered by
the difficulty of deploying large
monoliths, where a small change
in part of the monolith could
cause the whole deployment to
fail. A key principle of microservices is thatservices are componentsand thus are independently deployable. So now when you
make a change, you only have to
test and deploy a small service. If
you mess it up, you wont bring
down the entire system. After all,
due the need to design for failure, even a complete failure of
your component shouldnt stop
other parts of the system from
working, albeit with some form
of graceful degradation.
This relationship is a twoway street. With many microservices needing to deploy frequently, its essential you have
your deployment act together.
Thats why rapid application deployment and rapid provisioning
of infrastructure areMicroservice
Prerequisites. For anything beyond the basics, you need to be
doing continuous delivery.
The great benefit of continuous delivery is the reduction in
cycle-time between an idea and
running software. Organizations
that do this can respond quickly
to market changes, and introduce new features faster than
their competition.
Although many people cite
continuous delivery as a reason
to use microservices, its essential to mention that even large
monoliths can be delivered continuously too. Facebook and Etsy
are the two best known cases.
There are also plenty of cases
where attempted microservices

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

architectures fail at independent


deployment, where multiple services need their releases to be
carefully coordinated[2]. While I
do hear plenty of people arguing
that its much easier to do continuous delivery with microservices, Im less convinced of this
than their practical importance
for modularity - although naturally modularity does correlate
strongly with delivery speed.

Operational
Complexity
Being able to swiftly deploy
small independent units is a
great boon for development, but
it puts additional strain on operations as half-a-dozen applications now turn into hundreds of
little microservices. Many organizations will find the difficulty of
handling such a swarm of rapidly
changing tools to be prohibitive.
This reinforces the important role of continuous delivery.
While continuous delivery is a
valuable skill for monoliths, one
thats almost always worth the
effort to get, it becomes essential
for a serious microservices setup. Theres just no way to handle
dozens of services without the
automation and collaboration
that continuous delivery fosters.
Operational complexity is also
increased due to the increased
demands on managing these
services and monitoring. Again
a level of maturity that is useful
for monolithic applications becomes necessary if microservices
are in the mix.
Microservice proponents
like to point out that since each
service is smaller its easier to
understand. But the danger is
that complexity isnt eliminated, its merely shifted around to
the interconnections between
services. This can then surface
as increased operational complexity, such as the difficulties in
debugging behavior that spans
services. Good choices of service

boundaries will reduce this problem, but boundaries in the wrong


place makes it much worse.
Handling this operational complexity requires a host of
new skills and tools - with the
greatest emphasis being on the
skills. Tooling is still immature,
but my instinct tells me that even
with better tooling, the low bar
for skill is higher in a microservice environment.
Yet this need for better skills
and tooling isnt the hardest part
of handling these operational
complexities. To do all this effectively you also need to introduce
a devops culture: greater collaboration between developers,
operations, and everyone else
involved in software delivery.
Cultural change is difficult, especially in larger and older organizations. If you dont make this
up-skilling and cultural change,
your monolithic applications will
be hampered, but your microservice applications will be traumatized.

another part of the system. Dealing with library versioning issues


is one of those problems that
gets exponentially harder as the
code base gets bigger.
There is a danger here that
there is so much technology diversity that the development
organization can get overwhelmed. Most organizations I
know do encourage a limited set
of technologies. This encouragement is supported by supplying
common tools for such things as
monitoring that make it easier for
services to stick to a small portfolio of common environments.
Dont underestimate the
value of supporting experimentation. With a monolithic system,
early decisions on languages and
frameworks are difficult to reverse. After a decade or so such
decisions can lock teams into
awkward technologies. Microservices allow teams to experiment
with new tools, and also to gradually migrate systems one service at a time should a superior
technology become relevant.

Technology
Diversity

Secondary Factors

Since each microservice is an


independently deployable unit,
you have considerable freedom
in your technology choices within it. Microservices can be written
in different languages, use different libraries, and use different
data stores. This allows teams to
choose an appropriate tool for
the job, some languages and libraries are better suited for certain kinds of problems.
Discussion of technical diversity often centers on best tool
for the job, but often the biggest
benefit of microservices is the
more prosaic issue of versioning.
In a monolith you can only use a
single version of a library, a situation that often leads to problematic upgrades. One part of the
system may require an upgrade
to use its new features but cannot because the upgrade breaks

I see the items as above as the


primary trade-offs to think about.
Heres a couple more things that
come up that I think are less important.
Microservice proponents
often say that services are easier
to scale, since if one service gets
a lot of load you can scale just it,
rather than the entire application. However Im struggling to
recall a decent experience report
that convinced me that it was
actually more efficient to do this
selective scaling compared to
doing cookie-cutter scaling by
copying the full application.
Microservices allow you to
separate sensitive data and add
more careful security to that
data. Furthermore by ensuring
all traffic between microservices
is secured, a microservices approach could make it harder to

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

It's important
to stress that it's
perfectly possible to
have firm module
boundaries with
a monolith, but it
requires discipline.
Similarly you can
get a Big Ball of
Microservice Mud,
but it requires more
effort to do the
wrong thing

10

exploit a break-in. As security


issues grow in importance, this
could migrate to becoming a major consideration for using microservices. Even without that, its a
not unusual for primarily monolithic systems to create separate
services to handle sensitive data.
Critics of microservices talk
about the greater difficulty in
testing a microservices application than a monolith. While this
is a true difficulty - part of the
greater complexity of a distributed application - there aregood
approaches to testing with microservices. The most important
thing here is to have the discipline to take testing seriously,
compared to that the differences
between testing monoliths and
testing microservices are secondary.

Summing Up
Any general post on any architectural style suffers from
the Limitations Of General Advice. So reading a post like this
cant lay out the decision for
you, but such articles can help
ensure you consider the various
factors that you should take into
account. Each cost and benefit
here will have a different weight

for different systems, even swapping between cost and benefit


(strong module boundaries are
good in more complex systems,
but a handicap to simple ones)
Any decision you make depends
on applying such criteria to your
context, assessing which factors
matter most for your system and
how they impact your particular
context. Furthermore, our experience of microservice architectures is relatively limited. You can
usually only judge architectural
decisions after a system has matured and youve learned what
its like to work with years after
development began. We dont
have many anecdotes yet about
long-lived microservice architectures.
Monoliths and microservices are not a simple binary
choice. Both are fuzzy definitions that mean many systems
would lie in a blurred boundary
area. Theres also other systems
that dont fit into either category. Most people, including myself, talk about microservices in
contrast to monoliths because
it makes sense to contrast them
with the more common style, but
we must remember that there
are systems out there that dont

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

fit comfortably into either category. I think of monoliths and


microservcies as two regions in
the space of architectures. They
are worth naming because they
have interesting characteristics
that are useful to discuss, but no
sensible architect treats them as
a comprehensive partitioning of
the architectural space.
That said, one general summary point that seems to be
widely accepted is there is aMicroservice Premium: microservices impose a cost on productivity that can only be made up
for in more complex systems. So
if you can manage your systems
complexity with a monolithic architecture then you shouldnt be
using microservices.
But the volume of the microservices conversation should
not let us forget the more important issues that drive the
success and failure of software
projects. Soft factors such as the
quality of people on the team,
how well they collaborate with
each other, and the degree of
communication with domain experts, will have a bigger impact
than whether to use microservices or not. On a purely technical level, its more important to
focus on things like clean code,
good testing, and attention to
evolutionary architecture.
Footnotes
1:Some people consider monolith to be an insult, always implying poor modular structure.
Most people in the microservices
world dont do this, they define
monolith purely as an application built as a single unit. Certainly microservices-advocates
believe that most monoliths end
up being Big Balls of Mud, but I
dont know any who would argue that its impossible to built a
well-structured monolith.
2:The ability to deploy services independently ispart of the
definition of microservices. So its

reasonable to say that a suite of


services that must have its deployments coordinated is not a
microservice architecture. It is
also reasonable to say that many
teams that attempt a microservice architecture get into trouble
because they end up having to
coordinate service deployments.
Further Reading

Sam Newman gives his list of


the benefits of microservices
inChapter 1 of his book(the
essential source for details of
building a microservices system).
Benjamin
Woottons
post, Microservices - Not A
Free Lunch! on High Scaleability, was one of the earliest,
and best, summaries of the
downsides of using microservices.

Acknowledgements
Brian Mason, Chris Ford, Rebecca
Parsons, Rob Miles, Scott Robinson, Stefan Tilkov, Steven Lowe,
and Unmesh Joshi discussed
drafts of this article with me.

For more information about


microservices, start with
my Microservice Resource
Guide, where Ive selected
the best information on the
what, when, how, and who
of microservices.
Sam Newmans book Is
the key resource if you want
to find out more about how
to build a microservice system.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

11

Listen on SE Radio

Eric Evans on
Domain-Driven Design at 10 Years

THE INTERVIEWEE
Eric Evans is the author of Domain-Driven Design: Tackling Complexity in Software. Eric now
leads Domain Language, a consulting group which coaches and trains teams applying domaindriven design, helping them to make their development work more productive and more valuable to their business.

THE INTERVIEWER
Eberhard Wolff works as a freelance consultant, architect and trainer in Germany. He is currently interested in Continuous Delivery and technologies such as NoSQL and Java. He is the
au-thor of several books and articles and regularly speaks at national and international conferences.

The show will be about Domain-Driven Design at 10; so


its already 10 years that you
came up with the idea. Some of
the listeners might not be that
familiar with domain-driven
design, so Eric can you give us
a short introduction about domain-driven design, what it is
and how it is special?
In its essence, domain-driven design is a way of using models for
12

creating software, especially the


part of the software that handles
complex business requirements
into such behavior.
So the particular way in domain-driven design, the thing
that we focus on, is that we want
a language where we can really
crisply, concisely describe any
situation in the domain and describe how were going to solve
it or what kind of calculations we
need to do. That language would
be shared between business peo-

ple, specialists in that domain, as


well as software people who will
be writing the software, and that
we call the ubiquitous language
because it runs through that
whole process.
We dont do as most projects do. We dont talk to the
business people sort of on their
terms and then go and have very
technical conversations about
how the software will work, separately. We try to bring those
conversations together to create

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

this conceptual model with very


clear language. And thats a very
difficult thing to do and it cant
be done in a global sense. You
cant come up with the model
for your entire organization. Attempts to do that are very counterproductive.
So the other ingredient in
that is what we call the bounded context. So I have a clear
boundary, perhaps a particular
application that were working
on. Within this boundary, we say
this is what my words mean, this
is the relationship between the
concepts. Everything is clear and
we work hard to make it clear.
But then outside that boundary,
all sorts of different rules apply
in some parts of the system, perhaps no rules really apply.
Out of that comes the rest
of domain-driven design, but
thats really the essence of it. Its
a particular way of dealing with
these complex parts of our systems.

And I guess that is also why so


many people are interested in
that because thats really what
a lot of software engineers do.
Can you give an example of
such a bounded context and
how models might be different
there? Because I think thats
one of the very interesting
parts of DDD; at least it was for
me.
I can give some examples. One
common thing is that different
parts of an organization might
deal with the domain in a very
different way. And there may
even already be software, there
probably is already software, that
deals with those different parts.
So take some company that
does e-commerce. So theres a
part of the software where were
taking orders. So we are very focused on what kind of items are

in the inventory and how much


they cost and how do we collect
these items together into some
kind of a shopping cart? And
then eventually the order is created and then theres payment
and all of those concerns.
But then in shipping, perhaps theyre not really that interested in most of those issues.
What they care about an item is
what kind of box will it fit into,
and how much does it weigh,
and which kind of shipping did
you pay for, and do we ship it all
in one box, or this ones out of
stock so were going to go ahead
and ship the part weve got and
then send the rest later, and how
do we keep track of an order
thats been partially shipped but
part of its still waiting?
Although in theory you
could create a single model that
would represent all these different aspects, in practice thats
not what people usually do and
it works better to separate those
two contexts and say, well we
basically have a shipping system
here and an order taking system
and perhaps other things too - Im
not saying it would just be those
two. You could create concepts
so general and versatile that you
could handle all these cases. But
were usually better off with more
specialized models: a model that
handles really well the idea of an
order as a set of physical objects
that fit into certain kinds of boxes and that you may or may not
have available at this time; and
another one that says, well, here
are items that youve chosen and
here are similar items; and just
totally different issues.

I think that thats a very good


short introduction to DDD,
and in particular the bounded
contexts is I think really one
of the interesting things here,
as you said, where you would

have totally different, lets say,


perspectives on items whether youre working on shipping
or ordering things. So looking back on those ten years of
DDD, what was the impact of
DDD in your opinion?
Well, its very hard for me to see
that. I mean I do have a sense its
had an impact and sometimes
people tell me that it had a big
influence on them. I think the
things that made me feel best is
occasionally someone says it sort
of brought back the fun of software development for them or
made them really enjoy software
development again, and that
particularly makes me feel good.
Its so hard for me to judge what
the overall impact is. I really dont
know.

Its always good if you can bring


back the joy in work again. To
me the impact of the book is
quite huge. I would even call
it a movement, sort of a DDD
movement. Do you think thats
true? Would you call it a movement or is it something different to you?
I think that probably that was
my intention, that I had the
sense that I wanted to start a
movement, maybe a very diffuse
movement but it would be nice
to think it had a little of that quality to it.
One of the things that I really emphasize, and this part had a
crusading quality to it, that when
were working on software, we
need to keep a really sharp focus
on the underlying domain in the
business. And we shouldnt look
at our jobs as just technologists
but really our job is -- and this
is a difficult job -- as this person
who can start to penetrate into
the complexity and tangle of
these domains and start to sift

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

13

that apart in a way that allows


nice software to be written. And
sometimes when you really do it
well, the problem becomes significantly easier on a technical
level.

Yeah. So I think its about, lets


say, model mining or knowledge mining what a lot of our
job is about.
Yes.

Was the impact of your book


and the movement, was it different from what you have expected? Are there any surprises?
Well, one pleasant surprise is that
it hasnt just kind of faded. Most
books go through a fairly short
cycle, they become very popular
for three or four years and then
they kind of drop into the background. And DDD has stayed a
pretty relevant topic for a long
time now; really like 11 years
since its now 2015. So that genuinely surprises me actually, but
obviously in a good way.

And I couldnt agree more.


To me its almost like a timeless thing that youve created
there. What part of the DDD
movement did you learn the
most from? What gave you the
biggest progress in your own
knowledge and skillset?
Oh, I think that -- and this I think
is closely related to why people still pay attention to DDD
-- that DDD has not been static
so the core concepts of it have
been very stable - lets say I can
express them better now. But
the people who are really doing
14

it, there have been a couple of


big shifts in how people tend to
build these systems.
So the first big one was
when event sourcing and CQRS
came on the scene and the way
that we went about designing and building a DDD system
changed a lot. When did that
happen? That was maybe 2007.
Anyway, after a few years, and it
may be that just about the time
that we would have been following that natural cycle of whats
new and moving on to that
thing, DDD kind of got a big renovation. I learned a tremendous
amount from those people, from
Greg Young, Udi Dahan, and the
others who went around and
around about this thing in CQRS
and just really shook things up.
My way of thinking about
things changed a lot and I think
the way most people think about
DDD now is significantly different because of that. There have
been a few other things but that
was certainly the first big one.

Do you think there are any circumstances where a DDD approach would fail? And how
would you deal with them or is
it something that can be made
to work in any project?
So there are a few aspects to that.
Thats an interesting question
because certainly DDD projects
fail all the time. Its not unusual.
Of course, some of that is just
anything difficult fails sometimes
so we neednt worry about that.
And I think DDD is hard. So what
would make a DDD project more
likely to fail than other times? I
think that some of the most common things are there is a tendency to slip into perfectionism:
whenever people are serious
about modeling and design, they
start slipping toward perfectionism. Other people start slipping

toward very broad scope: we will


model the whole thing - even if
we have five different bounded
contexts; but well model each
one with great elegance and all
at the same time.
So some projects fail because they get too ambitious.
They dont walk before they run.
Some of them, because they are
in an area where the strategy of
the organization isnt very clear.
Let me put it this way: the greatest value of DDD comes when
youre very close to the bone of
whats strategically important
within the company. Thats when
it pays off the most. You need a
certain degree of complexity, intricacy in the problem or at least
some fuzziness or else theres
no point to all that thinking. But
also, you need to be in some
strategically relevant thing. But
along with that goes a certain
amount of the rough and tumble
of the politics of organizations.
Some organizations change their
strategies.
Ive seen all these things
happen. There are all sorts of
other things. Of course, sometimes the skills of the team are
the problem. So you might have
a situation where they get off to
a good start and then their execution isnt very good. Of course,
bad programming will undermine any software approach.
Ultimately, the code has to be
good. Well, the code has to be
competent. That one hurts a lot
of projects.
Of course, since I mentioned bounded context earlier and I want to underline how
fundamental that is, its not an
easy discipline. If weve established that the shipping context
and the order taking context are
separate and so weve made a
boundary between them, there
is some kind of translation layer there. But some programmer
in the shipping context needs
a piece of data, and that data is

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

over in the order taking context.


Now, what does he do, does he
just say, Well, I can just write a
query on the order taking database. That will just take me 15
minutes. Or I could go and talk
to people about how are we going to enhance the interface and
translator between those two
contexts and then how would
this piece of information be
modeled within our context, and
then Ill do my feature.
So it takes a certain level
of discipline to maintain that
boundary. And when the boundary is gone, the two models start
clashing and it all goes down the
tubes pretty fast. Another variation on this of course is that if
you have a lot of legacy systems
and youre trying to do some
work. So ideally, you do that by
isolating the new work within a
bounded context and talking to
the legacy through some kind of
translator.
Anyway, I could go on but I
think its not too surprising that
some of them fail.

I agree. As you said, I guess


DDD only makes sense if you
have a certain complexity and
with that comes risk but also
by the potential value of the
software I guess. What I found
interesting about what you
just said is that people get
overambitious in some points
and try to reach for a perfect
state. To me that is natural for
a lot of technical people. So I
was wondering whether you
have any secret sauce how to
focus the ambition on those
parts of the system where its
really important, and to live
with not so great quality in the
other parts, and stop your ambition there? Is there any tip
that you have for that?

I think that you summed it up


very well then. Thats what you
need to do and you need to
have a boundary between the
two. Theres quite a bit in DDD
about part of the strategic design part is how to decide which
parts are which: like theres a
general category of generic
subdomains where we say, well,
theres nothing all that special;
were not creating something
here that we want to innovate;
this is something we want to do
in a standard way. In fact, the
ideal solution here would be lets
not even write any software; lets
see if we can get it off-the-shelf.
Then theres lots of stuff
that just keeps the lights on; that
whether you have some brilliant
insight or not isnt going to really
change the outcome very much.
And then there are those leverage points. This is the general
thing but its very hard, I admit,
because first of all, its hard to
know. Youll often get it wrong
and youll often choose a topic
which may turn out not to have
been very core of the strategic
value. But still I think theres a lot
of value in trying.
Another thing is the perfectionism, because even if you got
zeroed in on a certain part that
was strategically valuable, perfectionism can still kill you. You
have to deliver and you have to
deliver fairly quickly. In fact, DDD
depends on iteration. We assume that you dont understand
the problem very well at the beginning, that youll get it wrong
the first time. And so its essential
to get the first one done quick
and then get on to the second iteration and get that done quick
too, because youll probably get
that wrong too. And then get on
to the third one which is where
youre probably going to have
some clue by then and that third
one might be fairly good. And if
you can get it done fairly quick,
then youll have time to do the

fourth one which is going to be


really elegant.
And Im serious. I think that
its a weird paradox but perfectionism prevents people from
creating really elegant designs,
because it slows them down in
the early iterations so they dont
get to do enough iterations.
Multiple iterations, I mean iteration as in doing the same thing
over, not iteration as when people really are talking about increments where Lets do a little
iterative requirement at a time.
I mean doing the same feature
set then redoing the same feature set again but with a nicer
design, with a new insight into
what it means. Thats the key:
move quick.

That sounds pretty interesting


to focus on the number of reiterations instead of reaching
for a perfect solution at the
very beginning. One thing Im
really wondering is if you ever
plan to update the book, is
there anything you would like
to change in the book?
Im pretty sure Im not going to
update the book. I might write
something new at some point.
But anyway, I think I probably
wont change the book. But if
I did or rather if I were going to
try and explain DDD better, certainly one thing that I have realized for a long time is that the
concept of the bounded context
is much too late in the book. All
the strategic design stuff is way
back at the back. I treat it as an
advanced topic. Theres some
logic to that but the trouble is
that its so far back that most
people never get to it really. So
I would at least weave that in
to the first couple of chapters.
The ubiquitous language is in
chapter 2 so thats all right. But

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

15

I would have bounded context


there in chapter 2 or three.
Another thing I would do is
try to change the presentation
of the building blocks. The building blocks are things like the entities and value objects and layer domain events and stuff like
that. They are important things
but there is a big chunk of that
stuff right in the early middle
part. Most people dont get past
it and they come away thinking
that thats really the core of DDD,
whereas in fact its really not. Its
an important part just because
it helps people bridge from the
conceptual modeling of DDD to
the necessity of having a program, having code that really reflects that model, and bridging
that gap is difficult; thats why
there was so much emphasis on
that. But I really think that the
way I arranged the book gives
people the wrong emphasis. So
thats the biggest part of what Id
do is rearrange those things.

It makes a lot of sense I guess.


I agree that strategic design is
really, really important and really one of those things that a
lot of people dont think about
when they hear the term DDD.
Recently, we have seen a trend
towards microservices architectures and weve already
have had quite a few discussions about microservices on
the show. So how does microservices fit into the DDD world?
Is there erudition?
Im quite enthusiastic about microservices. I think that it helps
people who want to do DDD
quite a bit. And I also think that
certain aspects of DDD can help
people do microservices better.
So when I say it helps people do
DDD, Ive already talked about
bounded contexts and how important that is. If you think about
16

what the people do when they


do microservices in a serious
way, the interior implementation of microservices is very isolated. Everything is supposed to
go through that interface. Any
kind of data that they contain is
exclusively held by them. Theres
really a tight boundary and that
is what you need.
The bounded context is a
concept which in more traditional architectures, there werent
very good ways to implement
that concept; to really establish
the boundary. So it seems to me
that microservices has delivered
us a practical and popular way
of defining and sticking to those
boundaries. And thats a big
deal. And the emphasis on the
micro, well, someone once asked
me, Whats the difference between microservices and the old
SOA services? And I said, Well,
I think part of it is the micro.
These services are smaller. Thats
just a convention, of course, but
its an important convention.
The idea that a very small piece
of software would be very isolated and doing their own thing. If
you think about my example of
the order taking versus shipping,
of course those are too big to be
a single microservice. It would
probably be a little cluster of
them each. But this notion that
you would take them separately would come very natural in a
microservices architecture. So
thats one way, the big way, in
which I see that it helps DDD.

So when you say that shipping


an order would be a cluster of
microservices, does that mean
that you would think that the
bounded context would be
a cluster of microservices? Is
that what you are saying?
That is exactly what Im saying.
And this, by the way, kind of

points into where I think DDD


can be helpful to microservices,
because they have the same
problem that SOA had in the
sense that there is a vagueness
about who can understand the
interface of a service.
So within a bounded context, lets say the interior of a microservice, there are things that
are crystal clear, or at least they
could be. So lets say that weve
declared one of these microservices to be a context and every
concept in there is very consistent throughout. So we have said
that an order line item means
this, and it has certain properties
and certain rules about how you
combine it with other line items
and whatever; all of these things
are very clear in terms of their
language and their rules.
Now we go to the interface
of the service. And so there we
would have certain language -unless you view a service thats
just some strings and numbers
going in and out. But thats not
the way people view services
and not the way they do well-designed services. Well-designed
services have a kind of language
about what they do. They have a
contract.
So if we say, all right, well,
then when you send an order to
this microservice, this is what it
means. And I dont just mean the
fields that are in it. I mean this is
the concept of what it is. Now, if
you zoom out a little bit, you see
that typically what people do is
that they have little clusters of
services that essentially speak
the same language.
So if my team is working on
an ordering system, we may have
a model and we might -- lets say
we have five microservices and
they speak the same language
and weve worked hard to make
that language clear. And then
over here were saying, well, we
really are dealing with a different
set of problems. These microser-

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

vices speak a different language.


If you send an order over here, it
might mean a little bit different
thing. And even if we try to disambiguate with longer names,
we wont always get that right.
So its better to just say over
here is a different language. And
that means that if a message
goes from a microservice within
the cluster to another one, thats
going to be really easy. But if a
message goes from one of these
clusters to another, we might
need to put it through a component that can translate it. So this
is one way in which I think that
once we look at multiple microservices, we need to think do
clusters of them belong in different bounded contexts?
Theres also the issue of the
inside and the outside. The outside of these -- micro services are
what Im really talking about now
-- the outside of a microservice
might speak a different language
than the inside. You know you
might say pass an order to this
microservice or pass this stream
of orders to this microservice.
Inside its going to crunch away
with a model that views orders in
a statistical fashion, lets say, and
comes up with some recommendations or something.
Well, the interior then is using a quite different model. The
interior of that microservice is a
different bounded context. But
as I said, that whole cluster is
speaking the same language to
each other. So we have this interchange context where we define
what messages mean as they
move between contexts and
then we have the interior context
of each service.

It makes a lot of sense. So what


Im wondering is if a bounded context is usually a cluster
of microservices, is there any
way that you can think of to

tell whether certain functionality should be implemented


in a microservice on its own
or just be part of another microservice? Because obviously if there is a cluster that is a
bounded context, its not one
bounded context is one microservice, its a set of microservices. So Im wondering whether there is a rule that would
give us an idea of whether we
should break this functionality
apart into an additional microservice and a bounded context
doesnt seem to cut it.
So first of all, yeah, if a cluster
of microservices is a context, or
rather the exterior, the message
passing between them would be
a microservice, and then inside
of each is a -- sorry is a bounded context -- and then the inside of each is another bounded
context. But now youre saying,
well, suppose that we have a
new functionality we need to
put somewhere, should I put it
as another microservice in this
cluster?
Well, I think that this is basically though the same kind
of question we always have to
answer when were designing
things. Does this piece of functionality fit within some existing
structure? Or is it going to start
distorting that structure out of
a good shape? And if so, then
where else should I put it? I think
the factors that go into that, Im
not being too original here, is
how coupled is it to the other
functions within the cluster? Like
if theres a necessarily a chatty
relationship with three different
components of that cluster, then
it seems very likely were going to
want to keep it in that cluster.
Another factor though
would be the expressiveness of
that cluster. The expressiveness
of that particular bounded contexts language - does it express

this concept well? Can I extend


that language to express this
concept well? And if so, then it
might be a good fit. If not, then
how much price am I going to
pay in proliferation of different
bounded contexts? You know
theres a tradeoff, of course.
So theres no real answer
there. Its like heres where we
have to do good work as designers.

And thats probably the hard


part about it.
A little trial and error helps too.
Thats another reason to not be
too perfectionist. You wont get it
right anyway and save time for iteration. Go back and do it again.

Yeah, and do it better the next


time. Okay. So you already
mentioned the term CQRS. Can
you explain what that is?
I remember trying to understand
that for a couple of years. So I
will say that event sourcing and
CQRS came along at almost the
same time, and the community
of people that were working on
them was very interwoven and
the two things were not so clearly delineated; certainly not at the
time. But I do think there are two
distinct architectural concepts
there. They often work well together but its useful to think of
them separately.
The one that immediately made sense to me that just
spoke to me instantly was event
sourcing. And then CQRS was a
little bit less obvious to me. But
I do think its a good technique.
So in essence, CQRS says that you
break your system into components that either are read-only

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

17

things or they are things that


process updates.
So lets take the order taking; you know that ordering example. When a new order comes
in, in CQRS wed put that in the
form of a command. A command
meaning the C in CQRS. And the
command would be enter this
order or something like that.
Now, it goes to some microservice, lets imagine, whose job is
to take this kind of command
and see if it can be done; like it
might say, Oh, Im not going to
take this order because we no
longer carry that item.
So commands, as theyre
defined in CQRS, you can say no.
So that would be the response to
that. Or lets say, okay, we do go
ahead and we process the order
and we reduce the inventory and
we initiate a shipment and we
send a message about that. Some
events come out: some events
that say things like the inventory
has been reduced and another
event that says theres a new order that has to be shipped. This
is the responsibility of that command processing part.
Now the query, thats the
Q, sometimes youd say, well, a
user might want to look at the
catalog, decide what he wants to
order. The user might want to see
the status of his order, Has my
order been shipped yet? things
like that. So this is the Q part. I
want to see the status of my order. And the idea is that this part
of the system would be kept very
simple. There would be a part of
the system where youd have to
figure out how to ship an order.
But once it had been shipped,
youd update the status in a query part that would say this order
has been shipped.
So queries that way can
scale differently than the command processing. And in a system where you have to do a lot
-- if this were an e-commerce
system where we were handling
18

thousands of orders a minute,


but maybe were handling even
more queries, but we can scale
them independently, we can
recognize that queries take less
processing power perhaps; that
since theres no change happening, we dont have to worry
about consistency rules.
So the query part is very
simple and fast and scales that
way. The command part is where
we have to deal with all the issues
of, well, what if a command came
in to cancel the order and weve
already shipped it, what are the
rules around that? Does the command still get processed? I mean
it will get processed but does it
still get cancelled? On and on.
All that rule stuff goes into that,
figuring out how to respond to a
command.

So we should probably explain


that CQRS is Command Query
Responsibility Segregation if I
remember correctly?
Yes.

You already said that there is


a relation to event sourcing. It
seemed to me that the C part,
the commands, are the events
in event sourcing. Is that what
the relationship is like?
Well, I think you could have an
event source system that was not
CQRS. So for example, you could
just have a module that responds
to queries and also can process
commands, and if you had that
you wouldnt really be practicing CQRS because you wouldnt
be separating them. But another
thing is that in event sourcing,
lets say that we have an order
object. The old traditional OO
way of doing this is that that order object might have a -- it says

its been shipped. In event sourcing we say, well, we dont have a


field or anything like that. What
we have is a series of events and
when it shipped there is created
an event has shipped.
So when we want to know
the status of that order, we just
go and find the events relevant
to it and then roll them up. The
classic example might be -- well,
Ill use the example I first heard
Greg Young use to explain the
point.
So lets say that you are doing some kind of stock trading
application. Someone says sell
10,000 shares of IBM above a certain price. So this order goes out.
Its 10,000. And now the question
comes, well, how many shares
are still to be sold? So each time
we execute this order, lets say we
sell 1,000, and then in a separate
transaction we sell 2,000 more.
So here we have two events -really three events. One was sell
10,000, and then there were two
events that said: we sold 1,000,
and then another event that said
we sold 2,000. Now the question
is how much remains to be sold?
How much IBM should we sell
at that price? At the time of the
query, we can find what events
are visible to us and we can calculate it.
So in the old days, wed
have had that object and it
would have had 10,000 and then
the first sale comes in and wed
subtract 1,000. So now it would
say sell 9,000 and then another
2,000 come in and wed say sell
7,000. And in event sourcing systems dont even have that field.
Or if you do, its an immutable
field that expresses the original
order to sell 10,000 and then
youve got a completely separate
object, an event object, that says
we sold 1,000 and another one
that says we sold 2,000. If you
want to know, you can figure it
out. You look at all those events

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

and you say, 10,000 minus 1,000


minus 2,000.

And that is the concept of event


sourcing basically. So what Im
wondering is what is the relationship to DDD then of CQRS
and event sourcing?
Well, event sourcing I think its
easier to illustrate because its a
modeling technique. Its talking
about how you would represent
the domain. If you look at before
that and the emphasis on entities and values, this is placing the
emphasis in a little bit different
place. Its saying certain things
happen in the domain; in the domain we have got an order execution and that should be explicitly modeled.
So its really making the
model more explicit about the
changes. If you take the old style
OO system where things change
and the objects represent the
way things are in our most current view, and this also is the typical relational database approach
too; but they dont show you
what happened. They just show
you how things are now. Whereas event sourcing shifts and says
lets model the state change
rather than the state. And we can
derive the state from the state
change.
So now we say we executed an order; thats the thing
we store. We executed an order
again; thats the thing we store. If
you want to have that other view,
its just a rollup of this. So its really a modeling technique and
it emerged from trying to apply
DDD to certain kinds of problems
that were very event centric and
also where they had to get very
high volume. With this, you can
scale up the updates because if
your updates are very frequent
and your reads are less frequent,
for example, you can be inserting

events into the system without


having to update an object in
place every time.
The objects all become immutable, which has certain technical benefits, especially if youre
trying to scale things; parallelize
things. So I think it fit into DDD
so naturally because -- its really a revamping of the building
blocks, is one way to look at it,
but its a little more radical than
that.

One thing that Im really wondering about is if I look at DDD


and in particular, well, on the
model part, it really seems
to be an object-oriented approach, I guess, because there
are those value objects entities
and all these kinds of things.
Its rather easy to think about
how that would be implemented using object-oriented techniques. In the last few years
there has been a shift to functional programming. So do you
think that DDD can be applied
to functional programming
too, even though it was originally expressed in a rather object-oriented terms?
Yes, that is one of the big things
thats happened over these 11
years. The reason that everythings expressed in terms of
objects is because objects were
king in 2003, 2004; and what else
would I have described it as? People who wanted to address complex domains wanted to try to
develop a model of that domain
to help them deal with the complexity; they used objects. And
the building blocks were an attempt to describe certain things
that help those kind of models to
actually succeed.
Now, if you are going at it
from a functional point of view,
then your model is going to look
quite different, or rather your

implementation is going to look


quite different. I think that the
event sourcing actually points a
good way, because you know I
mentioned that if youve applied
full-on event sourcing, the objects are immutable, which is a
start toward the functional perspective; because instead of having objects we change in place,
we have some kind of data structure that we use a function to derive another data structure from.
So if you imagine then
an event source system where
-- and let me just throw microservices in. You have a microservice, pass some events to it and it
computes the results and passes
out another stream of the events
that say well as a consequence
of this, this is what happens. So
I pass in we executed a trade for
2,000 and we executed another
trade for 1,000 and it passes out
an event that says the order has
been reduced to 7,000, whatever,
So its pretty easy to imagine implementing that as a function actually, perhaps more natural than OO in fact. Youve got a
stream of events and you want to
use it to compute another stream
of events; that really cries out for
a function to me.

Yeah, absolutely. It sounds


somewhat like an Actor model
even.
Yes. Well, some people in the
DDD community have really been into the Actor model.
Vaughn Vernon, for example,
talked a lot about using Actor
model. Indeed, it does seem to
be a good fit. It seems like that it
corresponds closely to another
one of the building blocks which
we havent talked about yet. In
the original book, it talked about
a building book called aggre-

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

19

gate, which was sort of trying to


describe a set of objects which
would have rules about their
internal data consistency and
somehow would be allowed to
enforce those rules.
So people have said, well,
if you take that unit, that unit of
whatever Im trying to make consistent, at any given change and
you give that responsibility to
a single actor, now you imagine
an actor is receiving events, or
commands, and it has to figure
out whether it can maintain that
state in a consistent -- so move
from one state to another in a
way that respects the invariance
of that particular aggregate. And
so thats an application of Actor
model to pulling in a little bit of
the old aggregates plus events
and commands. A lot has been
going on when we start talking
about it. The way people really
build these things is so different.

respects these concepts and expresses them.


So theres a lot of different
state within a big system and an
aggregate says, well, one of the
things that will keep you from
going parallel, like you do in Actor, is that you have no boundary
where you can say that the result
of this computation does not immediately affect anything else;
that we can handle that asynchronously. And thats exactly
what aggregates do. They define
a subset of the state which has
rules about how you can change
that state. And you say any kind
of resulting changes elsewhere
will be handled asynchronously.
Thats what an aggregate
does. And its related to the domain because you have to look at
the business to know what really
can be changed independently;
where will there be consequences to getting things out of sync?

We should probably say a few


a words about Actor. Well, an
Actor is something that gets
events from the outside and
executes them sequentially. It
is a model for parallel computation where you have multiple
Actors exchanging events and
each of them works sequentially. But the system as a whole is
parallel because all these Actors work parallel on their own
event streams. Thats basically
the idea and that seems to be
a good fit to the aggregates as
you just said; the DDD aggregates.

Yes. It seems like a good fit of a


certain technology or technical
approach to a certain domain
approach.

Right. And your description is a


very nice summary of the technical properties of these things.
And if I try to describe why this
is so useful when we try to -- we
have these conceptual models
of the domain and were trying
to make a software system that

20

Yeah, because when we first


were doing the aggregate thing,
well before I wrote my book,
back in the late 90s at least, it
was difficult to implement the
aggregates; there wasnt really a
technical artefact to hang your
hat on. So the nice thing about
Actor is that it gives you something to say we have decided
that were going to make each
aggregate the responsibility of
one Actor. Now I can really tell
another programmer, okay, this
is my aggregate because I made
an Actor for it.
It really helps if you can
have an explicit thing. This is
why, by the way, I think objects
are still a valuable concept. It says
that heres a software artefact
that makes explicit something

in our conceptual model; that


theres this thing, an important
thing in the domain that we have
defined. Theres a certain state
around it. Theres some rules and
behaviours around it.
Personally, Ive taken a holiday from OO for a few years to
freshen up my thinking, but
we dont want to throw out the
baby with the bathwater. Making
things explicit is good.

Another technology that has


been on the rise in the last few
years is NoSQL. Is there any
relation between NoSQL and
DDD too or are they not related?
NoSQL, of course, is not -- unlike
event sourcing and CQRS -- people who came up with those concepts really were DDDers who
were trying to come up with a
better way to do DDD. Thats not
true at all with NoSQL; it came
from a totally different world.
They were very interested in
technical properties of what they
were doing and so on, and above
all I think the motivator of a lot
these things was speed. However, I actually think that its a great
boon for DDD.
But one of the biggest
handicaps that weve had for a
long time is this obsession with
everything being stored in a relational database. Data in a relational database has to be structured a certain way. In the days
when objects were dominant,
the relational database was also
still the dominant database. So
we used OR mappers, object-relational mappers. Of course,
people still use these; I say it as if
its in the past. And then people
would talk about the impedance
mismatch. Well, whats the impedance mismatch? It just says
that the fundamental conceptual
structure of an object is different

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

from the relational table or set


of tables. The way they relate to
each other is different.
The trouble here -- I think it
was Eric Meyer who I heard made
this point. He said when we say
NoSQL, we should make No be
an acronym for not only. So we
should say NotOnlySQL. And his
point was that the problem isnt
the relational database which is
a brilliant thing; theyre a wonderful, powerful tool. But when
you use them for everything,
then you encounter data that
doesnt fit into them, that isnt
the right shape, and you have
to twist them into that shape.
But when youre dealing with a
problem that fits them, theres
just nothing else -- theyre a fantastic tool but weve used them
for everything. Its hard to find
a tool that works well for everything, certainly objects; I think
thats another problem with objects because they were used for
everything, and of course theyre
not good at everything.
This relates to DDD because
were trying to take all these
different concepts from the domain, and we are trying to create
concepts that have a tangible
form in the software. That shape
-- sometimes theres a natural
shape to it that more often is object-like than is relational. If its
object-like, maybe you do want
to represent it as objects, but
then you have to cram it into a
relational table with relations.
So instead of that, maybe
you use a key value store, which
is a very natural fit to objects actually. Object structures really are
just references; to references of
references. Its got that same kind
of tree structure that -- graph
structure anyway -- though good
ones have more of a tree structure. So its a better fit to some
kinds of problems.
And then the nice thing
about NoSQL is that its a relatively diverse world. Theres the

graph databases, since I did mention graphs, but there are things
that are really nicely modeled as
graphs. If you say, How am I going to model this thing? sometimes people think modeling
means OO modeling. Oh, I have
to draw a UML diagram of it and
then implement it in C# or Java.
Thats not what modeling means.
Modeling means to create abstractions that represent important aspects of your problem and
then put those to work.
So sometimes the natural
abstraction is a graph. You want
to say, well, how do these people
relate to each other? You know,
the graph databases, Neo4j
and things like that, allow us to
choose a tool that actually fits
the kind of problem were trying
to solve. I dont now have to twist
it into objects and then figure
out how to do graph logic over
objects while, by the way, Im also
stuffing the object data into a relational database. Instead, I use
a graph database and ask graph
questions using a graph query
language. This is the world of
NoSQL to me that we can choose
a tool that fits well with the problem were trying to solve.

Yeah, and you know I agree with


you, and I think the main reason
that people think of it as primarily a scaling technique is because
thats where it came from; that
was the real driver behind it. It
probably took the absolute necessity of those big data people
having to as -- you know the equilibrium where we were so deeply
rooted to the relational database,
it would take something like that
to get us loose. But I do think
that the great opportunities for
NoSQL are in the world of complex problems where the things
we want to do just dont really fit
the relational model. Sometimes
they dont fit the OO model. We
can choose the thing that fits.

Thats actually a very good way


to sum it up. So thanks a lot for
taking the time. Thanks a lot
for all the interesting answers
and the interesting insights. I
enjoyed it a lot.
Oh, thank you.

I think the point that youre


making is quite important.
Obviously, what youre talking
about is how those NoSQL databases gives you an advantage concerning modeling
data, while a lot of people still
think that NoSQL is all about
scaling and big data issues. It
is one of the benefits, but its
probably not even the most
important one. Its more about
this flexibility, as you said, and
the more natural modeling
and different alternatives to
relational databases. So I think
thats a very good point.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

21

Watch on InfoQ

Service Architectures at Scale:


Lessons from Google and eBay

Randy Shoup has experience with service architecture at scale at Google and eBay. In his talk,
Service Architectures at Scale: Lessons from Google and eBay, he presents the major lessons
learned from his experiences at those companies.

Evolution of service
architectures
Service architectures of largescale systems, over time, seem to
evolve into systems with similar
characteristics.
In 1995, eBay was a monolithic Perl application. After five
rewrites, it is a set of microservices written in a polyglot of programming languages. Twitter,
on its third generation of architecture, went from a monolithic
Rails application to a set of polyglot microservices. Amazon.com
started out as a monolithic C++
application and moved to services written in Java and Scala.
Today, it is a set of polyglot microservices. In the case of Google
and eBay, there are hundreds to
22

thousands of independent services working together.


Unlike the old way of building services with strict tiers,
these services exist in an ecosystem with layers of dependencies.
These dependencies resemble
a graph of relationships rather
than a hierarchy.
These relationships
evolved
without centralized, top-down
design evolution
rather
than intelligent
design. Developers
create
services or extract them from
other
services

or products. Sometimes they


group these extracted services in
a common service. Services that
are no longer needed are deprecated or removed. Services must
justify their existence.
The diagram below is an example of a set of services at Goo-

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

gle. These services evolved to a


hierarchy of clean layers. The hierarchy was an emergent property; it was not designed that way.
Cloud Datastore is a NoSQL
service in Googles publicly available App Engine. Megastore
gives multiple-row transactions
and synchronous replication
among nearby data centers. Bigtable is a data-center-level structured storage of key-value pairs.
Everything is built on Googles
distributed file system Colossus.
At the lowest level is Borg, the
cluster-management infrastructure responsible for assigning
resources to processes and containers that need it.
Each layer adds something
not in the layer below, but is general enough to be used in the
layer above. Everything at Google runs on Colossus, and almost
everything uses Bigtable. Megastore has many different use
cases. It was originally written for
Google apps such as Gmail, and
the Cloud Datastore team ending up building on it.
This was never a top-down
design. It grew from the bottom up. Colossus was built first.
Several years later, Bigtable was
built. Several years later, Megastore came into being. Several
years after that, Cloud Datastore
migrated to Megastore.

Architecture without the


architect
Nobody at Google has the title of
architect. There is no central approval for technology decisions.
Mostly, individual teams make
technology decisions for their
own purposes.
The early days of eBay, central approval from its Architectural Review Board was required for
all large-scale projects. Despite
the great number of talented
people on that board, they usually got involved when it was far
too late to make changes. It ended up being a bottleneck. The

boards only influence was the


ability to say no at the last minute.
It would have been much
better to have these smart, experienced people work on something really usable by individual
teams in a library, or tool, or
service, or even a set of guidelines that people could use on
their own rather than having
the teams learn at the last minute that a particular replication
style (for example) was not going
to work.

Standardization without
central control
Standardizing the communication between IT services and the
infrastructure components is
very important.
At Google, there is a proprietary network protocol called
Stubby. Usually, eBay uses RESTful HTTP-style data formats. For
serialization formats, Google
uses protocol buffers; eBay tends
to use JSON. For a structured way
of expressing the interface, Google uses protocol buffers, eBay
usually uses a JSON schema.
Standardization occurs naturally because it is painful for
a particular service to support
many different network protocols with many different formats.
Common pieces of infrastructure are standardized without central control. Source-code
control, configuration-management mechanisms, cluster management, monitoring systems,
alerting systems, diagnostic debugging tools all evolve out of
conventions.
Standards become standards not by fiat, but by being
better than the alternatives.
Standards are encouraged rather
than enforced by having teams
provide a library that does, for
example, the network protocol.
Service dependencies on particular protocols or formats also encourage it.

Code reviews also provide a means for standardization. At Google, every piece of
code checked into the common
source-control system is reviewed by at least one peer programmer. Searching through
the codebase also encourages
standardization. You discover if
somebody else has done what
you need. It becomes easy to do
the right thing and harder to do
the wrong thing.
Nonetheless,
there
is
no standardization at Google
around the internals of a service.
There are conventions and common libraries, but no standardization. The four commonly used
programming languages are
C++, Java, Python, and Go. There
is no standardization around
frameworks or persistence mechanisms.
Proven capabilities that are
reusable are spun out as new
services, with a new team. The
Google File System was written
to support search and as a distributed, reliable file system, others used it. Bigtable was first used
by search, then more broadly.
Megastore was originally built
for Google application storage.
The Google App Engine came
from a small group of engineers
who saw the need to provide
a mechanism for building new
webpages. Gmail came out of an
internal side project. App Engine
and Gmail were later made available for the public.
When a service is no longer used or is a failure, its team
members are redeployed to other teams, not fired. Google Wave
was a failure, but the operational
transformation technology that
allowed real-time propagation
of typing events across the network ended up in Google Apps.
The idea of multiple people being able to concurrently edit a
document in Google Docs came
straight out of Google Wave.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

23

More common than a service being a failure is a new generation, or version of a service
that leads to deprecating the
older versions.

Building a service as a
service owner
A well-performing service in
a large-scale ecosystem has a
single purpose, a simple and
well-defined interface, and is
very modular and independent.
Nowadays, people call these microservices. While the word is
relatively new, the concept is relatively old. What has happened
is that the industry has learned
from its past mistakes.
A service owner has a small
team, typically three to five people. The teams goals are to provide client functionality, quality
software, stable performance,
and reliability. Over time, these
metrics should improve. Given
a limited set of people and resources, it makes sense to use
common, proven tools and infrastructure, to build on top of other services, and to automate the
building, deploying, operating,
and monitoring of the service.
Using the DevOps philosophy, the same team owns the
service from creation to deprecation, from design to deployment
to maintenance and operation.
Teams have freedom to choose
their technologies, methodologies, and working environment.
They also have accountability for
the results.
As a service owner, you are
focused on your service, not the
hundreds to thousands of services in the broader infrastructure. You do not have to worry
about the complete ecosystem.
There is a bounded cognitive
load. You only need, as they say
at Amazon, a team large enough
to be fed by two large pizzas. This
both bounds the complexity and
makes for high-bandwidth com-

24

munication. Conways law plays


out to your advantage.
The relationship between
service teams is very structured.
Although everyone is working
for the same company, you want
to think about other teams as
vendors or customers. You want
to be cooperative but very clear
about ownership and who is responsible for what. Defining and
maintaining a good interface is a
large part of it. The other critical
part is that the customer or client
team can choose whether or not
to use the service. No top-level
directive exists, for example, to
store data in Bigtable.
Teams end up defining a
service-level agreement that
their clients can rely on to meet
their own objectives. Otherwise,
the client can build whatever
functionality they need and that
new functionality could become
the next generation of the service.
To make sure that costs are
properly allocated, customer
teams pay for the use of a service
to meet the common economic
incentives. Things given for free
are not used optimally. In one
case, a service using App Engine
changed from 100% to 10% resource consumption overnight
when they had to pay for its use.
Begging and pleading that client
to reduce consumption did not
work because that team had other priorities. In the end, they got
better response times with the
reduced resource use.
On the other hand, since
the service team is charging
for use, they are driven to keep
service quality high by using
practices such as agile development and test-driven development. Charging for use provides
incentives for making small,
easy-to-understand changes. All
submitted code is peer reviewed.
A thousand-line change is not
ten times riskier than a hundred-line change but it is more

like a hundred times riskier. Every


submission to source-code control causes the automated tests
and acceptance tests to run on
all the dependent code. In aggregate, Google ends up running
millions of automated tests every
day, all in parallel.
Stability of the interface
is important. The key mantra is
never break your clients code,
so you often have to keep multiple versions of the interface, and
possibly multiple deployments.
Fortunately, most changes do
not affect the interface. On the
other hand, you do want to have
an explicit deprecation policy so
that you can move your clients to
the latest version and retire the
older version.
Predictable performance is
important. Service clients want
minimal performance variation.
Suppose a service has a median
latency of one millisecond, but at
the 99.99th percentile the latency is one second, so it is a thousand times slower about 0.1%
of the time. If you use 5,000 machines as a Google scale operation might, you are going to wind
up being 50% slow. Predictable
performance is more important
than average performance. Low
latency with inconsistent performance is not low latency at
all. It is also easier for clients to
program against a service with
consistent performance. The latency at the 99.99th percentile
becomes much more important as services use lots of other
services and lots of different instances.
Low-level details can be
important even in large systems.
The Google App Engine found
that periodic reallocation of C++
STL containers resulted in latency
spikes at a really low but periodic
rate that was visible to clients.
Large-scale systems are exposed to failures. While many failures can happen in software and
hardware, the interesting ones

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

are the sharks and backhoes.


Google has suffered network disruptions because sharks apparently like the taste of trans-Atlantic cables and bite through
them. Lots of fiber goes through
lightly populated areas of the
central United States. Once, a
guy digging several meters deep
to bury a horse with a backhoe
cut through a line of Google fiber. You can never predict these
occurrences but you need to be
able to recover from them.
Most disruptions occur by
mistake. Just like security needs
defense in depth, you need resilience in depth. You need to be
resilient to machine, cluster, and
data-center failures. You need
load balancing and flow control
when invoking other services because failures can happen on the
network and off the network. You
need to be able to rapidly roll
back changes that have errors.
You never deploy to hundreds, thousands, or tens of
thousands of machines at once.
You typically pick one system,
or one instance of a new software version, as a test canary in
a coal mine. If that looks good,
you stage a rollout to 10% or
20%. Then you do a staged rollout to the rest of the system.
Sometimes you may have to roll
back when you are at 50%. It is
extremely important that you are
able to rapidly roll back. Feature
flags, which allow you through
configuration to turn features
on and off, are very useful. They
allow you to decouple code deployment from feature deployment. You might completely deploy with the feature turned off,
and then turn on the features in
a staged fashion. If there is a performance issue, business failure,
or bug, you can turn it off faster
than having to roll back code.

Anti-patterns
You can never have too much
monitoring. You can have too
much alerting, so you want to
avoid alert fatigue.
Service-oriented architecture has gotten a bad name, not
because the ideas were wrong
but because of the mistakes that
industry made along the way
from lack of experience.
One anti-pattern is a service
that does too much. Amazon.
com, eBay, Twitter, and Google
have ecosystems of tiny, clean
services. A service that is too
large or has too much responsibility ends up being a miniature
monolith. It becomes difficult
to understand and very scary to
change. It ends up increasing or
instilling lots more upstream and
downstream dependencies than
you would otherwise want.
Shared persistence is another anti-pattern. If you share a
persistence layer among services,
you break encapsulation. People
can accidently, or on purpose, do
reads and writes into your service and disrupt it without going
through the public interface. You
end up unwittingly reintroducing coupled services.
The modern approach of
microservices has small, isolated
services independent of one another. The resulting ecosystems
are healthy and growing.

Nobody at Google
has the title of
architect. There is
no central approval
for technology
decisions. Most
technology
decisions are made
by individual teams
for their own
purposes.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

25

Watch on InfoQ

Randy Shoup on Microservices, Conways Law,


and Evolutionary Architecture

In April 2015, Shoup delivered From the Monolith to Microservices:


Lessons from Google and eBay at Craft Conference in Budapest. InfoQ
spoke with him after his lecture.

InfoQ: Could you brief us on


what you have been talking
about today?
Sure. Most of the big web sites
that we know about Amazon, eBay, Google, Twitter, etc.
started out as monoliths and
have ultimately all gone through
convergent evolutions, not coordinated in any way, and have
ended up at what we are starting
to call polyglot microservices. I
wanted to explore why that was
the case.
I talked a bit about a monolith and why you might want to
have one. I mean, we say monolith and we mean it as a slur often, but the reality is that for most
systems, or certainly most stages
26

of companies, that is a perfectly


appropriate architectural approach. You do not need to distribute if you do not need to. I
talked a little bit about the pros
and cons of monoliths and then
flipped over to what it looks like
in microservices and again, pros
and cons of this simple individual pieces but now you have
to deal with coordination among
them. I talked a little bit about
what is it like to be an owner of
a service in a large-scale microservice ecosystem, like at Google or Netflix. Then I closed with
some anti-patterns, which I have
personally committed in every
sense of the word committed,
and I talked about why those are
inappropriate and why you could
do something better.

InfoQ: You mentioned that a lot


of your experience comes from
large organizations such as
Google and eBay. Do you think
these lessons apply to smaller
organizations and is there, perhaps, a cut-off with respect to
the size of the organization?
That is an excellent question. Yes,
I think the lessons are applicable, but the better answer is that
there is a spectrum, and when
is it appropriate to apply one or
the other? It was not something
that I addressed in this talk specifically, but on my SlideShare, I
briefly mention phases of startups in particular and when it is
appropriate to take one step or
another step.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

In the early phase of a startup, we do not even have a business model, we dont have product market fit, we do not have a
product. So, it is inappropriate, I
think, to think about any architecture or even any technology.
If a WordPress blog or buying
ads on Google is the right way
for you to test your hypothesis
about how to move forward, you
should totally do that and not
build anything. Then there is a
phase where we have a product
market fit and we think people
are willing to pay for it, and now
we are trying to grow that business and typically that is slower
than we would like to ramp up.
Again, that is a situation where
we started from minimal. It is not
about the technology and it is
certainly not about scaling that
technology or the organization.
We typically have a group
of people that can fit around the
conference table. This is not the
point at which to split the architecture up into small services,
divide into small teams, etc. That
comes later! Right now, we are
one team and we are building
one thing: the simplest thing
that could possibly work. Then,
one hopes that you will start to
hit the limits of the monolithic

organization and the monolithic architecture and that is what I


call the scaling phase, where you
hit the certain inflection point.
The point seems to be in company size and organization size,
between that 20-25-person mark
and the 100-person mark.
I dont know why this is
true, but I have observed that
there are sort of stable points for
organization size. Everyone fits
around a conference table up to
20-25, and the next point seems
to be around 100. It is in that
transition point where you can
make a single team work, even
at 20-25. You are rickety, but you
can still behave as a single team
with fluid roles and so on at the
20-25 mark. But as soon as you
are beyond that, and certainly
if you scale up to 100, you need
to flip the organization and the
technology to subdivide into
teams with well-defined responsibilities, and that is a good point
at which to switch from a monolithic approach to what I would
term microservices.

InfoQ: If the organization has


grown to the point at which
they say, Now is the right
time, what are the first steps
either at a technical level or at
an organizational level that it
should take?
I am glad you asked that with
both technical and organizational aspects in mind. Conways law
teaches us that the organizational structure is reflected in architecture. So, it is maybe a bit counterintuitive, but when you are at
the point where the monolith is
slowing you down not earlier
the first step you should make,
or at least coextensive with dealing with the technology, is to
change the organization to the
structure you want to have. So,
subdivide into three-to-five-person teams typically two-pizza
in the Amazon.com metaphor
and have them be responsible
for individual parts.
That naturally lends itself
to the technology being split up.
There are lots of monoliths that
are very highly coupled, in fact
most of them, and so it is not a
trivial exercise to break them up.
As a practical matter, here is what
I recommend to people that I
consult with now. First, we think
it is a good idea to move to the
new model and so first we have
to agree to that.
Step zero is to take a real
customer problem, something
with real customer benefit, maybe a reimplementation of something you have or ideally some
new piece of functionality that
was hard to do in the monolith.
Do that in the new way, first, and
what you are trying to do there
is to learn from mistakes you inevitably will make going through
that first transition, right? You do
it in a relatively safe way, but at
the same time you do with real
benefit at the end. At the end, if
nothing else, you have produced

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

27

real tangible business value out


of this work.
That is step zero, and what
we have done is that we have
gotten comfortable with the
idea that we can do it. It hopefully achieves the goals of velocity
and isolation and so on that we
were expecting, and we have
learned a lot. Now, we should
go through the most important
problems, with the highest ROI,
vertical slice, with real customer
benefit next, and then keep going until you run out of patience.
That is how eBay did it. When
eBay went from version two,
which was a monolithic C++ DLL,
to a more partitioned, individual
Java application. When it went
through that transition, which
overall took many years, it first
did that step zero on more than
a pilot, way more than a prototype, but something that was
tangibly highly valuable was produced. Then eBay reverse-sorted
the pages of the site by revenue
and worked on the highest revenue ones first (and did a bunch of
things in parallel), which seems a
bit odd and a bit risky, except it
has already de-risked it by doing
that step zero.
So now you are saying, I
only have limited resources to
apply against this migration path
over time, and at some point I
am going to run out of ROI, I am
going to run out of organizational resources that I am interested
in investing in this. That is okay,
because you have done the biggest ones first. This certainly was
true in 2010 or 2011 when I was
last at eBay and it might still be
true there were still pages on
the site that were on version-two
architecture, simply because
they continue to work. They got
100,000 hits a day, no big deal,
and they were neither painful
enough to migrate nor had sufficient ROI to migrate. So, they just
stayed and they happily stayed.

28

InfoQ: In the talk you mentioned that, with Google, the


architecture was evolutionary
and not necessarily by design.
Amazon.com, Google, and the
like are known for having the
brightest and the best hands.
Do you think there are more
guidelines required for small
organizations?
Well, it is always nice to have the
best and the brightest, but I think
there are lots of good and bright
all around. There are many more
smart people that do not work at
Google and Amazon.com than
those who do work at Google
or Amazon.com. So I dont worry too much about that. But are
there guidelines for smaller organizations? Absolutely. And again,
the meta-point with all these
things is only solve problems
that you actually have. I think it is
great to talk about these things.
Maybe people find some value in
listening to me talk about them,
but I am increasingly trying to be
very clear, when I describe what
works well for eBay or Google,
to describe why that is true and
that everything is a trade-off.
Google and Amazon.com are
intentionally trying to optimize
for velocity of large-scale organizations, which means lots of
things moving in parallel with
little coordination. They behave
like a thousand tiny companies
rather than one monster company and that is why those companies move fast and other ones
do not. But in the case of smaller organizations, if you are all
one team, do not subdivide. You
should continue to be one team.
If you do subdivide into two or
three teams, be pragmatic and
subdivide the architecture step
by step not into a thousand different things, but in two things,
three things, 10 things, and so on.
I think it is important to know
that you are going to evolve.

Again, every successful company has evolved. Ill say it another way: no successful company
that we have ever heard of has
the same architecture today that
it had when it started. Dont get
too bitter and angry with yourself that the first thing you try is
not the thing that lasts forever.
In fact, if you had done the thing
that was going to live for five or
10 years when you started out,
we would have probably never
heard of you because you would
have spent all your time building
for some far future that never
came rather than building things
that met near-term customer
needs in near term.

InfoQ: One thing I picked up


from your talk was the need
to standardize among microservices the connectivity,
if youd like. Do you have any
guidelines for how to lead or
manage those standardization
efforts?
Sure. I just want to repeat that
part because so often large enterprises, many of whom I have
worked for, have this visceral idea
that we should never duplicate
effort and we should standardize
on technologies and operating
procedures and so on.
One of the things that may
be interesting to know about the
Netflixes and the Amazon.coms
and the Googles of the world is
that they tend not to standardize
on the internals of services. So, a
service has a well-defined interface boundary that isolates and
encapsulates a module within it,
and as long as the implementation respects the interface that
they export and have agreed to,
it really does not matter what is
inside. Is it Haskell? Is it Ruby? Is it
Basic? It actually should not matter as long as it meets the outside
needs, and that is what encapsu-

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

One of the things


that is maybe
interesting to know
about the Netflixs
and the Amazons
and the Googles of
the world is that
they tend not to
standardize on the
internals of services.

lation and isolation actually mean.


So, those big ecosystems do not
standardize the internals of services, although there are common
conventions and it is not like people are inventing new ways to do
things all the time. But what you
do need to standardize is the communication.
It is a big network and you
need to standardize in some sense
the arcs in the network, not the internals of the nodes. That should
match our intuition about, for example, how economies work or
how human interactions work.
When we are having a conversation, we have to agree on a common language. If I speak English
and you speak Hungarian back I
do not speak Hungarian, unfortunately that would not work.
It is the same with economic interactions. If you are a shop
owner and I want to buy some fruit
from you, we have this agreement:
I am going to pay you in some currency and that has some meaning.
But, by the same token, that does
not mean that we have to have a
global language or a global currency because in reality we do not
have either global language or
global currency. We just need to
agree on conventions for particular interactions. So, how do you
deal with that?
Well, Ill describe what happens at the Amazon.coms and
the Googles. They often start with
one thing, because they are small
at the time, and there is that one
standard that everybody communicates with, and if that is perfect
then it can always be that way.
But over time they are going
to learn that Oh, I can make this
faster and add more flow control.
There is a bunch of things that you
can add to a network protocol that
solve problems that you have at
scale. What happens in reality is
that there becomes version two
of the protocol, version three of
the protocol and so on, and over
time those things get kind of ad-

opted by more and more services,


as those services need the capabilities of the new version or as
the consumers of those services
demand, in some sense, the capabilities that are in that protocol. So
that is how it happens: evolutionarily more than by dictate.

InfoQ: What do you think is coming after microservices?


Maybe I am insufficiently imaginative, but microservices as a word
is new, but the concept is old. It is
SOA done properly. Are there any
other ways of organizing software?
Of course there are. But there is a
reason why the Amazon.coms and
the Googles and the Netflixes and
the Gilts and the Yelps and everybody is ultimately rediscovering
through convergent evolution this
same general concept. So I think
microservices is a real thing. Maybe the word will die, but I think
that if we have this conversation
in three or four years that there
will no longer be microservices in
anybodys talk titles. We will not be
talking about microservices, because it is just going to be a thing
that we do.
The analogy that I think of
here is NoSQL. If we were having
this conversation three or four
years ago, when the hot topic was
not Docker and microservices because neither of them existed, but
NoSQL systems. Now, it is not that
NoSQL systems have gone away, it
is not that they are not important
anymore, but the fact that Netflix
uses Cassandra is not the subject
of a talk is only a line item: Oh, we
use Cassandra. And that is sufficiently descriptive of that thing
that we do not say much more
about it. Anyway, I think that the
next thing about microservices is
that we will stop talking about microservices, but we will continue
doing them.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

29

Read on InfoQ

ABOUT THE INTERVIEWEES

Lessons Learned Adopting Microservices at


Gilt, Hailo, and nearForm

Richard Rodger is a technology entrepreneur who has been involved in the Irish Internet
industry since its infancy. Richard founded the Internet startup Ricebridge.com in 2003. He
subsequently joined the Telecommunication Software and Systems Group (TSSG) and became
CTO of one of its successful spin-off companies, FeedHenry Ltd. More recently, he became
CTO and founder of nearForm.com. Richard holds degrees in computer science (WIT) and
mathematics and philosophy (Trinity College, Dublin). Richard is a regular conference speaker
and is a thought leader on system architectures using Node.js. Richard is the author of Mobile
Application Development in the Cloud, published by Wiley. He tweets at @rjrodger and blogs
here.

Feidhlim ONeill has spent over 20 years working in a variety of tech companies in the UK and
US, from startups to NASDAQ 100 companies. He spent 10 years at Yahoo in a variety of senior
positions in service and infrastructure engineering. Feidhlim works at Hailo where he oversees
their new Go-language microservices platform built on AWS.

Adrian Trenaman is VP of engineering at Gilt. Ade is an experienced, outspoken software


engineer, communicator, and leader with more than 20 years of experience working with
technology teams throughout Europe, the US, and Asia in industries as diverse as financial
services, telecom, retail, and manufacturing. He specializes in high-performance middleware,
messaging, and application development, and is pragmatic, hard-working, collaborative,
and results-oriented. In the past, he has held the positions of CTO of Gilt Japan, tech lead at
Gilt Groupe Ireland, distinguished consultant at FuseSource, Progress Software, and IONA
Technologies, and lecturer at Maynooth University (formerly the National University of Ireland
Maynooth). He became a committer for the Apache Software Foundation in 2010, has acted as
an expert reviewer to the European Commission, and has spoken at numerous tech events. Ade
holds a Ph.D. in computer science from the National University of Ireland Maynooth, a diploma
in business development from the Irish Management Institute, and a B.A. (Mod.) in computer
science from Trinity College, Dublin.

30

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

If we were to consider Gartners Hype Cycle, microservices are


perhaps about to hit the peak of inflated expectations. There is a
good number of early adopters, and microservices are quite present
in the specialized media, including here on InfoQ. Successful
implementations like those at Amazon, Google, and Netflix, etc., have
demonstrated that this technology is viable and worth considering.

We interviewed representatives
from three companies Gilt,
Hailo, and nearForm who have
agreed to share their experiences
in either building a microservices
platform from scratch or in re-architecting a monolithic platform
by gradually introducing microservices. The interviewees are:
Adrian Trenaman, SVP of engineering at Gilt; Feidhlim ONeill,
VP of platform and technical
operations for Hailo; and Richard
Rodger, CTO of nearForm.

InfoQ: Please tell us about


the microservices adoption
process in your company. Why
microservices? What technologies have you used to implement them? How long did it
take?
Adrian Trenaman: Adoption is
high, now at approximately 300
services. Adoption was driven
by an organization structure of
autonomous, KPI-driven teams,
supported by integrated tooling
that makes it easy to create and
deploy services. Adoption was
also spurred by the adoption of
Scala as a new way to write services.
We had a number of large
monolithic applications and services. It was getting increasingly
harder to innovate fast as multiple teams committed to the
same codebase and competed
for test and deployment windows. Adopting a microservices
architecture offered smaller,

easy-to-understand units of deployment that teams can deploy


at will.
We are using Scala, SBT,
ZooKeeper, Zeus (Riverbed)
Transaction Manager, PostgreSQL, RDS, Mongo, Java, Ruby,
Backbone, Kafka, RabbitMQ, Kinesis, Akka, Actors, Gerrit, OpenGrok, Jenkins, REST, and apiDoc.
The adoption process took
a period of 1.5-2 years, and is ongoing.
Feidhlim ONeill: Hailo
went through a re-platforming
exercise and our new platform
was built from the ground up using microservices. Microservices
was just evolving as a viable software architecture and we felt it
supported how we wanted to
work.
We trialed a number of
technologies and ultimately decided on a combination of what
we knew (Cassandra, ZooKeeper, etc.) and some new technologies. Selecting Go as our primary language was one of the
riskiest choices but has paid off.
From project kick off to the first
live components was about six
months. The full migration was
around 12 months.
Richard Rodger: We are an
enterprise Node.js consultancy
(one of the largest!), so we were
naturally drawn towards the microservice style, as it is a natural
fit for the lightweight and network-friendly nature of Node.js.
We began to adopt after inviting
Fred George, one of the earliest
advocates, to speak at one of our

meetups. We found him to be inspirational.


As we began to adopt
microservices, we tried out a
number of approaches. In some
sense, there is a tiering to the
architecture, in that many adoptees are simply spitting large
web apps into lots of little web
apps, whereas people like Fred
are going fully asynchronous
for each unit of business logic.
We have run all these variants in
production and what we have
found is that this choice is not as
important as it looks on the surface. More important is to provide a message-transportation
layer between services that abstracts this question away. Then
you have the freedom to arrange
communications between your
services as appropriate, whilst
ensuring that your developers
do not have to worry about the
transport layer, or the evils of service discovery.
We use the microservice
architecture for a very simple
reason: we can build better systems and delivery them more
quickly. It is much easier to deal
with changing requirements, before and after go-live, because
you change only small pieces at
a time, rather than making highrisk full redeployments. Microservices are easy to specify and
test. If you think about, they are
black boxes that react to certain
messages, and possibly emit
certain other messages. This is
a very clean interface that you
can define and test very clearly.
Scaling is much easier. The whole

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

31

application does not have to be


performant, only the parts that
are the bottlenecks. And you
can scale them by adding more
instances of a given service
a service that, by definition, is
stateless, and therefore easy to
scale linearly. Finally, microservices make project management
so much easier. Each microservice should take about a week
to write. This gives you nice, easy
blocks of effort to work with. If
a developer makes a mess, you
can, literally, genuinely, throw
the code away and start again.
Its only a week of work. This
means that technical debt accumulates much more slowly. And
that helps you move faster.
We build using Node.js,
and it really is perfect for microservices. For communication between services, we have an abstraction layer (http://senecajs.
org) that gives us the flexibility
we need. As weve grown, weve
found the need to build some
services in other languages. This
can happen for many reasons:
performance, integration, or
simply the availability of talent
both internally and at our clients.
Weve defined the abstraction
layer as a simple protocol so that
its easy to add new services in
other languages. Calling it a protocol is almost too much in fact
its really just the exchange of
JSON documents, embellished
with some pattern matching. For
message transport weve used
everything from point-to-point
HTTP and TCP to web sockets to
messaging systems, and even Redis Pub/Sub.
Our learning and adoption
took about two years to fully
develop a high-performing approach. These days, theres so
much more reference material,
books, case studies, and conference talks that this time is
much shorter. In fact, speaking
of books, our team is writing at
least two on the subject of mi32

croservices, so look out for those


later this year.

InfoQ: When does it make


sense to do microservices?
AT: It makes sense:
When you can isolate a piece
of domain functionality that
a single service can own.
When the service can fully
own read and write access to
its own data store.
When multiple teams are
contributing to a monolithic
system but keep on stepping
on each others toes.
When you want to implement continuous deployment.
When you favor an emergent
architecture rather than a
top-down design.
FO: We really wanted to
have parallel mission product-development teams that were fully
independent. By decomposing
our business logic into hundreds
of microservices, we are able to
sustain parallel changes across
multiple business lines.
RR: It makes sense whenever you prefer to keep your business open, even if that means
losing a percentage of revenue
due to errors. Its funny how the
world of enterprise software
seems to glorify the notion of
absolute correctness. Every database and application should
only have ACID transactions. And
then, when you ask the leadership of those organizations
which they prefer, you find that
keeping the shop doors open is
much more important. For example, consumer barcodes are not
always accurate the price at
the till does not always match the
price on the label. Supermarkets
somehow seem to stay open.
Microservices, as an architecture, value availability over

consistency. They keep your site,


mobile app, or service up and
running. There will be errors in
some percentage of the data.
You get to tune that percentage
by increasing capacity, but you
never get away from it completely. If your business can tolerate
errors, then microservices are for
you.
Obviously, there are systems that need to be 100% accurate. And the best way to achieve
this is with large-scale (and
expensive) monoliths, both in
terms of software and hardware.
Financial, medical, and real-time
systems are obvious examples.
But there are large amounts of
software that is pointlessly slow
and expensive to build simply
because we arent paying attention to business realities.

InfoQ: What are some of the


difficulties in implementing
microservices?
AT:
Its hard to replicate production into a staging environment. You need to either test
in production or invest in
sandbox/stage automation.
Ownership you end up
with a lot of services. As
teams change, services can
become orphaned.
Performance the call stack
can become complex with
cycles and redundant calls.
You can solve this with lambda-architectural approaches.
Deployment you need to
have clear, consistent technology on how to continuously deploy software.
Client dependencies
avoid writing service clients
that pull in alarge numbers of dependent libraries,
which can lead to conflicts.
Also, rolling out en-masse

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

changes to those libraries is


time consuming.
Audit and alerting you
need to move towards tracking and auditing business
metrics rather than just
low-level performance metrics.
Reporting having decentralized your data, your data
team will probably need
you to send data to the data
warehouse for analysis. You
need to look at real-time
data transports to get the
data out of your service.
Accidental complexity the
complexity of the system
moves out of the code and
gets lost in the white space,
the interconnectivity between services.
FO: [You need to resist]
the temptation to go live before
you have all the automation and
tooling complete. For example,
debugging 200 services without the right tools at the macro (whats changed) and micro
(trace) levels is neigh on impossible. You need to figure in automation and tooling from day
one, not as an afterthought.
RR: Not abstracting your
message transportation will end
in tears. Typically, people start
out by writing lots of small webservers, a sort of mini-SOA with
JSON, then run into problems
with dependency management
and the really nasty one, service
discovery. If your service needs
to know where any other services are on the network, youre
heading for a world of pain. Why?
Youve just replicated a monolith,
but now your function calls are
HTTP calls: not a huge win once
things get big. Instead, think
messages first. Your service sends
messages out into the world, but
does not know or care who will
get them. Your service receives
messages from the world, but
does not know or care who sent
them. Its up to you as architect

to make this a reality, but its not


that hard. Even if you are doing point-to-point behind the
scenes for performance reasons,
still make sure your service code
does not know this, by writing a
library to serve as an abstraction
layer.

so much more expensive, so it


makes sense for the team to be
micromanaged. In the microservice world, because the basic engineering approach is just much
better suited to underspecified
and rapidly changing requirements, you have less need for
control. Again, one week of bad
code wont kill, and youll see it
right away.

InfoQ: What are the benefits of


implementing microservices?
AT:
Faster time to market.
Continuous deployment.
Easy to understand components (notwithstanding that
the complexity sometimes
just moves elsewhere in the
system).
Easy to create, easy to tear
down (although you need to
have a clean shop mentality).
FO: Parallel continuous deployments and ease of refactoring services.
RR: The big benefit is speed
of execution. You and your team
will deliver faster on an ongoing
basis because you have reduced
deployment risk (its easy to roll
back, just stop the new service!)
and removed the need for big
refactoring its only a week
of code, and youve removed
hard-coded dependencies of language platforms or even things
like databases.
The other benefit is that you
have less need of project-management ceremony. Microservice systems suffer so much less
from the standard pathologies of
software development that strict
development processes are not
as effective in ensuring delivery.
Its easy to see why high levels of
unit-test coverage are a must for
monoliths. Or that pair programming is going to help, or any of
the agile techniques. The costs of
technical debt in a monolith are

InfoQ: How do microservices


compare to a traditional SOA
system?
AT: For me, microservices is just
taking SOA further, adapting the
concept to avoid monolithic services/codebases and focusing on
delivering continuous innovation to production across multiple teams.
FO: Decomposing the business logic into independent services is probably the main takeaway. Someone once described
microservice architecture to me
as a SOA design pattern and I
guess that makes a lot of sense.
There are lots of similarities
monolith versus micro being the
main difference.
RR: Its a radically different
approach. Theres no concept of
strict schemas. Theres an insistence on small services. Theres
recognition that the edges are
smart and the network dumb so
complexity does not build up in
weird places. You dont have to
deal with versioning issues. Why?
You run new and old versions of
a service together, at the same
time, and gradually migrate over,
all the while watching your performance and correctness measures. The lack of strict schemas
is exactly what makes this possible.

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

33

Watch on InfoQ

Building a Modern Microservices Architecture


at Gilt: The Essentials

Since 2010, Lead Software Engineer Yoni Goldberg has led the engineering behind several
critical projects at Gilt--including personalization, the Gilt Insider loyalty program, SEO/
optimization, and other customer-facing initiatives.

After living with microservices for three years, Gilt can see advantages
in team ownership, boundaries defined by APIs, and complex problems
broken down into small ones, Yoni Goldberg explained in a presentation
at the QCon London 2015 conference. Challenges still exist in tooling,
integration environments, and monitoring.
Goldberg, lead software engineer at Gilt, describes the company as a flash-sales business. A
typical sale offers a limited but
discounted inventory starting at
a specific time and running for a
specific period, usually 36 hours.
With tens of thousands of people
coming to the website at once
to buy items, Gilt experiences an
extreme and short spike in traffic
that generates about 80% of the
revenue. Every decision that may
affect website performance has
to take into consideration this
traffic spike of 50 to 100 times
the regular traffic.

34

As a traditional startup in
2007, Gilt used Ruby on Rails,
PostgreSQL, and Memcached.
Things went great but two years
later they had a 200,000-line
codebase and increasing traffic
overloaded the thousands of required Ruby processes running
with a database. With everyone
working on the same codebase,
deployment could take up to two
weeks due to all integration tests
needed. The biggest hurdle was
that if something went wrong,
they had a really hard time finding the root cause.

Macro/microservices
era
At this point, besides moving to
the JVM, Gilt entered what Goldberg calls a macro/microservices
era. He distinguishes between
a macroservice that handles a
specific domain, e.g. sales or payments, and a microservice that
you get by breaking a macroservice down into smaller services.
Gilt created 10 macroservices for
the core business, services that
are still in use. With all other services depending on these, the
core services need to perform

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

with good SLAs to keep downtime to a minimum. Their checkout service is one example of a
core: when that service is not
responding, users cant place
orders and the company then
doesnt make any money. A set
of less-critical supporting services, e.g. for user preferences,
use the core services; these are
good for the user experience but
the business will still function if
one goes down. On top of these
supporting services, another set
of services generates views for
all users.
While Gilt built services, it
also introduced dedicated data
stores for each service, providing
the best database for each ones
need. This new architecture
solved 99% of their scaling problems but left developers with
some of the problems as the
new services were semi-monolithic and lacked clear ownership
of code. Problems with deployments and long integration cycles remained. The main problem, though, was that it wasnt
fun to develop code.

Moving to microservices
To overcome the remaining
problems, Gilt created a lot more
microservices and empowered
teams to take responsibility for
not only developing a service
but also for testing, deploying, and monitoring it. This also
clarified ownership; a team basically became the owner of a
service. According to Goldberg,
the biggest benefit came from
the smaller scope of a microservice, which made it easier to
grasp. Its easy to understand a
service composed of just a few
thousand lines, and to understand another teams microservice when you move there to
contribute as a developer. The
architecture removed the huge
pain point of deployment dependency among teams. Now,
they could move to continuous

deployment with each team deciding for itself when to deploy,


even multiple times a day.
During this move, Gilt
started to work with what it
calls LOSA, lots of small applications, by breaking webpages
into smaller pieces basically microservices for web apps.
This lets teams work more independently from other teams and
Goldberg thinks it has created a
lot of innovation and a focus on
the user experience.

Current challenges
Despite Gilts successful move
from Rails to a microservices
architecture, Goldberg emphasizes that the company still has
some core challenges.
Deployment
From the start, each team
semi-manually deployed services with its own, different
method. A lack of integration
made it hard to execute tests to
make sure a change didnt break
something else. Gilt solved this
by building a tool around sbt
that helped teams to first deploy to an integration test environment and then to release to
production. During the last year,
the company has been working
to bring operations to the teams,
adopting Docker and moving to
the cloud. One downside Goldberg notes is that deployments
now are slower, but he hopes
that it will speed up in the coming years.
APIs
During the last year, Gilt has
been moving away from a RPC
style of communication, instead
building REST APIs. The main advantage Goldberg sees is that a
well-defined API solves a couple
of problems, most importantly discoverability. Because all
APIs are available in one place,
finding what is available can be
done with one search. The API

will also provide documentation; by looking at the models


and the resources, its possible to
understand whats available and
how its meant to work. With the
documentation generated from
the code, it will always be correct
and reflect any changes done to
the exposed resources.
Dependencies
All these microservices have
a lot of dependencies among
them. The biggest challenge
Goldberg sees for developers is
that for every change they have
to make sure they dont break
any other service. If they do a
breaking change, they must do
it in small steps and all clients
must be moved to the new endpoint before the old one can be
deleted. Another problem they
have experienced is that many
of their small web applications
repeat calls to one or several
services to, for example, generate a user profile. To limit all
these redundant calls, Gilt has
created what Goldberg calls a
mid-tier microservice, a service
that knows the calls needed to
create the user profile and which
the web applications can call instead. This mid-tier microservice
knows how to optimize, maybe
by caching, to reduce the number of calls made.
Ownership
As in most organizations, staff
turns over at Gilt. With all the
microservices around, the company must make sure that they
have enough developers who
understand the different codebases and for Goldberg, the
main solution is code reviews.
When every commit needs to be
reviewed by at least one other
developer, it increases the possibility that one more developer really understands the code.
Goldberg also emphasizes that
teams and not people owns
services because even though

Architectures you Always Wondered About // eMag Issue 31 - Aug 2015

35

individuals may leave, teams usually stay


longer. Teams also have the ability to
transform a service between teams, which
he really values.
Another concept is data ownership.
The move to one database per microservice has resulted in around a hundred relational databases and Gilt must manage
the schema for each. Goldberg describes
how Gilt completely separates the schema
from the service code, which brings some
subtle but important details: changes are
required to be incremental and there are
no rollbacks, so they have to be really conscious about every change they make.
Monitoring
With so many dependencies among services, Goldberg emphasizes that monitoring is important. Gilt uses several opensource tools to get the different matrices
it especially cares about e.g. New Relic,
Boundary, and Graphite but has also developed its own monitoring system, CAVE,
also open source. CAVEs basic function
is to set up rules and to raise alerts for
example, when the total order value of all
US shipments during any five-minute window drops below a threshold value or if
the 99th-percentile response time exceeds
a set value. This is a technique Goldberg
finds better than metrics.

Takeaway
For Goldberg, the biggest advantage Gilt
has gained from microservices is ownership by team. He believes that when team
members own a service, they tend to treat
it like their baby. Another big promise of
microservices he mentions is the breaking
of complex problems into small ones that
everyone can understand, one service at a
time.
Two challenges that Goldberg thinks
still stand are monitoring, for lack of tooling, and the integration and developer environment.
Goldbergs final advice for starting
with microservices is to begin with a feature that does not yet exist. Build that as
a microservice. He thinks that it is hard to
get acceptance to break down something
that already exists and works. Building
something new will be much easier for
people to accept.

PREVIOUS ISSUES

29

QCon New York 2015


Report

This eMag provides a round-up of what we and some


of our attendees felt were the most important sessions
from QCon New York 2015.

28

Advanced DevOps
Toolchain

30

Description, Discovery,
and Profiles - The
Next Level in Web APIs

This eMag focuses on three key areas of meta-language for


Web APIs: API Description, API Discovery, and API Profiles.
Youll see articles covering all three of these important trends
as well as interviews with some of the key personalities in
this fast-moving space.

In this eMag we provide both implementation examples and comparisons of different possible approaches on a range of topics from immutable infrastructure
to self-service ops platforms and service discovery. In
addition, we talk about the Docker ecosystem and the
different aspects to consider when moving to this increasingly popular system for shipping and running applications.

27

Technical Debt and


Software Craftsmanship

This e-mag brings together a number of authors who


have addressed the topic on InfoQ and suggests ways
to identify, monetize, and tackle technical debt in your
products.

You might also like