R

Previous Next

DOWNLOAD PDF

Previous Next
FEBRUARY 2014

ZeroMQ:
Previous Next

The Design
Previous Next

of Messaging
Download

Middleware
A look at how one of the most popular messaging
Subscribe

layers was designed and implemented. >>

Powered by

PLUS: 10 Things Tech Writers Want Developers to Know >>

 Register
Previous Next

DOWNLOAD PDF

Previous Next
ZeroMQ: The Design of Messaging Middleware
Previous Next
Building a large-scale distributed system in a systematic manner
By Martin Sstrik
Previous Next

Download

Subscribe MQ is a messaging system, or

Education And Networking
Learn how cloud computing,
software-defined networking,
virtualization, wireless, and other
key technologies work together
to drive business at Interop Las
Vegas. It happens March 31
to April 4.
Register
Previous Next
message-oriented middleware if
you will. It is used in environments
as diverse as financial services,
game development, embedded
systems, academic research, and aerospace.
Messaging systems work basically as instant
messaging for applications. An application
decides to communicate an event to another
application (or multiple applications), it as-
sembles the data to be sent, hits the send
button, and the messaging system takes care
of the rest. Unlike instant messaging, though,
messaging systems have no GUI and assume
no human beings at the endpoints capable
of intelligent intervention when something
goes wrong. Messaging systems thus have to
be both fault-tolerant and much faster than
common instant messaging.
MQ was originally conceived as an ultra-fast
messaging system for stock trading and so the
focus was on extreme optimization. The first
year of the project was spent devising bench-
marking methodology and trying to define an
architecture that was as efficient as possible.
Later on, approximately in the second year
of development, the focus shifted to provid-
ing a generic system for building distributed
applications and supporting arbitrary mes-
saging patterns, various transport mecha-
nisms, arbitrary language bindings, etc.
During the third year, the focus was mainly
on improving usability and flattening the
learning curve. We adopted the BSD Sockets
API, tried to clean up the semantics of indi-
vidual messaging patterns, and so on.
This article will give insight into how the
three goals above translated into the internal
architecture of MQ, and provide some tips
for those who are struggling with the same
problems.
Since its third year, MQ has outgrown its
codebase; there is an initiative to standard-
ize the wire protocols it uses, and an experi-
mental implementation of a MQ-like mes-
saging system inside the Linux kernel, etc.
These topics are not covered here. However,
you can check online resources for further
details (http://250bpm.com/concepts and
http://is.gd/giDEEh).
Application vs. Library
MQ is a library, not a messaging server. It
took us several years working on the AMQP
protocol, which was a financial industry at-
tempt to standardize the wire protocol for
business messaging writing a reference

www.drdobbs.com February 2014 2

Previous Next
 Register
Previous Next
implementation for it and participating in
Previous Next
several large-scale projects heavily based
on messaging technology to realize that
Previous Next
theres something wrong with the classic cli-
ent/server model of a smart messaging server
(broker) and dumb messaging clients.
Previous Next
Our primary concern was with the perfor-
mance: If theres a server in the middle, each
message has to pass the network twice (from
Download the sender to the broker and from the broker
to the receiver) inducing a penalty in terms of
both latency and throughput. Moreover, if all
the messages are passed through the broker,
at some point, the server is bound to become
Subscribe the bottleneck.
A secondary concern was related to large-
scale deployments: when the deployment
crosses organizational boundaries the con-
cept of a central authority managing the
whole message flow doesnt apply anymore.
No company is willing to cede control to a
server in a different company due to trade se-
crets and legal liability. The result in practice
is that theres one messaging server per com-
pany, with hand-written bridges to connect
it to messaging systems in other companies.
The whole ecosystem is thus heavily frag-
mented, and maintaining a large number of
bridges for every company involved doesnt
www.drdobbs.com
make the situation better. To solve this prob-
lem, we need a fully distributed architecture,
an architecture where every component can
be possibly governed by a different business
entity. Given that the unit of management
in server-based architecture is the server, we
can solve the problem by installing a separate
server for each component. In such a case we
can further optimize the design by making
the server and the component share the same
processes. What we end up with is a messag-
ing library.
MQ was started when we got an idea
about how to make messaging work with-
out a central server. It required turning the
whole concept of messaging upside down
and replacing the model of an autonomous
centralized store of messages in the center
of the network with a smart endpoint, dumb
network architecture based on the end-to-
end principle. The technical consequence of
that decision was that MQ, from the very
beginning, was a library, not an application.
Weve been able to prove that this archi-
tecture is both more efficient (lower latency,
higher throughput) and more flexible (its
easy to build arbitrary complex topologies in-
stead of being tied to classic hub-and-spoke
model) than standard approaches.
[ZEROMQ]
One of the unintended consequences was
that opting for the library model improved
the usability of the product. Over and over
again users express their happiness about
the fact that they dont have to install and
manage a stand-alone messaging server. It
turns out that not having a server is a pre-
ferred option as it cuts operational cost (no
need to have a messaging server admin)
and improves time-to-market (no need to
negotiate the need to run the server with the
client, the management or the operations
team).
The lesson learned is that when starting a
new project, you should opt for the library
design if at all possible. Its pretty easy to cre-
ate an application from a library by invoking
it from a trivial program; however, its almost
impossible to create a library from an existing
executable. A library offers much more flex-
ibility to the users, at the same time sparing
them non-trivial administrative effort.
Global State
Global variables dont play well with libraries.
A library may be loaded several times in the
process but even then theres only a single
set of global variables. Figure 1 shows a MQ
library being used from two different and
February 2014 3
 Register
Previous Next
application
Previous Next
Previous Next IibA IibB
OMQ
Previous Next
Figure 1: MQ being used by different libraries.
Download independent libraries. The application then
uses both of those libraries.
When such a situation occurs, both in-
stances of MQ access the same variables,
resulting in race conditions, strange failures
Subscribe and undefined behavior. To prevent this prob-
lem, the MQ library has no global variables.
Instead, a user of the library is responsible for
creating the global state explicitly. The ob-
ject containing the global state is called con-
text. While from the users perspective con-
text looks more or less like a pool of worker
threads, from MQs perspective its just an
object to store any global state that we hap-
pen to need. In the picture above, libA would
have its own context and libB would have its
own as well. There would be no way for one of
them to break or subvert the other one.
The lesson here is pretty obvious: Dont use
global state in libraries. If you do, the library is
www.drdobbs.com
likely to break when it happens to be instanti-
ated twice in the same process.
Performance
When MQ was started, its primary goal
was to optimize performance. Performance
of messaging systems is expressed using
two metrics: throughput how many mes-
sages can be passed during a given amount
of time; and latency how long it takes for
a message to get from one endpoint to the
other.
Which metric should we focus on? Whats
the relationship between the two? Isnt it ob-
vious? Run the test, divide the overall time of
the test by number of messages passed and
what you get is latency. Divide the number
of messages by time and what you get is
throughput. In other words, latency is the in-
verse value of throughput. Trivial, right?
Instead of starting coding straight away we
spent some weeks investigating the perfor- average is 3 seconds, which is pretty far away
mance metrics in detail and we found out that
the relationship between throughput and
latency is much more subtle than that, and
often the metrics are quite counter-intuitive.
Imagine A sending messages to B (see Fig-
ure 2). The overall time of the test is 6 sec-
onds. There are 5 messages passed. Therefore,
[ZEROMQ]
A B
0 sec
1 sec
2 sec
3 sec
4 sec
5 sec
6 sec
Figure 2: Sending messages from A to B.
the throughput is 0.83 messages/sec (5/6)
and the latency is 1.2 sec (6/5), right?
Have a look at the diagram again. It takes a
different time for each message to get from
A to B: 2 sec, 2.5 sec, 3 sec, 3.5 sec, 4 sec. The
from our original calculation of 1.2 second.
This example shows the misconceptions peo-
ple are intuitively inclined to make about per-
formance metrics.
Now have a look at the throughput. The
overall time of the test is 6 seconds. However,
at A it takes just 2 seconds to send all the mes-
February 2014 4
 Register
Previous Next
sages. From As perspective the throughput is
Previous Next
2.5 msgs/sec (5/2). At B it takes 4 seconds to
receive all messages. So from Bs perspective,
Previous Next
the throughput is 1.25 msgs/sec (5/4). Neither
of these numbers matches our original calcu-
lation of 1.2 msgs/sec.
Previous Next
To make a long story short, latency and
throughput are two different metrics; that
much is obvious. The important thing is to un-
Download derstand the difference between the two and
their relationship. Latency can be measured
only between two different points in the sys-
tem; there is no such thing as latency at point
A. Each message has its own latency. You can
Subscribe average the latencies of multiple messages;
however, theres no such thing as latency of a
stream of messages.
Throughput, on the other hand, can be
measured only at a single point of the sys-
tem. Theres a throughput at the sender,
theres a throughput at the receiver, theres
a throughput at any intermediate point be-
tween the two, but theres no such thing as
overall throughput of the whole system. And
throughput make sense only for a set of mes-
sages; theres no such thing as throughput of
a single message.
As for the relationship between throughput
and latency, it turns out there really is a relation-
www.drdobbs.com
ship; however, the formula involves integrals
and we wont discuss it here. For more infor-
mation, read the literature on queuing theory.
There are many more pitfalls in benchmarking
the messaging systems that we wont go fur-
ther into. The stress should rather be placed on
the lesson learned: Make sure you understand
the problem you are solving. Even a problem as
simple as make it fast can take lot of work to
understand properly. Whats more, if you dont
understand the problem, you are likely to build
implicit assumptions and popular myths into
your code, making the solution either flawed or
at least much more complex or much less use-
ful than it could possibly be.
Critical Path
We discovered during the optimization pro-
cess that three factors have a crucial impact
on performance:
1. Number of memory allocations
2. Number of system calls
3. Concurrency model
However, not every memory allocation or
every system call has the same effect on per-
formance. The performance we are interested
in messaging systems is the number of mes-
[ZEROMQ]
sages we can transfer between two endpoints
during a given amount of time. Alternatively,
we may be interested in how long it takes for a
message to get from one endpoint to another.
However, given that MQ is designed for
scenarios with long-lived connections, the
time it takes to establish a connection or the
time needed to handle a connection error is
basically irrelevant. These events happen very
rarely and so their impact on overall perfor-
mance is negligible.
The part of a codebase that gets used very
frequently, over and over again, is called the
critical path; optimization should focus on the
critical path.
Lets have a look at an example: MQ is not
extremely optimized with respect to memory
allocations. For example, when manipulating
strings, it often allocates a new string for each
intermediate phase of the transformation.
However, if we look strictly at the critical
path the actual message passing well
find out that it uses almost no memory al-
locations. If messages are small, its just one
memory allocation per 256 messages (these
messages are held in a single large allocated
memory chunk). If, in addition, the stream
of messages is steady, without huge traffic
peaks, the number of memory allocations on
February 2014 5
 Register
Previous Next [ZEROMQ]

the critical path drops to zero (the allocated memory chunks are not
returned to the system, but reused repeatedly).
Previous Next Small Message Large Message
Lesson learned: Optimize where it makes difference. Optimizing
pieces of code that are not on the critical path is wasted effort. handle handle
Previous Next
data pointer
Allocating Memory
Assuming that all the infrastructure was initialized and a connection
Previous Next
between two endpoints has been established, theres only one thing
to allocate when sending a message: the message itself. Thus, to opti- data
mize the critical path we had to look into how messages are allocated
dynamically allocated memory
Download and passed up and down the stack.
Figure 3: Message copying (or not).
Its common knowledge in the high-performance networking
field that the best performance is achieved by carefully balancing
the cost of message allocation and the cost of message copying (for It makes sense to allocate the message once and pass a pointer
example, different handling of small, medium, and large messages; to the allocated block, instead of copying the data. This approach is
Subscribe see http://hal.inria.fr/docs/00/29/28/31/PDF/Open-MX-IOAT.pdf ). called zero-copy.
For small messages, copying is much cheaper than allocating mem- MQ handles both cases in a transparent manner. A MQ mes-
ory. It makes sense to allocate no new memory chunks at all and sage is represented by an opaque handle. The content of very small
instead to copy the message to preallocated memory whenever messages is encoded directly in the handle. So making a copy of
needed. For large messages, on the other hand, copying is much the handle actually copies the message data. When the message
more expensive than memory allocation. is larger, its allocated in a separate buffer and the handle contains

www.drdobbs.com February 2014 6

 Register
Previous Next [ZEROMQ]

App Lesson learned: When thinking about per- especially if the messages are small and hun-
Previous Next
formance, dont assume theres a single best dreds of them can be packed into a single
MQ solution. batch.
Previous Next
It may happen that there are several sub- On the other hand, batching can have
TCP classes of the problem (for example, small negative impact on latency. Lets take, for
messages vs. large messages), each having its example, the well-known Nagles algorithm,
Previous Next IP
own optimal algorithm. as implemented in TCP. It delays the out-
NIC bound messages for a certain amount of
Figure 4: Sending four messages.
Batching time and merges all the accumulated data
Download It has already been mentioned that the sheer into a single packet. Obviously, the end-
number of system calls in a messaging system to-end latency of the first message in the
App can result in a performance bottleneck. Actu- packet is much worse than the latency of
ally, the problem is much more generic than the last one. Thus, its common for applica-
MQ
that. Theres a non-trivial performance pen- tions that need consistently low latency to
Subscribe TCP
alty associated with traversing the call stack switch Nagles algorithm off. Its even com-
and thus, when creating high-performance mon to switch off batching on all levels of
IP applications, its wise to avoid as much stack the stack (for example, a NICs interrupt
traversing as possible. coalescing feature). But no batching means
NIC Consider Figure 4. To send four messages, extensive traversing of the stack and results
you have to traverse the entire network stack in low message throughput. We seem to
Figure 5: Batching messages.
four times (MQ, glibc, user/kernel space be caught in a throughput versus latency
just a pointer to the buffer. Making a copy boundary, TCP implementation, IP implemen- dilemma.
of the handle doesnt result in copying the tation, Ethernet layer, the NIC itself and back MQ tries to deliver consistently low laten-
message data, which makes sense when the up the stack again). cies combined with high throughput using
message is megabytes long (Figure 3). It However, if you decide to join those mes- the following strategy: When message flow is
should be noted that in the latter case the sages into a single batch, there would be only sparse and doesnt exceed the network stacks
buffer is reference-counted so that it can be one traversal of the stack (Figure 5). The im- bandwidth, MQ turns all the batching off to
referenced by multiple handles without the pact on message throughput can be over- improve latency. The trade-off here is some-
need to copy the data. whelming: up to two orders of magnitude, what higher CPU usage we still have to
www.drdobbs.com February 2014 7
 Register
Previous Next [ZEROMQ]

network
traverse the stack frequently. However, that isnt considered to be a
Previous Next
problem in most cases.
When the message rate exceeds the bandwidth of the network stack,
Previous Next
the messages have to be queued stored in memory till the stack is worker: thread 1 worker: thread 2
ready to accept them. Queuing means the latency is going to grow. If
the message spends one second in the queue, end-to-end latency will be engine engine
Previous Next
at least one second. Whats even worse, as the size of the queue grows,
listener session session
latencies will increase gradually. If the size of the queue is not bound, the creates
latency can exceed any limit.
Download It has been observed that even though the network stack is tuned for
pipes
lowest possible latency (Nagles algorithm switched off, NIC interrupt
coalescing turned off, and so on) latencies can still be dismal because
of the queuing effect, as described above.
In such situations it makes sense to start batching aggressively.
Subscribe Theres nothing to lose as the latencies are already high anyway. On
the other hand, aggressive batching improves throughput and can socket
empty the queue of pending messages which in turn means the
latency will gradually drop as the queuing delay decreases. Once
there are no outstanding messages in the queue, the batching can be
API
turned off to improve the latency even further.
application thread
One additional observation is that the batching should only be done
at the topmost level. If the messages are batched there, the lower lay- Figure 6: MQ architecture.
ers have nothing to batch anyway, and so all the batching algorithms
underneath do nothing except introduce additional latency. Architecture Overview
Lesson learned: To get optimal throughput combined with optimal Up to this point I have focused on generic principles that make MQ
response time in an asynchronous system, turn off all the batching fast. Now, lets have a look at the actual architecture of the system
algorithms on the low layers of the stack and batch on the topmost (Figure 6).
level. Batch only when new data are arriving faster than they can be The user interacts with MQ using so-called sockets. They are
processed. pretty similar to TCP sockets, the main difference being that each
www.drdobbs.com February 2014 8
 Register
Previous Next
socket can handle communication with multi-
Previous Next
ple peers, a bit like unbound UDP sockets do.
The socket object lives in the users thread
Previous Next
(see the discussion of threading models in
the next section). Aside from that, MQ is
running multiple worker threads that handle
Previous Next
the asynchronous part of the communication:
reading data from the network, queuing mes-
sages, accepting incoming connections, etc.
Download There are various objects living in the
worker threads. Each of these objects is
owned by exactly one parent object (owner-
ship is denoted by a simple full line in the
diagram). The parent can live in a different
Subscribe thread than the child. Most objects are owned
directly by sockets; however, there are couple
of cases where an object is owned by an ob-
ject which is owned by the socket. What we
get is a tree of objects, with one such tree per
socket. The tree is used during shut down; no
object can shut itself down until it closes all
its children. This way we can ensure that the
shutdown process works as expected; for ex-
ample, that pending outbound messages are
pushed to the network prior to terminating
the sending process.
Roughly speaking, there are two kinds of
asynchronous objects: objects that are not in-
volved in message passing and those that are.
www.drdobbs.com
The former have to do mainly with connec-
tion management. For example, a TCP listener
object listens for incoming TCP connections
and creates an engine/session object for each
new connection. Similarly, a TCP connector
object tries to connect to the TCP peer and
when it succeeds it creates an engine/session
object to manage the connection. When such
connection fails, the connector object tries to
re-establish it.
The latter are objects that are handling
data transfer itself. These objects are com-
posed of two parts: the session object is
responsible for interacting with the MQ
socket, and the engine object is respon-
sible for communication with the network.
Theres only one kind of the session object,
but theres a different engine type for each
underlying protocol MQ supports. Thus, we
have TCP engines, IPC (inter-process commu-
nication) engines, PGM engines (a reliable
multicast protocol, see RFC 3208), etc. The
set of engines is extensible in the future
we may choose to implement, say, a Web-
Socket engine or an SCTP engine.
The sessions are exchanging messages with
the sockets. There are two directions to pass
messages in and each direction is handled by
a pipe object. Each pipe is basically a lock-free
[ZEROMQ]
queue optimized for fast passing of messages
between threads.
Finally, theres a context object (discussed in
the previous sections but not shown on the
diagram) that holds the global state and is
accessible by all the sockets and all the asyn-
chronous objects.
Concurrency Model
One of the requirements for MQ is to take
advantage of multi-core boxes; in other
words, to scale the throughput linearly with
the number of available CPU cores.
Our previous experience with messaging
systems showed that using multiple threads
in a classic way (critical sections, semaphores,
etc.) doesnt yield much performance improve-
ment. In fact, a multi-threaded version of a
messaging system can be slower than a sin-
gle-threaded version, even if measured on
a multi-core box. Individual threads are sim-
ply spending too much time waiting for each
other while, at the same time, eliciting a lot of
context switching that slows the system down.
Given these problems, we decided to go for a
different model. The goal was to avoid locking
entirely and let each thread run at full speed.
The communication between threads was
to be provided via asynchronous messages
February 2014 9
 Register
Previous Next [ZEROMQ]

(events) passed between the threads. This, as insiders know, is the classic kind of cooperative multitasking. This means we needed a scheduler;
Previous Next
actor model. objects need to be event-driven rather than being in control of the
The idea was to launch one worker thread per CPU core having entire event loop. That is, we have to take care of arbitrary sequences
Previous Next
two threads sharing the same core would only mean a lot of context of events, even very rare ones and we have to make sure that no object
switching for no particular advantage. Each internal MQ object, such holds the CPU for too long; etc.
as say, a TCP engine, would be tightly bound to a particular worker In short, the whole system had to become fully asynchronous. No
Previous Next
thread. That, in turn, means that theres no need for critical sections, object can afford to do a blocking operation, because it would not
mutexes, semaphores and the like. Additionally, these MQ objects only block itself but also all the other objects sharing the same worker
wont be migrated between CPU cores so would thus avoid the nega- thread. All objects must become, whether explicitly or implicitly, state
Download tive performance impact of cache pollution (Figure 7). machines. With hundreds or thousands of state machines running in
parallel you have to take care of all the possible interactions between
worker: thread 1 worker: thread 2 them and most importantly of the shutdown process.
It turns out that shutting down a fully asynchronous system in a
clean way is a dauntingly complex task. Trying to shut down a thou-
Subscribe connection connection sand moving parts, some of them working, some idle, some in the
process of being initiated, some of them already shutting down by
connection themselves, is prone to all kinds of race conditions, resource leaks and
connection similar. The shutdown subsystem is definitely the most complex part of
listener MQ. A quick check of the bug tracker indicates that some 30--50% of
reported bugs are related to shutdown in one way or another.
connection connection
Lesson learned: When striving for extreme performance and scal-
ability, consider the actor model; its almost the only game in town in
such cases. However, if you are not using a specialized system like Er-
lang or MQ itself, youll have to write and debug a lot of infrastruc-
Figure 7: Multiple worker threads.
ture by hand. Additionally, think, from the very beginning, about
the procedure to shut down the system. Its going to be the most
This design makes a lot of traditional multi-threading problems complex part of the codebase and if you have no clear idea how to
disappear. Nevertheless, theres a need to share the worker thread implement it, you should probably reconsider using the actor model
among many objects, which in turn means there needs to be some in the first place.
www.drdobbs.com February 2014 10
 Register
Previous Next [ZEROMQ]

Lock-Free Algorithms
Previous Next queue reader: thread 1
Lock-free algorithms have been in vogue lately. They are simple mech-
anisms for inter-thread communication that dont rely on the kernel-
Previous Next
provided synchronization primitives, such as mutexes or semaphores; writer thread queue reader: thread 2
rather, they do the synchronization using atomic CPU operations, such
as atomic compare-and-swap (CAS). It should be understood that they
Previous Next
are not literally lock-free instead, locking is done behind the scenes queue reader: thread 3
at the hardware level.
MQ uses a lock-free queue in pipe objects to pass messages be-
Figure 8: Queues.
Download tween the users threads and MQs worker threads. There are two
interesting aspects to how MQ uses the lock-free queue.
owned by writer thread owned by reader thread
First, each queue has exactly one writer thread and exactly one
reader thread. If theres a need for 1-to-N communication, multiple
queues are created (Figure 8). Given that this way the queue doesnt
Subscribe have to take care of synchronizing the writers (theres only one writer) pre-write pre-read
or readers (theres only one reader) it can be implemented in an extra-
efficient way. writer: flush reader: flush
Second, we realized that while lock-free algorithms were more effi-
cient than classic mutex-based algorithms, atomic CPU operations are
Figure 9: Lock-free queue.
still rather expensive (especially when theres contention between CPU
cores) and doing an atomic operation for each message written and/or
each message read was slower than we were willing to accept. portion of the queue thats accessed solely by the writer thread, and
The way to speed it up was batching once again. Imagine you had then flush it using a single atomic operation.
10 messages to be written to the queue. It can happen, for example, The same applies to reading from the queue. Imagine the 10 mes-
when you receive a network packet containing 10 small messages. sages above were already flushed to the queue. The reader thread
Receiving a packet is an atomic event; you cannot get half of it. This can extract each message from the queue using an atomic operation.
atomic event results in the need to write 10 messages to the lock-free However, its overkill; instead, it can move all the pending messages
queue. Theres not much point in doing an atomic operation for each to a pre-read portion of the queue using a single atomic operation.
message. Instead, you can accumulate the messages in a pre-write Afterwards, it can retrieve the messages from the pre-read buffer one
www.drdobbs.com February 2014 11
 Register
Previous Next
by one. Pre-read is owned and accessed solely by the reader thread
Previous Next
and thus no synchronization whatsoever is needed in that phase.
The arrow on the left of Figure 9 shows how the pre-write buffer can
Previous Next
be flushed to the queue simply by modifying a single pointer. The arrow
on the right shows how the whole content of the queue can be shifted
to the pre-read by doing nothing but modifying another pointer.
Previous Next
Lesson learned: Lock-free algorithms are hard to invent, trouble-
some to implement and almost impossible to debug. If at all possible,
use an existing proven algorithm rather than inventing your own.
Download When extreme performance is required, dont rely solely on lock-free
algorithms. While they are fast, the performance can be significantly
improved by doing smart batching on top of them.
API
Subscribe The user interface is the most important part of any product. Its the
only part of your program visible to the outside world and if you get it
wrong, the world will hate you. In end-user products its either the GUI
or the command line interface. In libraries its the API.
In early versions of MQ the API was based on AMQPs model of
exchanges and queues. (See the AMQP specification at http://www.
amqp.org/specification/1.0/amqp-org-download.) From a historical
perspective its interesting to have a look at the white paper from
2007 (http://zeromq.org/whitepapers:messaging-enabled-network)
that tries to reconcile AMQP with a brokerless model of messaging.
I spent the end of 2009 rewriting it almost from scratch to use the
BSD Socket API instead. That was the turning point; MQ adoption
soared from that point on. While before it was a niche product used
by a bunch of messaging experts, afterwards it became a handy com-
monplace tool for anybody. In a year or so, the size of the community
www.drdobbs.com
[ZEROMQ]
increased tenfold, some 20 bindings to different languages were
implemented, etc.
The user interface defines the perception of a product. With basically
no change to the functionality just by changing the API MQ
changed from an enterprise messaging product to a networking
product. In other words, the perception changed from a complex
piece of infrastructure for big banks to hey, this helps me to send my
10-byte-long message from application A to application B.
Lesson learned: Understand what you want your project to be and de-
sign the user interface accordingly. Having a user interface that doesnt
align with the vision of the project is a 100% guaranteed way to fail.
One of the important aspects of the move to the BSD Sockets API
was that it wasnt a revolutionary freshly invented API, but an existing
and well-known one. Actually, the BSD Sockets API is one of the oldest
APIs still in active use today; it dates back to 1983 and 4.2BSD Unix. Its
been widely used and stable for literally decades.
The above fact brings a lot of advantages. Firstly, its an API that
everybody knows, so the learning curve is very short. Even if youve
never heard of MQ, you can build your first application in couple of
minutes thanks to the fact that you are able to reuse your BSD Sockets
knowledge.
Additionally, using a widely implemented API enables integration of
MQ with existing technologies. For example, exposing MQ objects
as sockets or file descriptors allows for processing TCP, UDP, pipe,
file and MQ events in the same event loop. Another example: the ex-
perimental project to bring MQ-like functionality to the Linux kernel
(https://github.com/250bpm/linux-2.6 ) turned out to be pretty simple
to implement. By sharing the same conceptual framework it can re-use
a lot of infrastructure already in place.
February 2014 12
 Register
Previous Next
Most importantly, the fact that the BSD Sockets API has survived
Previous Next
almost three decades despite numerous attempts to replace it means
that there is something inherently right in the design. BSD Sockets
Previous Next
API designers have whether deliberately or by chance made the
right design decisions. By adopting the API we can automatically share
those design decisions without even knowing what they were and
Previous Next
what problem they were solving.
Dont succumb to Not Invented Here
Download syndrome.
Lesson learned: While code reuse has been promoted from time im-
memorial and pattern reuse joined in later on, its important to think
Subscribe of reuse in an even more generic way. When designing a product, have
a look at similar products. Check which have failed and which have
succeeded; learn from the successful projects. Dont succumb to Not
Invented Here syndrome. Reuse the ideas, the APIs, the conceptual
frameworks, whatever you find appropriate. By doing so you are al-
lowing users to reuse their existing knowledge. At the same time you
may be avoiding technical pitfalls you are not even aware of at the
moment.
Messaging Patterns
In any messaging system, the most important design problem is that
of how to provide a way for the user to specify which messages are
routed to which destinations. There are two main approaches, and I
believe this dichotomy is quite generic and applicable to basically any
problem encountered in the domain of software.
www.drdobbs.com
[ZEROMQ]
One approach is to adopt the UNIX philosophy of do one thing and
do it well. What this means is that the problem domain should be ar-
tificially restricted to a small and well-understood area. The program
should then solve this restricted problem in a correct and exhaustive
way. An example of such approach in the messaging area is MQTT
(http://mqtt.org). Its a protocol for distributing messages to a set of
consumers. It cant be used for anything else (say for RPC) but it is easy
to use and does message distribution well.
The other approach is to focus on generality and provide a powerful
and highly configurable system. AMQP is an example of such a system.
Its model of queues and exchanges provides the user with the means
to define programmatically almost any routing algorithm they can
think of. The trade-off, of course, is a lot of options to take care of.
MQ opts for the former model because it allows the resulting prod-
uct to be used by basically anyone, while the generic model requires
messaging experts to use it. To demonstrate the point, lets have a look
how the model affects the complexity of the API. What follows is an
implementation of an RPC client on top of a generic system (AMQP):
connect (192.168.0.111)
exchange.declare (exchange=requests, type=direct,
passive=false,
durable=true, no-wait=true, arguments={})
exchange.declare (exchange=replies, type=direct,
passive=false,
durable=true, no-wait=true, arguments={})
reply-queue = queue.declare (queue=, passive=false,
durable=false, exclusive=true, auto-delete=true,
no-wait=false, arguments={})
February 2014 13
 Register
Previous Next [ZEROMQ]

queue.bind (queue=reply-queue, exchange=replies, s = socket (REQ)

Previous Next
routing-key=reply-queue) s.connect (tcp://192.168.0.111:5555)
queue.consume (queue=reply-queue, consumer-tag=, s.send (Hello World!)
Previous Next
no-local=false, reply = s.recv ()
no-ack=false, exclusive=true, no-wait=true, arguments={})
request = new-message (Hello World!)
Previous Next
request.reply-to = reply-queue Up to this point, weve argued that specific solutions are better than
request.correlation-id = generate-unique-id () generic solutions. We want our solution to be as specific as possible.
basic.publish (exchange=requests, routing-key=my-service, However, at the same time, we want to provide our customers with as
Download mandatory=true, immediate=false) wide a range of functionality as possible. How can we solve this appar-
reply = get-message () ent contradiction?
The answer consists of two steps:
On the other hand, MQ splits the messaging landscape into so-
called messaging patterns. Examples of the patterns are publish/sub- 1. Define a layer of the stack to deal with a particular problem area
Subscribe scribe, request/reply or parallelized pipeline. Each messaging pat- (transport, routing, presentation, etc.).
tern is completely orthogonal to other patterns and can be thought of 2. Provide multiple implementations of the layer. There should be a
as a separate tool. separate non-intersecting implementation for each use case.
What follows is the re-implementation of the above application using
MQs request/reply pattern. Note how all the option tweaking is reduced Consider the example of the transport layer in the Internet stack. It
to the single step of choosing the right messaging pattern (REQ): is meant to provide services such as transferring data streams, apply-

www.drdobbs.com February 2014 14

 Register
Previous Next
ing flow control, providing reliability, etc., on
Previous Next
the top of the network layer (IP). It does so by
defining multiple non-intersecting solutions:
Previous Next
TCP for connection-oriented reliable stream
transfer, UDP for connectionless unreliable
packet transfer, SCTP for transfer of multiple
Previous Next
streams, DCCP for unreliable connections and
so on. Note that each implementation is com-
pletely orthogonal: A UDP endpoint cannot
Download speak to a TCP endpoint. Neither can a SCTP
endpoint speak to a DCCP endpoint.
This means that new implementations can
be added to the stack at any moment without
affecting the existing portions of the stack.
Subscribe Conversely, failed implementations can be
forgotten and discarded without compromis-
ing the viability of the transport layer as a
whole.
The same principle applies to messaging pat-
terns as defined by MQ. Messaging patterns
form a layer (the so-called scalability layer)
on top of the transport layer (TCP and friends).
Individual messaging patterns are implemen-
tations of this layer.
They are strictly orthogonal the publish/
subscribe endpoint cant speak to the request/
reply endpoint, etc. Strict separation between
the patterns, in turn, means that new patterns
can be added as needed and that failed experi-
www.drdobbs.com
ments with new patterns wont hurt the exist-
ing patterns.
Lesson learned: When solving a complex
and multi-faceted problem, it may turn out
that a monolithic general-purpose solution
may not be the best way to go. Instead, we
can think of the problem area as an abstract
layer and provide multiple implementations
of this layer, each focused on a specific well-
defined use case. When doing so, delineate
the use case carefully.
Be sure about what is in the scope and what
is not. By restricting the use case too aggres-
sively, the application of your software may
be limited. If you define the problem too
broadly, however, the product may become
too complex, blurry, and confusing for the
users.
Conclusion
As our world becomes populated with lots of
small computers connected via the Internet
mobile phones, RFID readers, tablets and
laptops, GPS devices, etc. the problem
of distributed computing ceases to be the
domain of academic science and becomes a
common everyday problem for every devel-
oper to tackle. The solutions, unfortunately,
are mostly domain-specific hacks.
[ZEROMQ]
This article summarizes our experience with
building a large-scale distributed system in
a systematic manner. It focuses on problems
that are interesting from a software architec-
ture point of view, and I hope that designers
and programmers in the open source com-
munity will find it useful.
Martin Sstrik is an expert in the field of
messaging middleware. He participated in the
creation and reference implementation of the
AMQP standard and has been involved in vari-
ous messaging projects in the financial indus-
try. He is a founder of the MQ project, and
currently is working on integration of messag-
ing technology with operating systems and
the Internet stack. This article is excerpted
and adapted from The Architecture of Open
Source Applications: Volume II (http://is.gd/
pv0IqW)made available under the Creative
Commons Attribution 3.0 Unported license.
Please see the full description of the license
for details at http://aosabook.org/en/intro1.
html#license
Month 2013 15
 Register
Previous Next
DOWNLOAD PDF
Previous Next 10 Things Tech Writers Want Developers
Previous Next to Know
Good documentation is often what makes good software great. Creating quality docs is the
Previous Next result of good habits and paying attention to the right details.
By Edward J. Joyce
I
Download
know a developer Ill call Bob (his real name
is Robert). Bob does great work, producing
Subscribe quality software in short order, but he has a
common weak point: documentation. This
begs the question: Should Bob even be
concerned with writing documentation? Bob is
paid to write code, not technical prose.
But Bob the developer does have docu-
mentation responsibilities as the supplier of
raw materials. He must deliver specifications,
screen shots, diagrams, demonstrations, re-
corded sessions, and other relevant infor-
mation to a technical writer who will sculpt
it all into easily assimilated knowledge for
consumption by end users. Project manag-
ers, product marketers, support staff, execu-
tives, sales representatives, and ultimately
www.drdobbs.com
customers rely on that information. Without
decent documentation, the best software
and hardware may well be destined to be
shelf-ware.
To help Bob provide premium raw materi-
als to the documentation machinists on the
assembly line, here are ten tried and true tips
culled from seasoned technical writers.
Crisp, Clear, Concise Communication
Sometimes it takes more than cajoling, coax-
ing, and cursing to convince a technologist
to write crisp, clear, concise communication.
Sometimes it takes a federal edict. In 2010,
President Barack Obama and Im not mak-
ing this up signed the Plain Writing Act,
a law that requires federal agencies to use
[DOCUMENTATION TIPS]
communication that the public can under-
stand and use.
The Center for Plain Language has taken
the law a step further by citing violations
in communication clarity from all organiza-
tions. It bestows a WonderMark after the
Centers judges read a document, shake
their heads, and say We wonder what they
meant.
The following two examples are parts of
text that have earned the Centers Wonder-
Mark awards:
Our goal is to keep pace with all new techni-
cal and security-related requirements. For this
reason, we inform you today about the transi-
tion to a new PHP version. Please note that after
February 2014 16
 Register
Previous Next
the transition, the option Register Globals will
Previous Next
not be supported anymore.
Previous Next
and
Welcome to the online Policies and Proce-
Previous Next
dures Manual. Use the buttons on the left to
view the table of the contents, or to search for
documents by title or document ID number. To
Download download any of the software packages please
click the appropriate button on this page.
Avoid WonderMark-class prose, and instead
strive for ClearMark awards, which celebrate
Subscribe the best in clear communication and plain
language.
Precise Instructions
If your life revolves around a niche software
product such as Pixels with Personality, you
can probably navigate the menus blindfolded.
The statement Use the Preferences dialog to
change the cursor is a clear roadmap for you.
If youre a technical writer charged with
writing tomes for several software products,
that statement may not get you far because
launching the Preferences dialog could very
well require different navigation in the vari-
ous products.
www.drdobbs.com
Instead of leaving unmarked forks on the
highways of the roadmap, nip any budding
confusion by using explicit terminology such
as Use the dialog under Edit/Preferences/
Cursors to change the cursor format.
Avoid OON
(Overheating the Optic Nerve)
Overheating the Optic Nerve (OON) syn-
drome has yet to be chronicled in the Jour-
nal of the American Medical Association;
but if you read manuals, user guides, and
technical specifications, you have no doubt
suffered its symptoms. It starts as gentle
pressure in the forehead while you peruse
technical literature searching for an elusive
piece of information.
How does a developer cause OON? By satu-
rating text with excessive, excruciating, exas-
perating details such as this example from the
introduction of a product manual:
The shipping contents of the WS-1035 in-
clude a touch-screen LCD monitor station, a
433 MHz wireless temperature sensor, the re-
spective connecting cables, a 110 VAC adapter,
and the software package on a 4.7 GB DVD
disk.
[DOCUMENTATION TIPS]
You can save wear and tear on the optic
nerve and cut the word count of the previous
paragraph in half with this abridgment:
The WS-1035 product includes a touch-
screen, remote temperature sensor, connecting
cables, power adapter, and the software pack-
age on disk.
There is a place for technical details. Its called
an appendix. Put the minutia at the back of
the book where it wont obscure the concepts
and principles that need to be conveyed to the
reader as the first order of business.
Purge Needless Politeness
Needless politeness means using words such
as please, thank you, and we appreciate your
devotion to our insanely great products.
Consider the statement: If you observe the
0x5422DA error 17 times in a minute, please
call our corporate technical support hotline
for advice. Thank you. That statement con-
tributes to deforestation by requiring un-
necessary paper stock to express a simple
instruction. No one will be offended by the
politeness-free form: If you observe the
0x5422DA error 17 times in a minute, call
technical support.
February 2014 17
 Register
Previous Next
Save those courtesies for making amends
Previous Next
with technical writers whom you offended
in the past with incomplete, inaccurate, or
Previous Next
incomprehensible drafts of documentation.
Accurate, Thorough, Timely Reviews
Previous Next
If you think of technical documentation as
a publication, the developer starts out as a
research assistant collecting facts, describing
Download processes, and assembling other raw informa-
tion for authors (the technical communica-
tors). At the end of the process, the research
assistant dons the hat of editor and reviews
and critiques the authors draft document.
Subscribe A review has three main properties: accu-
racy, thoroughness, and timeliness. Accuracy
is self-explanatory. The entire review rates a
failing grade if the end result is wrong.
Thoroughness, an often overlooked goal
of a technical review, distinguishes key ref-
erence documents from permanent fixtures
on dusty bookshelves. It means reviewing
the overall content rather than narrowly
focusing on your particular area of respon-
sibility.
For example, if you provided input for a Con-
figuration section describing a Cursor Selec-
tion dialog, you would feel obligated to review
it to ensure that it is accurate. What about the
www.drdobbs.com
Introduction, System Requirements, and In-
stallation sections of the document? Do they
include statements such as This product is not
compatible with Windows 95, or This product
requires a minimum of 2 GB RAM? The earnest
reader, when coming upon these statements,
is forced to wonder if the documentation has
not been updated since a bygone era when
Windows 95 and hardware with 2 GB or less of
RAM was in vogue.
Other red flags are entries in the Known
Issues section that list problems which were
corrected umpteen releases ago.
The final key to a good review is timeliness.
Dont wait until the 11th hour to deliver to
the technical writer a 200-page manual with
enough edits to drain several red pens. Give
the writer sufficient time to incorporate your
updates without creating panic during the
waning minutes of the release cycle.
Reasonable Edits
Related to timely reviews are reasonable
edits. In other words, trust the writers in-
stincts and dont pepper the draft with re-
wordings of sentences due to capricious
preferences.
You are guilty of capricious preferences if
you asked a writer to change text such as:
[DOCUMENTATION TIPS]
Use the dialog under Edit/Preferences/Cur-
sors to change the cursor format.
to
Choose the dialog under Edit/Preferences/
Cursors to change the cursor format.
and then to
Navigate to the dialog under Edit/Prefer-
ences/Cursors to change the cursor format.
Establish a threshold for revisions. If a pas-
sage is factually correct and accurately con-
veys the information, let it stand and trust
the writers instincts. Avoid the temptation
to micro-edit something just so it sounds
better in your opinion.
Revere Visuals
In the evolution of computing, documenta-
tion has marched lockstep with software
evolving from text to text-with-graphics to
hyperlinked prose to audio and video. As a
developer, you need to recognize that the
raw materials you put in the documentation
pipeline should also reflect a similar transfor-
mation.
February 2014 18
 Register
Previous Next
A video presentation from a developer
Previous Next
might be overkill, but you can at least share
your desktop for live demonstrations of the
Previous Next
software in action. A combination of a text-
based description and a good visual is source
input gold to a technical writer, said Brian Ev-
Previous Next
erett, Senior Technical Information Architect
at CA Technologies.
So, if you have a choice between feeding
Download gold or ordinary ore into the documentation
forge, opt for the gold.
Spell Out Acronyms
Consider the following statement:
Subscribe
MP performance can be impeded by noise in
an ISDN BRI which can also impact LCP negotia-
tion.
The acronyms in this passage can have a vari-
ety of meanings to the uninitiated. MP could be
interpreted as Multilink Protocol or MP3 (mis-
typed as MP) or if its part of an instruction
manual for an army command post Military
Police.
To avoid ambiguity, take a few seconds to
spell out acronyms and save yourself a round
of emails and phone calls from a befuddled
writer.
www.drdobbs.com
Be Monolingual
You might be tempted to show off your
worldly nature by interspersing a document
with Latin phrases and abbreviations such as
those in this example:
The wireless communication between the
base station and the remote module, i.e., the
client sensor, reaches distances of 100 meters in
unimpeded areas, e.g., direct line of sight, ergo
place the base and client in fields, beaches, ur-
ban lots, et al.
Consider the challenge of a translator who
is faced with translating this verbiage to
another language so the product can be
marketed in other countries. Those phrases
and abbreviations from the ancient Romans
constitute added effort to the jobs of the
translators. Not only must they translate the
English words, but they must also recognize
the Latin terms and translate them correctly.
Save foreign phrases for cocktail parties. In
technical documentation, be monolingual
and stick to plain language.
Expect Translation
If a product is successful in one country, its
apt to be successful in others. In that case, the
documentation that you and a technical writ-
[DOCUMENTATION TIPS]
ing team painstakingly crafted in English now
becomes raw material for translators.
As with the original draft documentation,
if the assembly line is filled with good qual-
ity materials that is, materials that were
prepared with translation as an option the
translators will have a better chance of quickly
producing foreign language versions.
The first factor to consider when preparing
documentation for translation is communica-
tion with the translator. The translator may
well be someone living on the other side of
the globe who has learned English as a sec-
ond language. Given cultural and time-zone
differences, most communication may be by
e-mail, as opposed to direct conversations
and meetings. So make an extra effort to
clarify any questions or concerns from the
translator in your email responses.
To help ensure that the documentation will
be translation-ready once it is delivered to the
translation team, here are some pointers:
Use the same term or expression for the
same concept. Consider this example:
To activate this software product, ensure
that you are logged in with administrator privi-
leges on the system. Download the kit to your
February 2014 19
 Register
Previous Next
computer. Run the installation utility to acti-
Previous Next
vate the program on your hardware.
Previous Next
In this case the words system, computer,
and hardware could all be replaced by com-
puter; and software product, kit, and program
Previous Next
could all be replaced by software. Consistency
in terminology will help dispel any ambiguity
and potential misinterpretation.
Download
Avoid expressions that are culture specific.
Your development colleagues may understand
this admonishment in a troubleshooting guide:
Subscribe The attribute not found error may be a red
herring. To level the playing field, run the data-
base table verification utility.
Good luck trying to translate that advice into
Mandarin. Instead, use the more universal:
The attribute not found error may be related
to another problem. Check the database by run-
ning the table verification utility.
Use plain words. To ease the translation
process and avoid violating the Plain Writing
Act, use simple words instead of pompous
synonyms. Here are a few examples:
www.drdobbs.com
[DOCUMENTATION TIPS]
Instead of: Use: Vertical lists, as shown in the second ital-
icized passage, are key to breaking down
ascertain find out technical information into easily assimilated
chunks. Use them liberally.
expedite hasten, speed up
Use active voice wherever possible. Com-
necessitate need, have to, require pare the active and passive forms of the fol-
lowing instruction:
prior to before Passive: Any change to the configuration
must be updated in the master file.
subsequent to after Active: When you change the configuration,
update the master file.
Use vertical lists. Read the following pas- The passive voice conceals who or what
sage a few times while noting the tempera- does the action, and it requires more words
ture of your optic nerve: than the active voice. In short, passive is more
difficult to translate than active, so choose ac-
In the Home screen, select one of the catego- tive wherever possible.
ries or select the Customize Selection to custom-
ize updates that you want to apply to the system If you get into the habit of following these
depending on the requirement. recommendations, your customers and your
colleagues will be thankful. Your customers
Now read: will especially reward you by using your prod-
ucts and using them correctly.
In the Home screen, do one of the following
depending on the requirement: Edward J. Joyce, a software engineering
manager, strives to provide source input gold
Select one of the categories. to his writer colleagues at CA Technologies.
Select the Customize Selection to
customize updates.
February 2014 20