Message Queues

* Beanstalkd * MQ (ZeroMQ) * RabbitMQ (AMQP)

Beanstalkd
Beanstalkd is a Job Queue Server. It listens on a TCP socket, where it receives jobs from other processes. These jobs stay in beanstalkd's job queue until a worker process reserves, executes them and finally deletes them. This allows the request processing process to be released, immediately after the job was put in the queue. Web Server HTTP request put(do-something) HTTP response reserve job beanstalkd Worker(s)

delete job

RabbitMQ (AMQP)
RabbitMQ is a Message Queue Broker, written in Erlang. It utilizes the openly specified AMQP (Advanced Message Queuing Protocol). It features 3 communication schemes: fanout (1-n), topic (1-n, aka pub/sub) and direct (1-1). Some use cases are: background processing by one or more workers, propagating events / updates from & to multiple nodes...

Use Case: fanout from web application

Web Server HTTP Request message HTTP Response
(published to a fanout exchange) all subscribed nodes receive the message

Database

RabbitMQ message broker

XMPP Gateway

... ...

MQ (ZeroMQ)
ZeroMQ is a message queue system, similar to AMQP, though a lot smaller. It doesn't rely on a separate broker process. Instead, one of the system's processes takes that role. Like AMQP, there are multiple communication schemes: request/response (1-1), pub/sub (1-n), ventilator/sink (1-x-1). The ventilator pattern works pretty much the same as beanstalkd (i.e. there are one or more workers, of which the first to shout here gets it), but also has a sink node, where workers push their results.

request/response
Client Response Request Server

publish/subscribe
Publisher

ventilator/sink
Ventilator

Worker

Worker

Worker

Subscriber Subscriber Subscriber

Sink

Resources
Beanstalkd:
Homepage: http://kr.github.com/beanstalkd/ Ruby Client: http://beanstalk.rubyforge.org/ Protocol Specification: https://github.com/kr/beanstalkd/blob/v1.1/doc/protocol.txt

RabbitMQ:
Homepage: http://www.rabbitmq.com/ Ruby Clients: - amqp: http://rubygems.org/gems/amqp (EventMachine based) - bunny: http://rubygems.org/gems/bunny Protocol Information: http://www.amqp.org/

MQ:
Homepage: http://www.zeromq.org/ Ruby Client: http://www.zeromq.org/bindings:ruby Manual: http://zguide.zeromq.org/page:all

Message Queues

* Beanstalkd * MQ (ZeroMQ) * RabbitMQ (AMQP)

Beanstalkd
Beanstalkd is a Job Queue Server. It listens on a TCP socket, where it receives jobs from other processes. These jobs stay in beanstalkd's job queue until a worker process reserves, executes them and finally deletes them. This allows the request processing process to be released, immediately after the job was put in the queue. Web Server HTTP request put(do-something) HTTP response reserve job beanstalkd Worker(s)

delete job
2

RabbitMQ (AMQP)
RabbitMQ is a Message Queue Broker, written in Erlang. It utilizes the openly specified AMQP (Advanced Message Queuing Protocol). It features 3 communication schemes: fanout (1-n), topic (1-n, aka pub/sub) and direct (1-1). Some use cases are: background processing by one or more workers, propagating events / updates from & to multiple nodes...

Use Case: fanout from web application

Web Server HTTP Request message HTTP Response
(published to a fanout exchange) all subscribed nodes receive the message

Database

RabbitMQ message broker

XMPP Gateway

... ...
3

MQ (ZeroMQ)
ZeroMQ is a message queue system, similar to AMQP, though a lot smaller. It doesn't rely on a separate broker process. Instead, one of the system's processes takes that role. Like AMQP, there are multiple communication schemes: request/response (1-1), pub/sub (1-n), ventilator/sink (1-x-1). The ventilator pattern works pretty much the same as beanstalkd (i.e. there are one or more workers, of which the first to shout here gets it), but also has a sink node, where workers push their results.

request/response
Client Response Request Server

publish/subscribe
Publisher

ventilator/sink
Ventilator

Worker

Worker

Worker

Subscriber Subscriber Subscriber

Sink

Resources
Beanstalkd:
Homepage: http://kr.github.com/beanstalkd/ Ruby Client: http://beanstalk.rubyforge.org/ Protocol Specification: https://github.com/kr/beanstalkd/blob/v1.1/doc/protocol.txt

RabbitMQ:
Homepage: http://www.rabbitmq.com/ Ruby Clients: - amqp: http://rubygems.org/gems/amqp (EventMachine based) - bunny: http://rubygems.org/gems/bunny Protocol Information: http://www.amqp.org/

MQ:
Homepage: http://www.zeromq.org/ Ruby Client: http://www.zeromq.org/bindings:ruby Manual: http://zguide.zeromq.org/page:all