Notes on Hot Technologies

local

Open APIs

http://github.com/toddmotto/public-apis

Interesting ones are:

Celery

http://www.celeryproject.org/

Celery is an asynchronous task queue/job queue based on distributed message passing.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet,or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery is used in production systems to process millions of tasks a day.

https://www.fullstackpython.com/task-queues.html

Alternatives to Celery

  • The RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. RQ is backed by Redis and is designed to have a low barrier to entry.
  • Taskmaster is a lightweight simple distributed queue for handling large volumes of one-off tasks.
  • Dramatiq is a fast and reliable alternative to Celery. It supports RabbitMQ and Redis as message brokers.

Hosted message and task queue services:

  • Iron.io - Hosted Celery platform to work with Amazon or Heroku, etc.
  • Amazon Simple Queue Service (SQS) is a set of five APIs for creating, sending, receiving, modifying and deleting messages. (Internally uses celery etc)
  • CloudAMQP is hosted RabbitMQ service

Compare RabbitMQ, Kafka, AQS

RabbitMQ is msg broker but AQS is hosted service. How to compare these two ? See http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/

See https://github.com/softwaremill/mqperf

Dictionary of Queuing systems Available

See http://queues.io/

Example Projects

Tips

  • Database is not a queue ... though for small application, it is OK. One should use message queue.
Interactive Graph
Table Of Contents
Notes on Hot Technologies
Open APIs
Celery
Alternatives to Celery
Hosted message and task queue services:
Compare RabbitMQ, Kafka, AQS
Dictionary of Queuing systems Available
Example Projects
Tips