python - Can i use Tornado+ Celery+ RabbitMQ + Redis?

Question

For real time streaming of data on the web, I'm planning to use the Redis as my Cache data layer, where data is instantaneous. Celery is the Queue manager and RabbitMQ is the broker which queues from Redis and gets to the Tornado layer. This layer then via websockets streams to the frontend.

I have never found the Redis + RabbitMQ Combination on-line. Can someone guide with a reliable solution for the same. The problem being is such integration possible and advisable?

Answer 1

I'm using Tornado and RabbitMQ (without Celery) together very successfully right now. I also did some prototyping with Redis plugged into the system as a db, as well. Your best chance at success is to find asynchronous libraries that can plug into Tornado's ioloop. For example there's tornado-redis, pika (for RabbitMQ), and tornado-celery. I should add that I've never used tornado-celery, so I'm not sure of its quality.

From a purely architectural standpoint, I don't think there's anything wrong with using both Redis and RabbitMQ together. While both can be used as message brokers, RabbitMQ isn't really appropriate for use as a cache, while Redis is well-suited for that use-case. If you were using memcached instead of Redis, it wouldn't seems very unusual at all, right?

Also (and perhaps this is the best argument I can make to suggest Redis + RabbitMQ can work), Pinterest apparently use (or at least used) both to run their site: http://blog.gopivotal.com/pivotal/case-studies-2/using-redis-at-pinterest-for-billions-of-relationships.

I found another source that says they eventually replaced Celery/RabbitMQ with another solution, but clearly it was working for them for a while.

edit: Found a Quora link from 2011 that says they were actually running Tornado as well!