While working on a full log integration solution, we have been asked to improve an already capable ELK stack with a buffer solution. Being a large production environment dealing with a little more than 10 millions messages per day, the idea was to prevent messages overflow while still being able to target maximum completeness and the highest availability rate possible.
We ended up setting up Redis in front of the stack, acting as a message buffer thanks to its “pubsub” abilities. And it does the work so well Redis is now drafted as a key component of our next iterations of products.