mehul-m-prajapati/gsoc-queue-interface

GSoC 2018: Queue interface

General Info

• Name: Mehul Prajapati
• Github: mehul-m-prajapati
• Email: mehulprajapati2802@gmail.com
• Organization: syslog-ng
• Mentor: Kókai Péter
• Development Branch Tag: gsoc-2018

Project Abstract

The syslog-ng has a queue for destinations, that has a current implementation as disk queue. The aim of this project is to develop a Redis based queue interface which could be an alternative to the disk queue.
Redis is an open source in-memory data structure store, used as a database, cache and message broker.

Blog Posts:

• Medium

Design Diagram

 --------                    ----------------                    -------------
| Source | ================ | syslog-ng core | ================ | Destination | 
 --------                    ----------------          ||        -------------
                                                       ||
                                                       ||
                                                       ||
                                                 ---------------      
                                                |  redis queue  |
                                                |    plugin     |
                                                 ---------------

Commits / Tasks

• Develop dummy redis queue plugin
• Add push, pop, ack, rewind backlog log queue methods
• Add redis server connection methods instead of dummy queue
• Develop redis read, write and delete methods
• Add following configurable parameters for redis queue plugin

• host address
• port
• auth
• key name prefix
• conn-timeout

• Handle backlog messages
• Develop unit test cases
• Implement rqtool to print-out data from redis server (Useful for debugging)
• Develop Redis connect, disconnect, reconnect, isconnected and send command methods
• Add Redis SET for listing only redis queues in rqtool

Pull Request (This link might get changed in future because of code review modifications)

• balabit/syslog-ng/#2231

To Do (In Future)

• Develop functional test cases in python
• Implement reliable and non-reliable redis queue
• Limit number of messages that can be stored in redis (i.e. using overflow queue)
• Redis as a cluster
• Merge rqtool and dqtool functionalities and develop qtool