Mongo-Redis-RabbitMQ-ELK-Stack

Node.js, MongoDB, Redis, Postgres, Kafka, ELK, and RabbitMQ Backend Project

Table of Contents

Introduction

This project is a sample, demo Node.js Backend project that demonstrates how to connect to MongoDB, Redis, PostgreSQL, Kafka, ELK-stack, and RabbitMQ. It demonstrates how these services can be used and interact with each other in a Node.js application.

Additionally, the project demonstrates how to use the ELK stack (Elasticsearch, Logstash, Kibana) for logging, as well as how to implement a round-robin load-balancing algorithm using Redis.

And we call this the NMRPKER-Stack! 🚀 (Just for fun, though)

Features

MongoDB

Redis-Mongo-Flow

RabbitMQ

Apache Kafka

Round-Robin Load Balancing Algorithm

ELK Stack

Here is a table of the features demonstrated in this project:

Feature Description
MongoDB Connect to MongoDB and perform CRUD operations.
Redis-Mongo-Flow Use Redis as a cache layer for MongoDB.
RabbitMQ Connect to RabbitMQ and publish and consume messages.
Apache Kafka Connect to Apache Kafka and produce and consume messages.
Round-Robin Load Balancing Algorithm Implement round-robin load balancing using Redis.
ELK Stack Use the ELK stack (Elasticsearch, Logstash, Kibana) for logging.
PostgreSQL Connect to PostgreSQL and perform CRUD operations.
Express.js Use Express.js as the web server framework.
Node.js Use Node.js as the runtime environment.
JavaScript Use JavaScript as the programming language.
NPM Use NPM as the package manager.
REST API Use REST API for communication.

Project Structure

The project has the following structure:

node-mongo-redis-project
├── index.js             # Main entry point for the project for testing the connections
├── config.js            # Configuration file for the project
├── package.json         # NPM package file
├── publish.js           # Script to publish a message to RabbitMQ
├── README.md
├── apache-kafka
│   └── kafkaService.js  # Core logic for Apache Kafka
├── round-robin
│   ├── index.js         # Core logic for round-robin load balancing & for testing the algorithm
│   └── config.js        # Configuration file for Redis
├── routes
│   └── test.js          # Sample routes for the project
├── redis-mongo-flow
│   ├── app.js           # Core logic of the Redis-Mongo flow
│   ├── config.js        # Configuration file for Redis and MongoDB         
│   ├── seed.js          # Script to populate MongoDB with sample data
│   └── test.js          # Script to test the Redis-Mongo flow
├── elk-stack
│   └── index.js         # Core logic for logging using the ELK stack & testing the stack
└── postgresql
    └── app.js           # Core logic for PostgreSQL
    └── config.js        # Configuration file for PostgreSQL

Getting Started

To get started, run the following commands:

  1. Start the MongoDB service: 
     brew services start mongodb-community
  2. Start the Redis service: 
     redis-server
  3. Start the RabbitMQ service: 
     brew services start rabbitmq
  4. Start the Apache Kafka service:
    • First, navigate to the Kafka directory: 
      cd kafka_2.13-3.8.0
    • Start the Zookeeper service: 
        bin/zookeeper-server-start.sh config/zookeeper.properties
    • Start the Kafka service: 
        bin/kafka-server-start.sh config/server.properties
  5. cd back into the project directory (fix the path if necessary): 
    cd node-mongo-redis-project
  6. Start the Node.js server: 
     node index.js
  7. (Optional) Test the RabbitMQ service by publishing a message (go to another terminal window): 
     node publish.js

    Verify that the message is received by the consumer by going to the terminal that is running the Node.js server (the one you started the index.js script). You should receive the following messages in the terminal:

     Server running on port 5000
 Visit http://localhost:5000/ to test the connection.
 Redis Connected
 MongoDB Connected
 Redis Test: Hello from Redis
 Kafka Producer and Consumer Connected
 Aggregation result: [
   { _id: 'Frank', totalOrderValue: 60 },
   { _id: 'David', totalOrderValue: 100 },
   { _id: 'Charlie', totalOrderValue: 35 },
   { _id: 'Grace', totalOrderValue: 75 },
   { _id: 'Alice', totalOrderValue: 50 },
   { _id: 'Bob', totalOrderValue: 80 },
   { _id: 'Ivy', totalOrderValue: 55 },
   { _id: 'Eve', totalOrderValue: 45 },
   { _id: 'Helen', totalOrderValue: 90 }
 ]
 Sent message to Kafka: Hello Kafka from Express!
 {
   topic: 'test-topic',
   partition: 0,
   value: 'Hello Kafka from Express!'
 }
 RabbitMQ Connected
 [*] Waiting for messages in task_queue. To exit press CTRL+C
 [x] Received 'This is a test message!'
  8. Visit http://localhost:5000/ to test the connections.
  9. (Optional) Test the Round-Robin Load Balancing Algorithm: 
     cd round-robin
 node index.js
  10. (Optional) Test the ELK Stack: 
    cd elk-stack
node index.js

Special Notes

Note: Before you get started, be sure to have the following installed on your machine by running the following commands (the following instructions are for MacOS):

  1. Install Homebrew: 
     /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  2. Install MongoDB: 
     brew tap mongodb/brew
 brew install mongodb-community
  3. Install Redis: 
     brew install redis
  4. Install RabbitMQ: 
     brew install rabbitmq
  5. Install Apache Kafka:
    • Go to the Apache Kafka website and download the latest version of Apache Kafka.
    • Currently, the most stable version is: Scala 2.13 version: kafka_2.13-3.8.0.tgz.
    • Extract the downloaded file to wherever you want to install Kafka on your machine.
    • Alternatively, you can use this direct link in your terminal to install Kafka: 
      wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
  6. Ensure Kafka is running:
    • First, navigate to the Kafka directory: 
      cd kafka_2.13-3.8.0
    • Start the Zookeeper service: 
        bin/zookeeper-server-start.sh config/zookeeper.properties
    • Start the Kafka service: 
        bin/kafka-server-start.sh config/server.properties
  7. Install Postgres: 
     brew install postgresql

    Start the Postgres service:

     brew services start postgresql

    Create a new database named davidnguyen (or any name you prefer - as long as you change the database credentials in the postgresql/config.js file):

     createdb davidnguyen

    Connect to the database:

     psql davidnguyen
  8. Install Node.js: 
     brew install node
  9. Install NPM: 
    brew install npm

    Then cd into the project directory and run:

    npm init -y
  10. Install all the required packages: 
    npm install
  11. For Windows, refer to the official installation guides for MongoDB, Redis, RabbitMQ, Apache Kafka, and PostgreSQL.

Live Deployment

This sample backend is also hosted on Render. You can check out the live demo here: Node.js Backend Demo.

Feel free to explore the backend’s routes and read the documentation at /docs.

Test Statuses

MongoDB PostgreSQL Redis RabbitMQ Apache Kafka ELK Stack REST API Round-Robin Load Balancing Algorithm

License

This project is licensed under the MIT License - see the LICENSE file for details.

Feel free to use this project for your own learning purposes or as a reference for your own projects!

Author

Thank you for checking out this demo backend project! 🚀 Feel free to reach out if you have any questions or feedback. 😊

This site is open source. Improve this page.