Logs Monitoring in Microservices using ELK - Architecture

The ELK Stack is an amassment of three open-source products — Elasticsearch, Logstash, and Kibana. ELK stack provides centralized logging in order to identify quandaries with servers or applications. It sanctions us to search all the logs in a single place. It withal avails to find issues in multiple servers by connecting logs during a concrete time frame.

The ELK stack provides a simple yet robust log analysis solution for our developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring.

Modern log management and analysis solutions include the following capabilities:
  • Aggregation – the facility to accumulate and ship logs from multiple data sources.
  • Processing – the faculty to transform log messages into consequential data for more facile analysis.
  • Storage – the faculty to store data for elongated time periods to sanction for monitoring, trend analysis, and security use cases.
  • Analysis – the faculty to dissect the data by querying it and engendering visualizations and dashboards on top of it.

ELK Stack Architecture

For a small-sized development environment, the classic architecture will look as follows:


The open-source, distributed, RESTful, JSON-predicated search engine. Facile to utilize, scalable, and flexible, it earned hyper-popularity among users and a company composed around it, we know, for search.
The transformed data from Logstash is Store, Search, and indexed.


Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to our favorite "stash."
Collect logs and events data. It even parses and transforms data.


Kibana is a free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch. 
Kibana uses Elasticsearch DB to Explore, Visualize, and Share

However, one more component is needed for Data collection called Beats. This led Elastic to rename ELK as the Elastic Stack.


Beats is a free and open platform for single-purport data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

While dealing with prodigiously and sizably voluminous quantities of data, we may need Kafka or RabbitMQ for buffering and resilience.

Apache Kafka:

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Substructure, inscribed in Scala and Java. The project aims to provide a cumulated, high-throughput, low-latency platform for handling real-time data aliments.


Redis is an in-recollection data structure store, utilized as a distributed, in-memory key–value database, cache and message broker, with optional durability.


Popular posts from this blog

Spring boot video streaming example-HTML5

Learn Java 8 streams with an example - print odd/even numbers from Array and List

Spring Boot + Mockito simple application with 100% code coverage

Spring Boot + OpenCSV Export Data to CSV Example

Custom Exception Handling in Quarkus REST API

DataTable-Pagination example with Spring boot, jQuery and ajax

Registration and Login with Spring Boot + Spring Security + Thymeleaf

Node JS mini projects with source code - free download

Spring boot web project free download:User Registration System

Spring Boot + Apache Commons Export Data to CSV Example