Logs Monitoring in Microservices using ELK - Architecture

The ELK Stack is an amassment of three open-source products — Elasticsearch, Logstash, and Kibana. ELK stack provides centralized logging in order to identify quandaries with servers or applications. It sanctions us to search all the logs in a single place. It withal avails to find issues in multiple servers by connecting logs during a concrete time frame.

The ELK stack provides a simple yet robust log analysis solution for our developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring.

Modern log management and analysis solutions include the following capabilities:
  • Aggregation – the facility to accumulate and ship logs from multiple data sources.
  • Processing – the faculty to transform log messages into consequential data for more facile analysis.
  • Storage – the faculty to store data for elongated time periods to sanction for monitoring, trend analysis, and security use cases.
  • Analysis – the faculty to dissect the data by querying it and engendering visualizations and dashboards on top of it.

ELK Stack Architecture

For a small-sized development environment, the classic architecture will look as follows:


Elasticsearch: 

The open-source, distributed, RESTful, JSON-predicated search engine. Facile to utilize, scalable, and flexible, it earned hyper-popularity among users and a company composed around it, we know, for search.
The transformed data from Logstash is Store, Search, and indexed.

Logstash: 

Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to our favorite "stash."
Collect logs and events data. It even parses and transforms data.

Kibana: 

Kibana is a free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch. 
Kibana uses Elasticsearch DB to Explore, Visualize, and Share


However, one more component is needed for Data collection called Beats. This led Elastic to rename ELK as the Elastic Stack.


Beats:

Beats is a free and open platform for single-purport data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.


While dealing with prodigiously and sizably voluminous quantities of data, we may need Kafka or RabbitMQ for buffering and resilience.


Apache Kafka:

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Substructure, inscribed in Scala and Java. The project aims to provide a cumulated, high-throughput, low-latency platform for handling real-time data aliments.

Redis:

Redis is an in-recollection data structure store, utilized as a distributed, in-memory key–value database, cache and message broker, with optional durability.


More...

Comments

  1. This comment has been removed by the author.

    ReplyDelete
  2. The information you've provided is quite useful about china export data. It's incredibly instructional because it provides some of the most useful information. Thank you for sharing that.

    ReplyDelete

Post a Comment