The ELK Stack offers multiple installation methods and is compatible with various operating systems and environments. You can set up ELK locally, in the cloud, with Docker, or through configuration management tools such as Ansible, Puppet, and Chef. Additionally, the stack can be installed using .tar or .zip packages, as well as directly from repositories.

So, In this article we will Setup a Resilient ELK cluster for high availability. Alos We'll try to build a complete solution for our prod application. Nodes and shards makes Elasticsearch distributed and scalable.

Table of Contents

1 Introduction to the ELK Stack

  • Overview of Elasticsearch, Logstash, and Kibana (ELK)
  • Beats: Filebeat, Metricbeat, etc.
  • Use Cases and Benefits

2 Pre-requisites

  • System Requirements
  • Dependencies
  • Security Considerations

3 Installation of ELK Stack

  • Elasticsearch
    • Single Node Installation
    • Multi-node Cluster Setup
  • Logstash
  • Kibana

4 Installation of Beats

  • Filebeat
  • Metricbeat
  • Other Beats (Heartbeat, Packetbeat, Auditbeat)

5 Configuration

  • Elasticsearch Configuration
    • Setting Roles for Nodes (Master, Data, Coordinating, etc.)
    • Enabling Security (TLS, Authentication, and Authorization)
    • Configuring Elasticsearch YML File
  • Logstash Configuration
    • Input, Filter, and Output Pipelines
    • Secure Pipeline Configuration
  • Kibana Configuration
    • Enabling TLS and Authentication
    • Configuring Default Spaces and Dashboards
  • Beats Configuration
    • Configuring Filebeat for Apache Logs
    • Secure Beats Communication

6 Cluster Creation and Management

  • Multi-node Cluster Setup
  • Managing Cluster Roles
  • Monitoring and Troubleshooting

7 Using ELK Stack for Apache Log Monitoring

  • Creating Pipelines in Logstash
  • Setting Up Filebeat for Apache Logs
  • Parsing and Analyzing Logs
  • Creating Dashboards in Kibana

8 Advanced Pipelines in Logstash

  • Pipeline Architecture
  • Conditional Logic and Data Routing
  • Using Ruby Filters for Advanced Transformations
  • Optimizing Pipelines for Performance

9 Security Hardening Tips

  • Securing Communication Between Nodes
  • Implementing Role-Based Access Control (RBAC)
  • Using Encryption for Data at Rest
  • Regular Security Audits and Patching

10 Kibana Dev Tools and Console

  • Introduction to Kibana Console
  • Writing and Testing Elasticsearch Queries
  • Managing Index Patterns
  • Examples of Advanced Queries

11 Creating Dashboards in Kibana

  • Designing Custom Dashboards
  • Using Visualizations
  • Examples: Apache Logs Monitoring Dashboard

12 Comprehensive Use Cases and Advanced Tips

  • Setting Up Monitoring and Alerts
  • Multi-Tenancy with Spaces
  • Integration with Third-party Tools
  • Handling High Data Volumes

13 Complete Reference Documentation

  • Elasticsearch Commands
  • Logstash Commands
  • Kibana Commands
  • Beats Commands

14 Appendix

  • Example Configuration Files
  • Troubleshooting Tips
  • Additional Resources

1. Introduction to the ELK Stack

Image description

The ELK Stack (Elasticsearch, Logstash, and Kibana) is a powerful suite for managing and analyzing logs and other structured/unstructured data. By integrating Beats, the stack supports lightweight data shippers for log and metric collection.

Elasticsearch

  • A distributed, RESTful search and analytics engine.
  • Stores, searches, and analyzes massive volumes of data quickly and in near real-time.

Logstash

  • A data processing pipeline that ingests data from multiple sources, transforms it, and sends it to Elasticsearch.

Kibana

  • A visualization tool that provides insights into data stored in Elasticsearch.
  • Enables users to create charts, graphs, and dashboards for monitoring and analytics.

Beats

  • Lightweight data shippers that send data from edge machines to Logstash or Elasticsearch.
  • Includes Filebeat for log files, Metricbeat for metrics, and others.

Use Cases

  • Log and event data analysis.
  • Real-time application monitoring.
  • Security and compliance reporting.

2. Pre-requisites

System Requirements

  • Minimum requirements for a single node:
    • CPU: 4 cores
    • RAM: 16GB
    • Disk Space: 50GB (varies based on retention needs)
  • For multi-node clusters:
    • Dedicated roles (Master, Data, and Ingest) recommended.
    • Nodes should reside on a private network.

Dependencies

  • Java: Elasticsearch and Logstash require Java 11 or later.
  • OpenSSL: For enabling TLS encryption.
  • Firewall Rules: Ensure ports 9200, 9300 (Elasticsearch), 5044 (Logstash), and 5601 (Kibana) are open as needed.

Security Considerations

  • Always enable TLS encryption for inter-node communication.
  • Use authentication and authorization mechanisms provided by Elasticsearch.
  • Restrict access to management tools such as Kibana using firewalls and authentication.

3. Installation of ELK Stack

Elasticsearch

Single Node Installation
  1. Download Elasticsearch: 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch--linux-x86_64.tar.gz
  1. Extract and Install: 
tar -xvf elasticsearch--linux-x86_64.tar.gz
   cd elasticsearch-

  1. Set JVM Options and Heap Size:

    • Edit config/jvm.options to allocate appropriate heap size.
    • Example: 
     -Xms8g
 -Xmx8g

  2. Run Elasticsearch:

./bin/elasticsearch
  1. Test Elasticsearch: Visit http://localhost:9200 in your browser or use: 
curl -X GET "localhost:9200/"
Multi-node Cluster Setup
  1. Prepare Each Node: Follow steps 1-4 above on each node.

  2. Configure Cluster Settings:

    • Edit elasticsearch.yml on each node: 
     cluster.name: my-cluster
 node.name: node-1
 network.host: 0.0.0.0
 discovery.seed_hosts: ["node1-ip", "node2-ip", "node3-ip"]
 cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

  3. Enable TLS:

    • Generate certificates using elasticsearch-certutil.
    • Configure elasticsearch.yml with TLS settings. Example TLS Configuration: 
xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.transport.ssl.verification_mode: certificate
   xpack.security.transport.ssl.key: /path/to/private.key
   xpack.security.transport.ssl.certificate: /path/to/certificate.crt
   xpack.security.transport.ssl.certificate_authorities: ["/path/to/ca.crt"]
  1. Start Cluster Nodes: Start each node individually and confirm cluster health: 
curl -X GET "http://node1-ip:9200/_cluster/health"

Logstash

  1. Download and Install Logstash: 
wget https://artifacts.elastic.co/downloads/logstash/logstash--linux-x86_64.tar.gz
   tar -xvf logstash--linux-x86_64.tar.gz
   cd logstash-

  1. Configure Input, Filter, and Output Pipelines:

    • Example Configuration (logstash.conf): 
     input {
   beats {
     port => 5044
   }
 }
 filter {
   grok {
     match => { "message" => "%{COMBINEDAPACHELOG}" }
   }
 }
 output {
   elasticsearch {
     hosts => ["localhost:9200"]
     index => "apache-logs"
   }
 }

  2. Run Logstash:

./bin/logstash -f logstash.conf

Kibana

  1. Download and Install Kibana: 
wget https://artifacts.elastic.co/downloads/kibana/kibana--linux-x86_64.tar.gz
   tar -xvf kibana--linux-x86_64.tar.gz
   cd kibana-

  1. Edit Configuration:

    • Update config/kibana.yml: 
     server.host: "0.0.0.0"
 elasticsearch.hosts: ["http://localhost:9200"]

  2. Run Kibana:

./bin/kibana
  1. Access Kibana: Visit http://:5601.

4. Installation of Beats

Filebeat Installation and Configuration

  1. Download Filebeat: 
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat--linux-x86_64.tar.gz
   tar -xvf filebeat--linux-x86_64.tar.gz
   cd filebeat-
  1. Edit Configuration File:

  • Open filebeat.yml and configure inputs:

     filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/apache2/access.log

  • Set Elasticsearch output:

     output.elasticsearch:
   hosts: ["localhost:9200"]
  1. Enable and Start Filebeat: 
./filebeat modules enable apache
   ./filebeat setup
   ./filebeat -e

Metricbeat Installation and Configuration

  1. Download Metricbeat: 
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat--linux-x86_64.tar.gz
   tar -xvf metricbeat--linux-x86_64.tar.gz
   cd metricbeat-
  1. Edit Configuration File:

  • Open metricbeat.yml and configure modules:

     metricbeat.modules:
 - module: system
   metricsets:
     - cpu
     - memory
     - network
   enabled: true
   period: 10s
   hosts: ["localhost:9200"]
  1. Enable and Start Metricbeat: 
./metricbeat modules enable system
   ./metricbeat setup
   ./metricbeat -e

Other Beats

  • Heartbeat: For uptime monitoring.
  • Packetbeat: For network packet analysis.
  • Auditbeat: For auditing user activity and processes.

Installation steps are similar to Filebeat and Metricbeat, with module-specific configurations.

6. Detailed Cluster Creation

Multi-node Cluster Configuration

  1. Node Roles:
  • Assign roles to nodes for better scalability and fault tolerance.
    • Master Nodes: Responsible for cluster-wide management.
    • Data Nodes: Store and handle data-related tasks like indexing and searching.
    • Ingest Nodes: Pre-process and transform data before indexing.
    • Coordinating Nodes: Handle client requests and distribute tasks across the cluster.
  1. Prepare Environment:
  • Ensure all nodes have the same version of Elasticsearch installed.
  • Use a private network for communication between nodes.
  1. Node-Specific Configuration:

  • Update elasticsearch.yml for each node based on its role:

     cluster.name: my-cluster
 node.name: 
 node.roles: [, ]
 network.host: 0.0.0.0
 discovery.seed_hosts: ["node1-ip", "node2-ip", "node3-ip"]
 cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
  1. Secure the Cluster:
  • Generate and deploy SSL/TLS certificates to each node.

  • Enable security features in elasticsearch.yml:

     xpack.security.enabled: true
 xpack.security.transport.ssl.enabled: true
 xpack.security.transport.ssl.verification_mode: certificate
  1. Start and Monitor the Cluster:
  • Start Elasticsearch on each node.

  • Verify cluster health:

     curl -X GET "http://:9200/_cluster/health?pretty"

  • Check node details:

     curl -X GET "http://:9200/_cat/nodes?v"

8. Advanced Pipelines in Logstash

Pipeline Architecture

  • Inputs: Define data sources (e.g., Beats, syslog).
  • Filters: Apply transformations, parsing, or enrichment.
  • Outputs: Define destinations (e.g., Elasticsearch, file).

Conditional Logic and Data Routing

  • Example: 
if [type] == "error" {
    output {
      elasticsearch {
        hosts => ["localhost:9200"]
        index => "error-logs"
      }
    }
  }

Using Ruby Filters for Advanced Transformations

  • Example: 
filter {
    ruby {
      code => "event.set('formatted_message', event.get('message').upcase)"
    }
  }

Optimizing Pipelines for Performance

  • Reduce memory footprint by minimizing intermediate steps.
  • Use bulk indexing for Elasticsearch outputs.

9. Security Hardening Tips

Securing Communication Between Nodes

  • Use mutual TLS authentication.
  • Ensure certificates are from a trusted CA.

Implementing Role-Based Access Control (RBAC)

  • Assign roles based on the principle of least privilege.
  • Example roles:
    • Admin: Full access
    • Viewer: Read-only access to dashboards

Using Encryption for Data at Rest

  • Enable disk encryption for Elasticsearch data directories.

Regular Security Audits and Patching

  • Periodically update ELK Stack components to the latest versions.
  • Review logs for unauthorized access attempts.

10. Kibana Dev Tools and Console

Introduction to Kibana Console

  • Accessible via the "Dev Tools" section in Kibana.
  • Provides an interactive interface to query Elasticsearch.

Writing and Testing Elasticsearch Queries

  • Example query to fetch all documents: 
GET /_search
  {
    "query": {
      "match_all": {}
    }
  }

Managing Index Patterns

  • Define patterns to group similar indices (e.g., logstash-*).
  • Use the "Index Management" tool in Kibana.

Examples of Advanced Queries

  • Aggregation query: 
GET /logs/_search
  {
    "aggs