SOP for Installation of ELK Stack with Filebeat on AWS Servers

Standard Operating Procedure (SOP): Installation of ELK Stack with Filebeat on AWS Servers

1. Purpose

This SOP outlines the step-by-step process for installing and configuring the ELK (Elasticsearch, Logstash, and Kibana) stack with Filebeat on AWS servers. The document includes prerequisites, system requirements, cluster setup, and AWS-specific configurations.

2. Prerequisites

• AWS Account: Access to an active AWS account with appropriate IAM permissions.
• Networking:
  • VPC created for deployment.
  • Subnets for public/private resources.
  • Security groups with the required ports opened (refer to the Ports section).
• System Access:
  • SSH key pair for accessing EC2 instances.
  • Admin/root privileges on servers.
• Software Tools:
  • AWS CLI installed and configured locally.
  • Terraform/CloudFormation (optional for automation).
  • Remote terminal such as PuTTY or a compatible SSH client.

3. System Requirements

Compute Requirements

Storage Requirements

Ports to Open

4. Cluster Creation

AWS Perspective

1. VPC Setup:

   • Create a VPC and enable DNS hostnames.
   • Set up two subnets (public and private) for different components.
   • Create an Internet Gateway (IGW) and attach it to the VPC.

2. Security Groups:

   • Allow inbound traffic for required ports in the security group associated with EC2 instances.
   • Restrict access to trusted IP ranges for Kibana and Elasticsearch.

3. EC2 Instances:

   • Launch EC2 instances for Elasticsearch, Logstash, Kibana, and Filebeat.
   • Use Amazon Linux 2 or Ubuntu 20.04 for compatibility.
   • Allocate Elastic IPs for external access if required.

4. IAM Role:

   • Attach an IAM role with S3 and CloudWatch permissions for backup and monitoring.

Landing Zone Perspective

1. Use AWS Landing Zone (if applicable) to standardize account setup and ensure governance.
2. Configure logging and monitoring through AWS CloudTrail and AWS CloudWatch.
3. Ensure tagging standards are applied for resource identification.

5. Installation Steps

Elasticsearch

1. Install Elasticsearch:

sudo apt update
   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   sudo apt install apt-transport-https
   echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
   sudo apt update && sudo apt install elasticsearch

1. Configure Elasticsearch:

   • Update /etc/elasticsearch/elasticsearch.yml with the following:

    network.host: 0.0.0.0
    cluster.name: elk-cluster
   

• Restart the service:
  

   sudo systemctl enable elasticsearch
   sudo systemctl start elasticsearch
  

Logstash

1. Install Logstash:

sudo apt install logstash

1. Configure Logstash:

   • Create a configuration file at /etc/logstash/conf.d/logstash.conf:

    input {
        beats {
            port => 5044
        }
    }
   
    output {
        elasticsearch {
            hosts => ["http://:9200"]
            index => "logs-%{+YYYY.MM.dd}"
        }
    }
   

• Start the service:
  

   sudo systemctl enable logstash
   sudo systemctl start logstash
  

Kibana

1. Install Kibana:

sudo apt install kibana

1. Configure Kibana:

   • Update /etc/kibana/kibana.yml:

    server.host: "0.0.0.0"
    elasticsearch.hosts: ["http://:9200"]
   

• Restart the service:
  

   sudo systemctl enable kibana
   sudo systemctl start kibana
  

Filebeat

1. Install Filebeat:

sudo apt install filebeat

1. Configure Filebeat:

   • Update /etc/filebeat/filebeat.yml:

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/*.log
   
    output.logstash:
      hosts: [":5044"]
   

• Start Filebeat:
  

   sudo systemctl enable filebeat
   sudo systemctl start filebeat
  

6. Post-Installation Verification

1. Elasticsearch:

   • Verify connectivity:

    curl http://:9200
   

2. Logstash:

   • Check for log ingestion in Elasticsearch.

3. Kibana:

   • Access the Kibana UI at http://:5601 and configure an index pattern.

4. Filebeat:

   • Confirm logs are being sent to Logstash and indexed in Elasticsearch.

7. Maintenance and Monitoring

1. Backup:
   • Configure snapshot backups to an S3 bucket.
2. Monitoring:
   • Use Elastic Stack’s monitoring features or integrate with AWS CloudWatch.
3. Scaling:
   • Use AWS Auto Scaling Groups for horizontal scaling of Elasticsearch nodes.

8. Troubleshooting

• Logs:
  • Check service logs for Elasticsearch (/var/log/elasticsearch), Logstash (/var/log/logstash), and Kibana (/var/log/kibana).
• Network Issues:
  • Verify VPC routing tables and security group configurations.
• Resource Bottlenecks:
  • Monitor instance metrics and upgrade instance types or increase storage as needed.

This SOP ensures a standardized deployment of the ELK stack with Filebeat on AWS, facilitating efficient log management and monitoring.