How to Install Logstash

Logstash is a powerful, open-source data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and sends it to your preferred destinationwhether thats Elasticsearch, a database, or a data lake. As part of the Elastic Stack (formerly ELK Stack), Logstash plays a critical role in centralized logging, real-time analytics, and infrastructure monitoring. Its flexibility in handling structured and unstructured data makes it indispensable for DevOps teams, security analysts, and application developers aiming to gain actionable insights from vast volumes of log data.

Installing Logstash correctly is the foundation of any successful logging and monitoring architecture. A misconfigured or improperly installed Logstash instance can lead to data loss, performance bottlenecks, or security vulnerabilities. This comprehensive guide walks you through every step of the installation processfrom system requirements and dependency management to configuration validation and post-installation testing. Whether youre deploying on a Linux server, a cloud instance, or a containerized environment, this tutorial ensures you install Logstash securely, efficiently, and at scale.

Step-by-Step Guide

Prerequisites and System Requirements

Before installing Logstash, ensure your system meets the minimum hardware and software requirements. Logstash is a Java-based application and requires a compatible Java Runtime Environment (JRE). The latest versions of Logstash require Java 11 or Java 17. Java 8 is no longer supported as of Logstash 8.0.

Hardware Recommendations:

• Minimum: 2 CPU cores, 4 GB RAM
• Recommended for production: 4+ CPU cores, 8+ GB RAM
• Storage: SSD preferred; ensure at least 20 GB of free disk space for logs and temporary files

Software Requirements:

• Operating System: Linux (Ubuntu 20.04/22.04, CentOS 7/8, RHEL 8/9), macOS (for development), or Windows Server 2016+
• Java 11 or Java 17 (OpenJDK or Oracle JDK)
• Root or sudo access for installation and service management
• Internet access to download packages (or access to an internal repository)

Verify your Java installation by running:

java -version

If Java is not installed, proceed with installing OpenJDK 17:

On Ubuntu/Debian:

sudo apt update
sudo apt install openjdk-17-jdk -y

On CentOS/RHEL:

sudo yum install java-17-openjdk-devel -y
Or for newer versions using dnf:
sudo dnf install java-17-openjdk-devel -y

After installation, confirm the Java path:

which java

Typical output: /usr/bin/java

Installing Logstash on Linux (Ubuntu/Debian)

The most reliable method to install Logstash on Ubuntu or Debian is via the official Elastic APT repository. This ensures automatic updates and dependency resolution.

Step 1: Import the Elastic GPG Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Step 2: Add the Elastic APT Repository

echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

Step 3: Update Package Index

sudo apt update

Step 4: Install Logstash

sudo apt install logstash -y

Step 5: Verify Installation

After installation, check the Logstash version:

logstash --version

You should see output similar to:

logstash 8.12.0

Installing Logstash on Linux (CentOS/RHEL)

For Red Hat-based systems, use the YUM or DNF repository method.

Step 1: Import the Elastic GPG Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Step 2: Create the Elastic Repository File

sudo tee /etc/yum.repos.d/elastic-8.x.repo [elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Step 3: Install Logstash

sudo yum install logstash -y
Or on newer systems with dnf:
sudo dnf install logstash -y

Step 4: Verify Installation

logstash --version

Installing Logstash on macOS

For development or testing on macOS, Homebrew is the easiest method.

Step 1: Install Homebrew (if not already installed)

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2: Install Logstash via Homebrew

brew tap elastic/tap
brew install elastic/tap/logstash

Step 3: Verify Installation

logstash --version

Note: macOS installations are not recommended for production use due to performance and stability limitations.

Installing Logstash on Windows

Logstash can be installed on Windows Server 2016 or later. It is distributed as a ZIP archive.

Step 1: Download Logstash

Visit https://www.elastic.co/downloads/logstash and download the Windows ZIP file.

Step 2: Extract the Archive

Extract the ZIP file to a directory such as C:\logstash.

Step 3: Set Environment Variables

Set the JAVA_HOME environment variable to point to your Java installation (e.g., C:\Program Files\Java\jdk-17).

Add C:\logstash\bin to your systems PATH variable.

Step 4: Test Installation

Open a Command Prompt as Administrator and run:

logstash --version

Step 5: Run Logstash for Testing

cd C:\logstash\bin
logstash -e "input { stdin { } } output { stdout { } }"

This will start Logstash in interactive mode, accepting input from the terminal and printing output to the console.

Configuring Logstash: Basic Pipeline Setup

Logstash operates using pipelines defined in configuration files. By default, the main configuration file is located at:

• Linux: /etc/logstash/logstash.yml (global settings)
• Linux: /etc/logstash/conf.d/ (pipeline configurations)
• Windows: C:\logstash\config\

Create your first pipeline configuration file:

sudo nano /etc/logstash/conf.d/01-simple.conf

Add the following basic configuration:

input {
stdin { }
}
output {
stdout { codec => rubydebug }
}

This configuration tells Logstash to read input from the terminal and output structured data to the console in a readable format.

Test the configuration:

sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/01-simple.conf

If the configuration is valid, youll see:

Configuration OK

Start Logstash as a service:

sudo systemctl start logstash
sudo systemctl enable logstash

Check the service status:

sudo systemctl status logstash

View logs for errors:

sudo journalctl -u logstash -f

Installing Logstash with Docker

Containerized deployments are increasingly popular for scalability and portability. The official Logstash Docker image is maintained by Elastic.

Step 1: Pull the Logstash Docker Image

docker pull docker.elastic.co/logstash/logstash:8.12.0

Step 2: Create a Configuration Directory

mkdir -p ~/logstash/config
mkdir -p ~/logstash/pipelines

Step 3: Create a Pipeline Configuration

cat > ~/logstash/pipelines/logstash.conf input {
stdin { }
}
output {
stdout { codec => rubydebug }
}
EOF

Step 4: Run the Container

docker run -it --rm \
-v ~/logstash/pipelines:/usr/share/logstash/pipeline \
-v ~/logstash/config:/usr/share/logstash/config \
docker.elastic.co/logstash/logstash:8.12.0

This command mounts your local configuration into the container and starts Logstash interactively.

To run in detached mode:

docker run -d \
--name logstash \
-p 5044:5044 \
-v ~/logstash/pipelines:/usr/share/logstash/pipeline \
-v ~/logstash/config:/usr/share/logstash/config \
docker.elastic.co/logstash/logstash:8.12.0

Best Practices

Use Separate Pipeline Files

As your logging infrastructure grows, avoid monolithic configuration files. Instead, organize your pipelines into separate files in the /etc/logstash/conf.d/ directory. Name them numerically (e.g., 01-input.conf, 02-filter.conf, 03-output.conf) to control the order of execution. Logstash loads these files in alphabetical order.

Validate Configurations Before Restarting

Always test your configuration before restarting the Logstash service. Use the -t flag to perform a syntax check:

sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/your-pipeline.conf

This prevents service downtime due to malformed configuration files.

Optimize JVM Settings

Logstash runs on the Java Virtual Machine. For production deployments, tune the JVM heap size to avoid out-of-memory errors. Edit the jvm.options file located at /etc/logstash/jvm.options.

For a server with 8 GB RAM, set:

-Xms2g
-Xmx2g

Never set the heap size higher than 50% of your systems available RAM. Excessive heap allocation can lead to long garbage collection pauses.

Enable Logging and Monitoring

Logstash generates its own logs at /var/log/logstash/. Ensure log rotation is configured to prevent disk exhaustion. The default logrotate configuration should suffice, but verify:

sudo ls -la /etc/logrotate.d/logstash

Additionally, enable Logstashs built-in monitoring by adding to /etc/logstash/logstash.yml:

monitoring.enabled: true
monitoring.elasticsearch.hosts: ["http://localhost:9200"]

This allows you to view metrics in Kibana under the Monitoring section.

Secure Communication

If Logstash communicates with Elasticsearch or other services over HTTP, enforce TLS encryption. Generate certificates using OpenSSL or a certificate authority, then configure your output plugin:

output {
elasticsearch {
hosts => ["https://elasticsearch.example.com:9200"]
ssl => true
cacert => "/etc/logstash/certs/ca.crt"
user => "logstash_writer"
password => "your_secure_password"
}
}

Never use plaintext credentials in configuration files. Use Elasticsearchs built-in keystore to store sensitive data:

sudo /usr/share/logstash/bin/logstash-keystore create
sudo /usr/share/logstash/bin/logstash-keystore add ELASTIC_PASSWORD

Then reference it in your config:

password => "${ELASTIC_PASSWORD}"

Resource Management and Scaling

Logstash is single-threaded by default for each pipeline. To handle high throughput, increase the number of pipeline workers:

pipeline.workers: 4

Add this line to /etc/logstash/logstash.yml. The optimal number is typically equal to the number of CPU cores.

For very high-volume environments, consider deploying multiple Logstash instances behind a load balancer or using Logstashs built-in load balancing with Beats.

Use Filters Efficiently

Filters like grok, mutate, and date are powerful but can be CPU-intensive. Avoid over-filtering. Only parse fields you need. Use conditional statements to apply filters only when necessary:

if [type] == "apache_access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}

Test your grok patterns using online tools like Grok Debugger before deploying.

Tools and Resources

Official Documentation

Always refer to the official Elastic documentation for the most accurate and up-to-date information:

• Logstash Documentation
• Input Plugins
• Filter Plugins
• Output Plugins

Community and Support

Engage with the Elastic community for troubleshooting and best practices:

• Elastic Discuss Forum
• GitHub Repository
• Official Webinars and Training

Configuration Examples and Templates

Use pre-built configuration templates from trusted sources:

• Elastic Examples GitHub Real-world configurations for Nginx, Apache, Syslog, Windows Event Logs
• Official Logstash Plugins Repository
• Beats Input Examples For integration with Filebeat, Winlogbeat, etc.

Monitoring and Diagnostic Tools

Use these tools to observe Logstash performance:

• Kibana Monitoring Dashboard Visualize pipeline throughput, JVM usage, and error rates
• Logstash Metrics API Access real-time stats via curl http://localhost:9600/_node/stats
• htop / top Monitor CPU and memory usage
• netstat or ss Verify ports are listening (default: 5044 for Beats, 9600 for API)

Third-Party Tools

These utilities enhance Logstash workflows:

• Logstash-Runner A lightweight wrapper for managing multiple instances
• Ansible Roles Automate Logstash deployment across servers
• Terraform Modules Provision Logstash on AWS, GCP, or Azure
• Fluentd vs. Logstash Comparison Tools Evaluate alternatives for your use case

Real Examples

Example 1: Ingesting Nginx Access Logs

Lets say youre running Nginx on a web server and want to parse access logs into structured fields for analysis in Elasticsearch.

Step 1: Configure Filebeat to Ship Logs

On the Nginx server, install Filebeat and configure it to read /var/log/nginx/access.log:

filebeat.inputs:
- type: filestream
enabled: true
paths:
- /var/log/nginx/access.log
output.logstash:
hosts: ["logstash-server:5044"]

Step 2: Configure Logstash Pipeline

Create /etc/logstash/conf.d/10-nginx.conf:

input {
beats {
port => 5044
}
}
filter {
if [agent][type] == "filebeat" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
remove_field => [ "message", "timestamp" ]
}
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "nginx-access-%{+YYYY.MM.dd}"
user => "logstash_writer"
password => "${ELASTIC_PASSWORD}"
}
}

This pipeline ingests logs, parses them using the built-in COMBINEDAPACHELOG pattern, adds geolocation data, converts timestamps, and outputs to Elasticsearch with a daily index.

Example 2: Centralized Syslog Collection

Collect syslog messages from multiple Linux servers and forward them to Elasticsearch.

Step 1: Configure Remote Syslog on Clients

Edit /etc/rsyslog.conf on each server:

*.* @@logstash-server:5140

Restart rsyslog:

sudo systemctl restart rsyslog

Step 2: Configure Logstash to Receive Syslog

Create /etc/logstash/conf.d/20-syslog.conf:

input {
syslog {
port => 5140
type => "syslog"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "syslog-%{+YYYY.MM.dd}"
}
}

This setup allows you to aggregate logs from hundreds of servers into a single searchable index.

Example 3: Processing Windows Event Logs

Use Winlogbeat to ship Windows Event Logs to Logstash.

On Windows, install Winlogbeat and configure it to send to Logstash:

winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
output.logstash:
hosts: ["logstash.example.com:5044"]

On Logstash, create /etc/logstash/conf.d/30-windows.conf:

input {
beats {
port => 5044
}
}
filter {
if [winlog][event_id] {
mutate {
add_tag => [ "windows_event" ]
}
ruby {
code => "
event.set('[event][severity]', case event.get('[winlog][event_data][Level]')
when '0' then 'Emergency'
when '1' then 'Alert'
when '2' then 'Critical'
when '3' then 'Error'
when '4' then 'Warning'
when '5' then 'Notice'
when '6' then 'Informational'
when '7' then 'Debug'
else 'Unknown'
end)
"
}
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "windows-events-%{+YYYY.MM.dd}"
}
}

This transforms raw Windows event IDs into human-readable severity levels and enriches them for security monitoring.

FAQs

What is the latest version of Logstash?

As of 2024, the latest stable version is Logstash 8.12. Always check the official Elastic downloads page for the most recent release. Avoid using outdated versions due to security vulnerabilities and missing features.

Can I run Logstash without Elasticsearch?

Yes. Logstash can output to many destinations including files, databases (MySQL, PostgreSQL), Kafka, Redis, Amazon S3, or even stdout. Elasticsearch is commonly used but not required.

How much memory does Logstash use?

By default, Logstash allocates 1 GB of heap memory. In production, allocate 24 GB depending on throughput. Monitor usage via Kibana or the metrics API to avoid crashes.

Why is Logstash slow?

Common causes include:

• Overly complex grok patterns
• Insufficient CPU or memory
• Network latency to output destinations
• Large unbuffered input queues

Use the --debug flag to see processing times and optimize filters.

How do I upgrade Logstash?

For package installations:

sudo apt update && sudo apt upgrade logstash
or
sudo yum update logstash

Always back up your configuration files before upgrading. Test the new version in a staging environment first.

Does Logstash support JSON input?

Yes. Use the json codec or json filter:

input {
file {
path => "/var/log/app/events.json"
codec => json
}
}

This automatically parses each line as a JSON object and converts it into Logstash events.

Is Logstash secure?

Logstash supports TLS, authentication, and keystore-based secrets. Always disable the HTTP API (port 9600) in production unless secured with authentication and firewall rules.

Can I run multiple Logstash instances on one server?

Yes. Configure each instance with unique ports, pipeline IDs, and data directories. Use systemd service templates or Docker containers for isolation.

Whats the difference between Logstash and Fluentd?

Both are log shippers, but Logstash has richer filter plugins and deeper integration with the Elastic Stack. Fluentd is lighter and written in Ruby/C, often preferred in Kubernetes environments. Choose based on your ecosystem and performance needs.

How do I troubleshoot Logstash startup failures?

Check the logs:

sudo journalctl -u logstash -n 50 --no-pager

Common errors include:

• Invalid configuration syntax
• Missing Java version
• Port conflicts (e.g., 5044 already in use)
• Permission issues on config or log directories

Conclusion

Installing Logstash is more than a technical taskits the foundation of a scalable, secure, and efficient log management strategy. Whether youre ingesting application logs, system metrics, or security events, a properly installed and configured Logstash instance ensures data integrity, performance, and reliability. This guide has walked you through every critical step: from selecting the right Java version and configuring pipelines to securing communications and optimizing for production workloads.

Remember: Logstash thrives on clean, modular configurations and continuous monitoring. Avoid the temptation to overload pipelines with unnecessary filters. Test each change. Monitor performance. Secure your secrets. Leverage the community and official documentation to stay ahead of evolving best practices.

As data volumes grow and observability becomes a business imperative, Logstash remains one of the most powerful tools in the DevOps toolkit. By following the methods outlined here, youre not just installing softwareyoure building a resilient, intelligent logging infrastructure that empowers your team to detect issues before they impact users, uncover trends in real time, and make data-driven decisions with confidence.