How to Setup Prometheus

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability in dynamic, cloud-native environments. Originally developed by SoundCloud and later donated to the Cloud Native Computing Foundation (CNCF), Prometheus has become the de facto standard for monitoring Kubernetes clusters, microservices, and modern infrastructure. Its powerful query language (PromQL), multi-dimensional data model, and pull-based architecture make it uniquely suited for capturing real-time metrics in distributed systems.

Unlike traditional monitoring tools that rely on push-based models or complex agent installations, Prometheus scrapes metrics over HTTP at regular intervals, making it lightweight, resilient, and easy to integrate. Whether you're monitoring a single server, a containerized application, or a large-scale microservices architecture, setting up Prometheus correctly ensures you gain deep visibility into system performance, resource utilization, and application health.

This guide provides a comprehensive, step-by-step walkthrough on how to setup Prometheus from scratch. Youll learn how to install it on various platforms, configure targets, write effective alerts, visualize data with Grafana, and follow industry best practices to ensure long-term stability and performance. By the end of this tutorial, youll have a fully functional Prometheus monitoring stack ready for production use.

Step-by-Step Guide

Prerequisites

Before installing Prometheus, ensure your environment meets the following minimum requirements:

• A Linux-based system (Ubuntu 20.04/22.04, CentOS 8+, or Debian 11 recommended)
• At least 2 CPU cores and 4 GB of RAM (8 GB+ recommended for production)
• Uninterrupted internet access for downloading binaries and dependencies
• Basic command-line proficiency
• A user account with sudo privileges

Its strongly advised to install Prometheus on a dedicated server or virtual machine to avoid resource contention with other services. If you're using a cloud provider (AWS, GCP, Azure), launch a standard instance such as t3.medium or n2-standard-2.

Step 1: Download Prometheus

The first step in setting up Prometheus is downloading the latest stable binary release from the official GitHub repository.

Open your terminal and navigate to a directory where you want to store the Prometheus files:

cd /opt

Use wget to download the latest version. As of this writing, Prometheus 2.51.x is the latest stable release. Check Prometheus Releases for the most current version:

wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz

Extract the archive:

tar xvfz prometheus-2.51.2.linux-amd64.tar.gz

Navigate into the extracted directory:

cd prometheus-2.51.2.linux-amd64

Youll see two key files: prometheus (the main binary) and prometheus.yml (the default configuration file). Keep these files accessibletheyll be moved to system directories in the next steps.

Step 2: Create Prometheus User and Directories

For security and system hygiene, avoid running Prometheus as root. Create a dedicated system user:

sudo useradd --no-create-home --shell /bin/false prometheus

Now create the necessary directories to store configuration files, metrics data, and binaries:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

Step 3: Move Binaries and Configuration Files

Move the Prometheus binary to the systems executable path:

sudo mv prometheus /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus

Move the Prometheus configuration file to the configuration directory:

sudo mv prometheus.yml /etc/prometheus/
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Also move the console templates and rules (optional but recommended for alerting and dashboards):

sudo mkdir /etc/prometheus/console_templates
sudo mkdir /etc/prometheus/consoles
sudo mv console_templates/* /etc/prometheus/console_templates/
sudo mv consoles/* /etc/prometheus/consoles/
sudo chown -R prometheus:prometheus /etc/prometheus/console_templates/
sudo chown -R prometheus:prometheus /etc/prometheus/consoles/

Step 4: Configure Prometheus

The configuration file, /etc/prometheus/prometheus.yml, defines what Prometheus monitors and how. Open it for editing:

sudo nano /etc/prometheus/prometheus.yml

By default, it contains a minimal configuration that only scrapes Prometheus itself. Replace or extend it with the following example:

global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
rule_files:
- "/etc/prometheus/rules/*.rules"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node_exporter"
static_configs:
- targets: ["localhost:9100"]
- job_name: "blackbox_http"
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://example.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115

Lets break this down:

• global: Sets the default scrape and evaluation intervals. 15 seconds is ideal for most use cases.
• alerting: Configures alertmanager integration (well cover this later).
• rule_files: Points to custom alerting and recording rules stored in /etc/prometheus/rules/.
• scrape_configs: Defines the targets Prometheus will monitor.
• job_name: "prometheus" Scrapes its own metrics (essential for self-monitoring).
• job_name: "node_exporter" Monitors system-level metrics from a Node Exporter running on the same host.
• job_name: "blackbox_http" Performs HTTP endpoint health checks using the Blackbox Exporter.

Save and exit the file (Ctrl+O, then Ctrl+X in nano).

Step 5: Install Node Exporter (System Metrics)

To monitor server-level metrics such as CPU, memory, disk I/O, and network usage, install Node Exportera Prometheus exporter that exposes hardware and OS metrics.

Download the latest Node Exporter binary:

cd /opt
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.7.0.linux-amd64.tar.gz
cd node_exporter-1.7.0.linux-amd64

Move the binary to the system path:

sudo mv node_exporter /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/node_exporter

Create a systemd service file to manage Node Exporter as a background service:

sudo nano /etc/systemd/system/node_exporter.service

Insert the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Reload systemd and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify its running:

sudo systemctl status node_exporter

You should see active (running). Now visit http://your-server-ip:9100/metrics in your browser to confirm metrics are being exposed.

Step 6: Install Blackbox Exporter (HTTP/ICMP Monitoring)

Blackbox Exporter allows Prometheus to probe endpoints over HTTP, HTTPS, DNS, TCP, and ICMP. This is essential for uptime monitoring of external services.

Download and install:

cd /opt
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz
tar xvfz blackbox_exporter-0.24.0.linux-amd64.tar.gz
cd blackbox_exporter-0.24.0.linux-amd64

Move the binary:

sudo mv blackbox_exporter /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/blackbox_exporter

Create a configuration file for Blackbox Exporter:

sudo nano /etc/prometheus/blackbox.yml

Add the following:

modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2"]
valid_status_codes: [200, 201, 301, 302]
method: GET
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
valid_status_codes: [200, 201]
tcp_connect:
prober: tcp_connect
timeout: 5s
ping_icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"

Create a systemd service for Blackbox Exporter:

sudo nano /etc/systemd/system/blackbox_exporter.service

Insert:

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/prometheus/blackbox.yml
[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable blackbox_exporter
sudo systemctl start blackbox_exporter

Verify its active: sudo systemctl status blackbox_exporter

Step 7: Create Prometheus Systemd Service

Now create a systemd service to manage Prometheus itself:

sudo nano /etc/systemd/system/prometheus.service

Insert the following:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console-template=/etc/prometheus/consoles \
--web.console.templates=/etc/prometheus/consoles \
--web.listen-address=0.0.0.0:9090 \
--web.enable-admin-api \
--web.enable-lifecycle \
--storage.tsdb.retention.time=15d \
--storage.tsdb.retention.size=50GB \
--log.level=info
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

Key flags explained:

• --config.file Points to your configuration file.
• --storage.tsdb.path Where time-series data is stored.
• --web.listen-address Makes Prometheus accessible on all interfaces on port 9090.
• --web.enable-admin-api Enables administrative APIs (use with caution in production).
• --web.enable-lifecycle Allows reloading config via HTTP POST to /-/reload.
• --storage.tsdb.retention.time How long to keep data (15 days recommended for most).
• --storage.tsdb.retention.size Maximum disk usage before data is deleted.

Reload systemd and start Prometheus:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Check the status:

sudo systemctl status prometheus

If running, open your browser and navigate to http://your-server-ip:9090. You should see the Prometheus web UI.

Step 8: Test Your Setup

Once Prometheus is running, verify it can scrape targets:

1. Go to http://your-server-ip:9090/targets
2. All targets should show UP status.
3. Click on prometheus and node_exporter to inspect the metrics being collected.

Try a simple query in the Expression Browser:

node_cpu_seconds_total

Or check memory usage:

node_memory_MemAvailable_bytes

Click Execute to see live data. If you see time-series graphs, your setup is successful.

Step 9: Set Up Firewall Rules

Ensure your firewall allows traffic on ports 9090 (Prometheus), 9100 (Node Exporter), and 9115 (Blackbox Exporter).

On Ubuntu with UFW:

sudo ufw allow 9090
sudo ufw allow 9100
sudo ufw allow 9115
sudo ufw reload

On CentOS with firewalld:

sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=9100/tcp
sudo firewall-cmd --permanent --add-port=9115/tcp
sudo firewall-cmd --reload

Step 10: Integrate with Grafana (Optional but Recommended)

While Prometheus collects metrics, it lacks advanced visualization. Grafana is the standard dashboarding tool for Prometheus.

Install Grafana:

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

Start and enable Grafana:

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Access Grafana at http://your-server-ip:3000. Default login: admin/admin.

Go to Configuration ? Data Sources ? Add data source, select Prometheus, and set the URL to http://localhost:9090. Click Save & Test.

Now import a pre-built dashboard: Go to Create ? Import, enter ID 1860 (Node Exporter Full), and click Load. Select your Prometheus data source.

You now have a fully operational monitoring stack with real-time system metrics displayed in beautiful dashboards.

Best Practices

Use Labels Consistently

Prometheus relies on labels to identify and group metrics. Always use consistent, meaningful labels such as environment=production, service=api, or region=us-east. Avoid dynamic or high-cardinality labels like user IDs or request tokens unless absolutely necessarythese can overwhelm your TSDB.

Implement Alerting Rules Early

Dont wait for outages to create alerts. Define critical thresholds early:

• CPU usage > 85% for 5 minutes
• Memory usage > 90%
• HTTP error rate > 1% over 10 minutes
• Service downtime (Blackbox exporter failure)

Store alert rules in separate files under /etc/prometheus/rules/ and reference them in prometheus.yml using the rule_files directive.

Separate Monitoring from Application Infrastructure

Run Prometheus on a dedicated server or cluster. Avoid co-locating it with application workloads to prevent resource contention during spikes. Use separate networks or VPCs for monitoring traffic if possible.

Enable Retention Policies

Set appropriate retention based on your storage capacity and compliance needs. For most environments, 1530 days is sufficient. Use --storage.tsdb.retention.time and --storage.tsdb.retention.size to enforce limits. Monitor disk usage regularly.

Use Service Discovery for Dynamic Environments

Static configs work for small setups, but for Kubernetes, Docker Swarm, or cloud auto-scaling, use service discovery mechanisms like kubernetes_sd_configs, ec2_sd_configs, or dns_sd_configs to auto-discover targets.

Secure Your Prometheus Instance

Never expose Prometheus to the public internet without authentication. Use reverse proxies like Nginx with basic auth or integrate with OAuth2 via tools like oauth2-proxy. Enable TLS using the --web.tls-certificate and --web.tls-private-key flags.

Regularly Review and Optimize Scraping

Scrape intervals should balance granularity with performance. Too frequent scraping (e.g., 1s) can overload targets and your TSDB. Too infrequent (e.g., 60s) may miss critical events. 15s is a sweet spot for most applications.

Backup Your Configuration and Rules

Store your Prometheus configuration, alert rules, and dashboards in version control (Git). This ensures reproducibility, auditability, and disaster recovery. Use CI/CD pipelines to deploy configuration changes safely.

Monitor Prometheus Itself

Prometheus exposes its own metrics at /metrics. Create alerts for:

• Prometheus target scrape failures
• TSDB head chunks exceeding thresholds
• Rule evaluation failures
• Memory usage > 80%

This ensures your monitoring system remains healthy.

Plan for Scale

If you plan to monitor hundreds or thousands of targets, consider using Prometheus Federation or Thanos for horizontal scaling. For long-term storage, integrate with Cortex or Mimir.

Tools and Resources

Core Prometheus Ecosystem Tools

• Prometheus Server The core time-series database and scraper.
• Node Exporter Exposes host-level metrics (CPU, memory, disk, network).
• Blackbox Exporter Probes HTTP, TCP, ICMP endpoints for uptime monitoring.
• Alertmanager Handles alerts sent by Prometheus, deduplicates, routes, and notifies via email, Slack, PagerDuty, etc.
• Grafana Visualization platform for building dashboards.
• Pushgateway Allows ephemeral or batch jobs to push metrics (use sparingly).

Exporters for Common Services

Use exporters to monitor applications and services:

• MySQL Exporter For MySQL database metrics
• PostgreSQL Exporter For PostgreSQL metrics
• Redis Exporter For Redis instance monitoring
• Blackbox Exporter For HTTP endpoint health checks
• HAProxy Exporter For load balancer metrics
• Kube-State-Metrics For Kubernetes resource state (pods, deployments, nodes)
• cadvisor For container metrics (built into Kubernetes)

Configuration and Validation Tools

• promtool Official CLI tool to validate configs, test rules, and manage alerts.
• Prometheus Config Validator Online tool to check YAML syntax.
• json2yaml Convert JSON-based configs to YAML if needed.

Learning and Community Resources

• Official Prometheus Documentation
• PromQL Tutorial
• Grafana Dashboard Library
• Prometheus Community GitHub
• Prometheus YouTube Channel
• Prometheus Blog

Monitoring as Code Tools

Use infrastructure-as-code tools to automate Prometheus deployments:

• Ansible For configuration management
• Terraform For provisioning cloud infrastructure
• Helm For deploying Prometheus on Kubernetes
• ArgoCD For GitOps-based configuration sync

Real Examples

Example 1: Monitoring a Web Application Stack

Imagine you run a Python Flask app with PostgreSQL and Redis, hosted on a Linux server.

Steps:

1. Install Node Exporter to monitor server health.
2. Install PostgreSQL Exporter and configure it to connect to your DB.
3. Install Redis Exporter to track cache hits, evictions, and memory usage.
4. Use Blackbox Exporter to monitor the apps health endpoint: https://app.example.com/health.
5. Write a recording rule to calculate 95th percentile response time:

groups:
- name: webapp
rules:
- record: job:http_request_duration_seconds:95pct
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Then create an alert:

- alert: HighLatency
expr: job:http_request_duration_seconds:95pct > 1
for: 5m
labels:
severity: warning
annotations:
summary: "High latency detected on webapp ({{ $value }}s)"
description: "The 95th percentile request duration has exceeded 1 second for 5 minutes."

Import Grafana dashboard ID 1860 (Node Exporter) and 1860 (PostgreSQL) for full visibility.

Example 2: Kubernetes Monitoring with Helm

Deploying Prometheus on Kubernetes is streamlined with Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack

This installs:

• Prometheus Server
• Alertmanager
• Kube-State-Metrics
• Node Exporter
• Grafana
• Pre-configured dashboards and alert rules

Access Grafana via port-forward:

kubectl port-forward svc/prometheus-grafana 3000:80

Now you have enterprise-grade monitoring for your entire Kubernetes cluster with zero manual configuration.

Example 3: Alerting on E-commerce Traffic Spikes

For an online store, you want to be alerted if traffic drops below 10% of the 7-day average:

- alert: TrafficDrop
expr: sum(rate(http_requests_total[5m])) 
for: 10m
labels:
severity: critical
annotations:
summary: "Critical traffic drop detected"
description: "HTTP requests have dropped below 10% of the 7-day average. Potential outage or DDoS."

Combine this with a Blackbox probe on the checkout endpoint to detect if the issue is backend-related or network-wide.

FAQs

What is Prometheus used for?

Prometheus is used for monitoring and alerting in modern, dynamic environments. It collects metrics from services and infrastructure, stores them as time-series data, and allows querying with PromQL to visualize trends, detect anomalies, and trigger alerts.

Is Prometheus better than Zabbix or Nagios?

Prometheus excels in cloud-native, containerized environments due to its pull-based model, rich data model, and integration with Kubernetes. Zabbix and Nagios are more suited for traditional, static infrastructure. Prometheus is more scalable and flexible, while Zabbix offers broader out-of-the-box integrations.

How much disk space does Prometheus need?

Storage depends on the number of time series and retention period. A typical server with 1000 metrics and 15-day retention uses ~1020 GB. Use --storage.tsdb.retention.size to cap usage. For high-cardinality systems, plan for 100+ GB.

Can Prometheus monitor Windows servers?

Yes, using the Windows Exporter (https://github.com/prometheus-community/windows_exporter). It exposes metrics like CPU, memory, disk, and service status.

Does Prometheus support log monitoring?

No, Prometheus is designed for metrics only. For logs, use Loki (also from Grafana Labs) alongside Prometheus for a complete observability stack.

How do I reload Prometheus config without restarting?

Send a POST request to the /-/reload endpoint:

curl -X POST http://localhost:9090/-/reload

This is only available if --web.enable-lifecycle is enabled.

What is the difference between Prometheus and Grafana?

Prometheus collects and stores metrics. Grafana visualizes them. Prometheus is the data source; Grafana is the dashboarding layer. They work together but serve different purposes.

Can I run Prometheus on Docker?

Yes. Use the official image:

docker run -d -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

But for production, prefer systemd-based installation for better stability and resource control.

How do I monitor a third-party API with Prometheus?

Use the Blackbox Exporter to probe HTTP endpoints. Configure it to check status codes, response times, and content. Then scrape the Blackbox Exporters metrics.

Is Prometheus suitable for small projects?

Absolutely. Even a single server with a web app can benefit from Prometheus. The setup is lightweight, and the insights gainedlike identifying slow database queries or memory leaksare invaluable at any scale.

Conclusion

Setting up Prometheus is a foundational skill for modern DevOps and SRE teams. From its simple yet powerful architecture to its rich ecosystem of exporters and integrations, Prometheus provides unparalleled visibility into your systems without the complexity of legacy tools. This guide walked you through installing Prometheus, configuring targets, integrating exporters, securing your setup, and connecting it to Grafana for visualizationall essential steps for building a reliable monitoring infrastructure.

Remember: monitoring is not a one-time task. Its an ongoing practice. Regularly review your alerts, optimize your scrapes, and expand your coverage as your systems evolve. Use version control for your configurations, automate deployments with CI/CD, and always monitor Prometheus itself.

With Prometheus, youre not just collecting datayoure building a culture of observability. Whether youre managing a single VM or a global Kubernetes cluster, a properly configured Prometheus stack gives you the confidence to deploy faster, troubleshoot smarter, and sleep better at night.

Now that youve successfully set up Prometheus, the next step is to dive deeper into PromQL, create custom dashboards, and implement alerting policies tailored to your business needs. The journey from raw metrics to actionable insights begins here.