Ir al contenido

Distributed Observability Architecture with Grafana Stack

Author: Portelcore Team Objective: Our goal is to deploy a complete observability environment using Grafana, Mimir, Loki, Tempo, and Alloy...

0views

Author: Portelcore Team Objective: Our goal is to deploy a complete observability environment using Grafana, Mimir, Loki, Tempo, and Alloy on Ubuntu 24.04 LTS to ingest and analyze metrics, logs, and traces from a web application (WordPress).

We have 3 Ubuntu 24.04 LTS servers with the following roles:

ServerRole / ServicesIP Address
Server 1 (Backend)Mimir (Metrics) + Loki (Logs) + Tempo (Traces)172.20.0.151
Server 2 (Frontend)Grafana (UI Visualization Interface)172.20.0.166
Server 3 (App/Agent)Web Server (WordPress) + Grafana Alloy (Agent)172.20.0.176

Before starting, we must execute these commands on the 3 servers to ensure that the system is up to date, in the correct time zone, and has the necessary tools.

Ventana de terminal
sudo apt update && sudo apt upgrade -y
sudo timedatectl set-timezone Europe/Madrid
sudo apt install curl wget unzip nano -y

PHASE 1: Backend Server (Mimir, Loki, and Tempo)

Sección titulada «FASE 1: Servidor Backend (Mimir, Loki y Tempo)»

We will execute on Server 1 (172.20.0.151)

This server will receive all data sent by the agents. To manage them professionally, we will create systemd services so that they run in the background and start automatically with the system.

  1. We will download and install the binary:
Ventana de terminal
wget https://github.com/grafana/mimir/releases/latest/download/mimir-linux-amd64
chmod +x mimir-linux-amd64
sudo mv mimir-linux-amd64 /usr/local/bin/mimir
  1. We will create the configuration and directories:
Ventana de terminal
sudo mkdir -p /etc/mimir
sudo mkdir -p /var/lib/mimir/blocks
sudo mkdir -p /var/lib/mimir/compactor
sudo nano /etc/mimir/mimir.yaml

We will paste the following content into mimir.yaml:

multitenancy_enabled: false
server:
http_listen_port: 9009
grpc_listen_port: 9096
blocks_storage:
backend: filesystem
filesystem:
dir: /var/lib/mimir/blocks
ingester:
ring:
replication_factor: 1
kvstore:
store: inmemory
compactor:
data_dir: /var/lib/mimir/compactor
  1. We will create the Systemd service for Mimir:
Ventana de terminal
sudo tee /etc/systemd/system/mimir.service > /dev/null <<EOF
[Unit]
Description=Grafana Mimir
After=network.target
[Service]
ExecStart=/usr/local/bin/mimir -config.file=/etc/mimir/mimir.yaml
Restart=always
[Install]
WantedBy=multi-user.target
EOF
  1. We will download and install the binary:
Ventana de terminal
wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
chmod +x loki-linux-amd64
sudo mv loki-linux-amd64 /usr/local/bin/loki
  1. We will create the configuration and directories:
Ventana de terminal
sudo mkdir -p /etc/loki
sudo mkdir -p /var/lib/loki/chunks
sudo mkdir -p /var/lib/loki/rules
sudo nano /etc/loki/loki.yaml

We will paste the following content into loki.yaml:

auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9097
common:
path_prefix: /var/lib/loki
storage:
filesystem:
chunks_directory: /var/lib/loki/chunks
rules_directory: /var/lib/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2026-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
  1. We will create the Systemd service for Loki:
Ventana de terminal
sudo tee /etc/systemd/system/loki.service > /dev/null <<EOF
[Unit]
Description=Grafana Loki
After=network.target
[Service]
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yaml
Restart=always
[Install]
WantedBy=multi-user.target
EOF
  1. We will download and install the binary (pinned to stable version v2.10.1):
Ventana de terminal
wget https://github.com/grafana/tempo/releases/download/v2.10.1/tempo_2.10.1_linux_amd64.tar.gz
tar -xvzf tempo_2.10.1_linux_amd64.tar.gz
sudo mv tempo-linux-amd64 /usr/local/bin/tempo 2>/dev/null || sudo mv tempo /usr/local/bin/tempo
sudo chmod +x /usr/local/bin/tempo
  1. We will create the configuration and directories:
Ventana de terminal
sudo mkdir -p /etc/tempo
sudo mkdir -p /var/lib/tempo/traces
sudo nano /etc/tempo/tempo.yaml

We will paste the following content into tempo.yaml:

server:
http_listen_port: 3200
grpc_listen_port: 9095
distributor:
receivers:
otlp:
protocols:
http:
grpc:
storage:
trace:
backend: local
local:
path: /var/lib/tempo/traces
  1. We will create the Systemd service for Tempo:
Ventana de terminal
sudo tee /etc/systemd/system/tempo.service > /dev/null <<EOF
[Unit]
Description=Grafana Tempo
After=network.target
[Service]
ExecStart=/usr/local/bin/tempo -config.file=/etc/tempo/tempo.yaml
Restart=always
[Install]
WantedBy=multi-user.target
EOF

1.4 We will start all services at the same time and in the background

Sección titulada «1.4 Iniciaremos todos los servicios al mismo tiempo y en segundo plano»

Now that we have created the systemd rules for all three, we will reload the system configuration and enable/start them all at once with a single command:

Ventana de terminal
sudo systemctl daemon-reload
sudo systemctl enable --now mimir loki tempo

(We can verify that all three are running happily in the background with sudo systemctl status mimir loki tempo).

We will execute on Server 2 (172.20.0.166)

  1. We will install Grafana from the official repository:
Ventana de terminal
sudo apt install -y software-properties-common apt-transport-https
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana -y
  1. We will start and enable the service:
Ventana de terminal
sudo systemctl enable --now grafana-server
  1. We will access the Interface and Configure:
    • We open our web browser and go to: http://172.20.0.166:3000
    • Login: admin / Password: admin (it will ask to change it).
    • Let's go to Connections > Data Sources > Add data source and configure these 3 pointing to Server 1 (172.20.0.151):
      • Prometheus (For Mimir): URL http://172.20.0.151:9009/prometheus
      • Loki: URL http://172.20.0.151:3100
      • Tempo: URL http://172.20.0.151:3200

We will execute on Server 3 (172.20.0.176)

3.1 We will install the Application (WordPress LAMP Stack)

Sección titulada «3.1 Instalaremos la Aplicación (WordPress Stack LAMP)»
Ventana de terminal
# 1. Instalamos LAMP
sudo apt install apache2 mysql-server php php-mysql libapache2-mod-php -y
# 2. Habilitamos el módulo de estado de Apache (Necesario para obtener métricas)
sudo a2enmod status
sudo systemctl restart apache2
# 3. Configuramos la base de datos (Entramos con 'sudo mysql' y pegamos estas líneas)
sudo mysql -e "CREATE DATABASE wordpress;"
sudo mysql -e "CREATE USER 'wpuser'@'localhost' IDENTIFIED BY 'password';"
sudo mysql -e "GRANT ALL PRIVILEGES ON wordpress.* TO 'wpuser'@'localhost';"
sudo mysql -e "FLUSH PRIVILEGES;"
# 4. Descargamos e instalamos WordPress
cd /var/www/html
sudo rm index.html
sudo wget https://wordpress.org/latest.tar.gz
sudo tar -xvzf latest.tar.gz
sudo chown -R www-data:www-data wordpress

We will be able to access http://172.20.0.176/wordpress to finish the visual installation.

3.2 We will install and configure Grafana Alloy (The Global Agent)

Sección titulada «3.2 Instalaremos y Configuraremos Grafana Alloy (El Agente Global)»
  1. We will download and install the binary:
Ventana de terminal
cd ~
wget https://github.com/grafana/alloy/releases/latest/download/alloy-linux-amd64.zip
unzip alloy-linux-amd64.zip
chmod +x alloy-linux-amd64
sudo mv alloy-linux-amd64 /usr/local/bin/alloy
  1. We will create the complete configuration file:
Ventana de terminal
sudo mkdir -p /etc/alloy
sudo nano /etc/alloy/config.alloy

We will paste the following content. This configuration extracts Disk, CPU, Apache, MySQL, Logs, and Traces:

Ventana de terminal
logging {
level = "info"
}
// --------------------------------------------------------
// 1. MÉTRICAS DEL SISTEMA (Disco, CPU, RAM, Red)
// --------------------------------------------------------
prometheus.exporter.unix "node" {}
prometheus.scrape "node" {
targets = prometheus.exporter.unix.node.targets
forward_to = [prometheus.remote_write.mimir.receiver]
}
// --------------------------------------------------------
// 2. MÉTRICAS DE APACHE (Servidor Web)
// --------------------------------------------------------
prometheus.exporter.regular_apache = prometheus.exporter.apache "apache" {
scrape_uri = "http://localhost/server-status?auto"
}
prometheus.scrape "apache" {
targets = prometheus.exporter.apache.apache.targets
forward_to = [prometheus.remote_write.mimir.receiver]
}
// --------------------------------------------------------
// 3. MÉTRICAS DE MYSQL (Base de Datos)
// --------------------------------------------------------
prometheus.exporter.mysql "mysql" {
data_source_name = "wpuser:password@(localhost:3306)/"
}
prometheus.scrape "mysql" {
targets = prometheus.exporter.mysql.mysql.targets
forward_to = [prometheus.remote_write.mimir.receiver]
}
// --------------------------------------------------------
// 4. OBSERVABILIDAD INTERNA (Mimir, Loki, Tempo)
// --------------------------------------------------------
prometheus.scrape "backend_services" {
targets = [
{"__address__" = "172.20.0.151:9009", "job" = "mimir"},
{"__address__" = "172.20.0.151:3100", "job" = "loki"},
{"__address__" = "172.20.0.151:3200", "job" = "tempo"},
]
forward_to = [prometheus.remote_write.mimir.receiver]
scrape_interval = "15s"
}
// 5. ENVIAR TODAS LAS MÉTRICAS A MIMIR
prometheus.remote_write "mimir" {
endpoint {
url = "http://172.20.0.151:9009/api/v1/push"
}
}
// --------------------------------------------------------
// 6. LOGS (Sistema y Apache)
// --------------------------------------------------------
loki.source.file "system_logs" {
targets = [
{__path__ = "/var/log/syslog", "job" = "syslog"},
{__path__ = "/var/log/auth.log", "job" = "auth"},
{__path__ = "/var/log/apache2/*.log", "job" = "apache"},
]
forward_to = [loki.write.loki.receiver]
}
// 7. ENVIAR LOGS A LOKI
loki.write "loki" {
endpoint {
url = "http://172.20.0.151:3100/loki/api/v1/push"
}
}
// --------------------------------------------------------
// 8. TRACES (Recepción y Envío a Tempo)
// --------------------------------------------------------
otelcol.receiver.otlp "default" {
http {}
grpc {}
output {
traces = [otelcol.exporter.otlp.tempo.input]
}
}
otelcol.exporter.otlp "tempo" {
client {
endpoint = "172.20.0.151:4317"
tls {
insecure = true
}
}
}
  1. We will create the Systemd service for Alloy:
Ventana de terminal
sudo tee /etc/systemd/system/alloy.service > /dev/null <<EOF
[Unit]
Description=Grafana Alloy
After=network.target
[Service]
ExecStart=/usr/local/bin/alloy run /etc/alloy/config.alloy
Restart=always
[Install]
WantedBy=multi-user.target
EOF
  1. We will run Alloy (To collect and send data in the background):
Ventana de terminal
sudo systemctl daemon-reload
sudo systemctl enable --now alloy

To verify that everything is working, we will open Grafana (http://172.20.0.166:3000), go to Explore, and test the Data Sources.

Simulation 1: Brute Force Attack (Generates Logs and Latency) We will launch 50 simultaneous requests against the WordPress login to generate noise in the Apache logs and increase latency.

Ventana de terminal
for i in {1..50}; do curl -s -X POST http://172.20.0.176/wordpress/wp-login.php > /dev/null; done

In Grafana -> Explore -> Loki: We will search for {job="apache"} and see all accesses being recorded in real time.

Simulation 2: CPU Saturation (Generates high metrics) This command will stress a CPU core to its limit.

yes > /dev/null &

In Grafana -> Explore -> Prometheus (Mimir): We will look for node_cpu_seconds_total (By CPU) or mysql_up (To verify that the database is working). (To stop the CPU saturation, we will execute killall yes).

Trace Verification: In Grafana -> Explore -> Tempo: We will search for service.name="wordpress" to see the path of the HTTP requests we have made.

Since Alloy is using standard exporters (Node, Apache, MySQL), we don't have to create the charts from scratch. We can import official templates from the Grafana community.

Import Instructions:

  1. We will go to our Grafana (http://172.20.0.166:3000).
  2. In the left menu, we will click on the "+" symbol -> Import.
  3. In the "Import via grafana.com" box, we will enter one of the IDs listed below and click Load.
  4. We will select our Data Source (Prometheus/Mimir) from the dropdown below and press Import.
  • ID 1860 (Node Exporter Full): This is the most famous dashboard in the world for monitoring physical server health. It will show us CPU usage, remaining disk space on our WordPress server, RAM usage, network bandwidth, etc.
  • ID 3894 (Apache Overview): It will show us in real time how many web requests are entering our WordPress, the response time, active workers, and bytes transferred.
  • ID 7362 (MySQL Overview): It will give us visibility into the WordPress database engine: queries per second, active connections, cache memory, and InnoDB engine performance.

What about Logs and Traces? Loki (Logs) and Tempo (Traces) are usually best visualized through the Grafana Explore tab. If we want to add them to a Dashboard, we just need to edit any imported panel, add a new query (Query), select “Loki” as the Data Source, and type {job="apache"} to have the web access logs alongside the CPU graphs.