Distributed Observability Architecture with Grafana Stack
Author: Portelcore Team Objective: Our goal is to deploy a complete observability environment using Grafana, Mimir, Loki, Tempo, and Alloy...
Author: Portelcore Team Objective: Our goal is to deploy a complete observability environment using Grafana, Mimir, Loki, Tempo, and Alloy on Ubuntu 24.04 LTS to ingest and analyze metrics, logs, and traces from a web application (WordPress).
Architecture and Servers
Sección titulada «Arquitectura y Servidores»We have 3 Ubuntu 24.04 LTS servers with the following roles:
| Server | Role / Services | IP Address |
|---|---|---|
| Server 1 (Backend) | Mimir (Metrics) + Loki (Logs) + Tempo (Traces) | 172.20.0.151 |
| Server 2 (Frontend) | Grafana (UI Visualization Interface) | 172.20.0.166 |
| Server 3 (App/Agent) | Web Server (WordPress) + Grafana Alloy (Agent) | 172.20.0.176 |
PHASE 0: Initial Preparation (On the 3 Servers)
Sección titulada «FASE 0: Preparación Inicial (En los 3 Servidores)»Before starting, we must execute these commands on the 3 servers to ensure that the system is up to date, in the correct time zone, and has the necessary tools.
sudo apt update && sudo apt upgrade -ysudo timedatectl set-timezone Europe/Madridsudo apt install curl wget unzip nano -yPHASE 1: Backend Server (Mimir, Loki, and Tempo)
Sección titulada «FASE 1: Servidor Backend (Mimir, Loki y Tempo)»We will execute on Server 1 (172.20.0.151)
This server will receive all data sent by the agents. To manage them professionally, we will create systemd services so that they run in the background and start automatically with the system.
1.1 Metrics Ingestion: Grafana Mimir (Port 9009)
Sección titulada «1.1 Ingesta de Métricas: Grafana Mimir (Puerto 9009)»- We will download and install the binary:
wget https://github.com/grafana/mimir/releases/latest/download/mimir-linux-amd64chmod +x mimir-linux-amd64sudo mv mimir-linux-amd64 /usr/local/bin/mimir- We will create the configuration and directories:
sudo mkdir -p /etc/mimirsudo mkdir -p /var/lib/mimir/blockssudo mkdir -p /var/lib/mimir/compactorsudo nano /etc/mimir/mimir.yamlWe will paste the following content into mimir.yaml:
multitenancy_enabled: false
server: http_listen_port: 9009 grpc_listen_port: 9096
blocks_storage: backend: filesystem filesystem: dir: /var/lib/mimir/blocks
ingester: ring: replication_factor: 1 kvstore: store: inmemory
compactor: data_dir: /var/lib/mimir/compactor- We will create the Systemd service for Mimir:
sudo tee /etc/systemd/system/mimir.service > /dev/null <<EOF[Unit]Description=Grafana MimirAfter=network.target
[Service]ExecStart=/usr/local/bin/mimir -config.file=/etc/mimir/mimir.yamlRestart=always
[Install]WantedBy=multi-user.targetEOF1.2 Log Ingestion: Grafana Loki (Port 3100)
Sección titulada «1.2 Ingesta de Logs: Grafana Loki (Puerto 3100)»- We will download and install the binary:
wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zipunzip loki-linux-amd64.zipchmod +x loki-linux-amd64sudo mv loki-linux-amd64 /usr/local/bin/loki- We will create the configuration and directories:
sudo mkdir -p /etc/lokisudo mkdir -p /var/lib/loki/chunkssudo mkdir -p /var/lib/loki/rulessudo nano /etc/loki/loki.yamlWe will paste the following content into loki.yaml:
auth_enabled: false
server: http_listen_port: 3100 grpc_listen_port: 9097
common: path_prefix: /var/lib/loki storage: filesystem: chunks_directory: /var/lib/loki/chunks rules_directory: /var/lib/loki/rules replication_factor: 1 ring: kvstore: store: inmemory
schema_config: configs: - from: 2026-01-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h- We will create the Systemd service for Loki:
sudo tee /etc/systemd/system/loki.service > /dev/null <<EOF[Unit]Description=Grafana LokiAfter=network.target
[Service]ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yamlRestart=always
[Install]WantedBy=multi-user.targetEOF1.3 Trace Ingestion: Grafana Tempo (Port 3200)
Sección titulada «1.3 Ingesta de Traces: Grafana Tempo (Puerto 3200)»- We will download and install the binary (pinned to stable version v2.10.1):
wget https://github.com/grafana/tempo/releases/download/v2.10.1/tempo_2.10.1_linux_amd64.tar.gztar -xvzf tempo_2.10.1_linux_amd64.tar.gzsudo mv tempo-linux-amd64 /usr/local/bin/tempo 2>/dev/null || sudo mv tempo /usr/local/bin/temposudo chmod +x /usr/local/bin/tempo- We will create the configuration and directories:
sudo mkdir -p /etc/temposudo mkdir -p /var/lib/tempo/tracessudo nano /etc/tempo/tempo.yamlWe will paste the following content into tempo.yaml:
server: http_listen_port: 3200 grpc_listen_port: 9095
distributor: receivers: otlp: protocols: http: grpc:
storage: trace: backend: local local: path: /var/lib/tempo/traces- We will create the Systemd service for Tempo:
sudo tee /etc/systemd/system/tempo.service > /dev/null <<EOF[Unit]Description=Grafana TempoAfter=network.target
[Service]ExecStart=/usr/local/bin/tempo -config.file=/etc/tempo/tempo.yamlRestart=always
[Install]WantedBy=multi-user.targetEOF1.4 We will start all services at the same time and in the background
Sección titulada «1.4 Iniciaremos todos los servicios al mismo tiempo y en segundo plano»Now that we have created the systemd rules for all three, we will reload the system configuration and enable/start them all at once with a single command:
sudo systemctl daemon-reloadsudo systemctl enable --now mimir loki tempo(We can verify that all three are running happily in the background with sudo systemctl status mimir loki tempo).
PHASE 2: Frontend Server (Grafana UI)
Sección titulada «FASE 2: Servidor Frontend (Grafana UI)»We will execute on Server 2 (172.20.0.166)
- We will install Grafana from the official repository:
sudo apt install -y software-properties-common apt-transport-httpssudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.keyecho "deb https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.listsudo apt updatesudo apt install grafana -y- We will start and enable the service:
sudo systemctl enable --now grafana-server- We will access the Interface and Configure:
- We open our web browser and go to:
http://172.20.0.166:3000 - Login:
admin/ Password:admin(it will ask to change it). - Let's go to Connections > Data Sources > Add data source and configure these 3 pointing to Server 1 (
172.20.0.151):- Prometheus (For Mimir): URL
http://172.20.0.151:9009/prometheus - Loki: URL
http://172.20.0.151:3100 - Tempo: URL
http://172.20.0.151:3200
- Prometheus (For Mimir): URL
- We open our web browser and go to:
PHASE 3: Application Server (WordPress + Alloy)
Sección titulada «FASE 3: Servidor de Aplicación (WordPress + Alloy)»We will execute on Server 3 (172.20.0.176)
3.1 We will install the Application (WordPress LAMP Stack)
Sección titulada «3.1 Instalaremos la Aplicación (WordPress Stack LAMP)»# 1. Instalamos LAMPsudo apt install apache2 mysql-server php php-mysql libapache2-mod-php -y
# 2. Habilitamos el módulo de estado de Apache (Necesario para obtener métricas)sudo a2enmod statussudo systemctl restart apache2
# 3. Configuramos la base de datos (Entramos con 'sudo mysql' y pegamos estas líneas)sudo mysql -e "CREATE DATABASE wordpress;"sudo mysql -e "CREATE USER 'wpuser'@'localhost' IDENTIFIED BY 'password';"sudo mysql -e "GRANT ALL PRIVILEGES ON wordpress.* TO 'wpuser'@'localhost';"sudo mysql -e "FLUSH PRIVILEGES;"
# 4. Descargamos e instalamos WordPresscd /var/www/htmlsudo rm index.htmlsudo wget https://wordpress.org/latest.tar.gzsudo tar -xvzf latest.tar.gzsudo chown -R www-data:www-data wordpressWe will be able to access http://172.20.0.176/wordpress to finish the visual installation.
3.2 We will install and configure Grafana Alloy (The Global Agent)
Sección titulada «3.2 Instalaremos y Configuraremos Grafana Alloy (El Agente Global)»- We will download and install the binary:
cd ~wget https://github.com/grafana/alloy/releases/latest/download/alloy-linux-amd64.zipunzip alloy-linux-amd64.zipchmod +x alloy-linux-amd64sudo mv alloy-linux-amd64 /usr/local/bin/alloy- We will create the complete configuration file:
sudo mkdir -p /etc/alloysudo nano /etc/alloy/config.alloyWe will paste the following content. This configuration extracts Disk, CPU, Apache, MySQL, Logs, and Traces:
logging { level = "info"}
// --------------------------------------------------------// 1. MÉTRICAS DEL SISTEMA (Disco, CPU, RAM, Red)// --------------------------------------------------------prometheus.exporter.unix "node" {}
prometheus.scrape "node" { targets = prometheus.exporter.unix.node.targets forward_to = [prometheus.remote_write.mimir.receiver]}
// --------------------------------------------------------// 2. MÉTRICAS DE APACHE (Servidor Web)// --------------------------------------------------------prometheus.exporter.regular_apache = prometheus.exporter.apache "apache" { scrape_uri = "http://localhost/server-status?auto"}
prometheus.scrape "apache" { targets = prometheus.exporter.apache.apache.targets forward_to = [prometheus.remote_write.mimir.receiver]}
// --------------------------------------------------------// 3. MÉTRICAS DE MYSQL (Base de Datos)// --------------------------------------------------------prometheus.exporter.mysql "mysql" { data_source_name = "wpuser:password@(localhost:3306)/"}
prometheus.scrape "mysql" { targets = prometheus.exporter.mysql.mysql.targets forward_to = [prometheus.remote_write.mimir.receiver]}
// --------------------------------------------------------// 4. OBSERVABILIDAD INTERNA (Mimir, Loki, Tempo)// --------------------------------------------------------prometheus.scrape "backend_services" { targets = [ {"__address__" = "172.20.0.151:9009", "job" = "mimir"}, {"__address__" = "172.20.0.151:3100", "job" = "loki"}, {"__address__" = "172.20.0.151:3200", "job" = "tempo"}, ] forward_to = [prometheus.remote_write.mimir.receiver] scrape_interval = "15s"}
// 5. ENVIAR TODAS LAS MÉTRICAS A MIMIRprometheus.remote_write "mimir" { endpoint { url = "http://172.20.0.151:9009/api/v1/push" }}
// --------------------------------------------------------// 6. LOGS (Sistema y Apache)// --------------------------------------------------------loki.source.file "system_logs" { targets = [ {__path__ = "/var/log/syslog", "job" = "syslog"}, {__path__ = "/var/log/auth.log", "job" = "auth"}, {__path__ = "/var/log/apache2/*.log", "job" = "apache"}, ] forward_to = [loki.write.loki.receiver]}
// 7. ENVIAR LOGS A LOKIloki.write "loki" { endpoint { url = "http://172.20.0.151:3100/loki/api/v1/push" }}
// --------------------------------------------------------// 8. TRACES (Recepción y Envío a Tempo)// --------------------------------------------------------otelcol.receiver.otlp "default" { http {} grpc {} output { traces = [otelcol.exporter.otlp.tempo.input] }}
otelcol.exporter.otlp "tempo" { client { endpoint = "172.20.0.151:4317" tls { insecure = true } }}- We will create the Systemd service for Alloy:
sudo tee /etc/systemd/system/alloy.service > /dev/null <<EOF[Unit]Description=Grafana AlloyAfter=network.target
[Service]ExecStart=/usr/local/bin/alloy run /etc/alloy/config.alloyRestart=always
[Install]WantedBy=multi-user.targetEOF- We will run Alloy (To collect and send data in the background):
sudo systemctl daemon-reloadsudo systemctl enable --now alloyPHASE 4: Visualization and Stress Simulations
Sección titulada «FASE 4: Visualización y Simulaciones de Estrés»To verify that everything is working, we will open Grafana (http://172.20.0.166:3000), go to Explore, and test the Data Sources.
4.1 Simulations from our machine (or from Server 3)
Sección titulada «4.1 Simulaciones desde nuestro equipo (o desde el Servidor 3)»Simulation 1: Brute Force Attack (Generates Logs and Latency) We will launch 50 simultaneous requests against the WordPress login to generate noise in the Apache logs and increase latency.
for i in {1..50}; do curl -s -X POST http://172.20.0.176/wordpress/wp-login.php > /dev/null; doneIn Grafana -> Explore -> Loki: We will search for {job="apache"} and see all accesses being recorded in real time.
Simulation 2: CPU Saturation (Generates high metrics) This command will stress a CPU core to its limit.
yes > /dev/null &In Grafana -> Explore -> Prometheus (Mimir): We will look for node_cpu_seconds_total (By CPU) or mysql_up (To verify that the database is working).
(To stop the CPU saturation, we will execute killall yes).
Trace Verification: In Grafana -> Explore -> Tempo: We will search for service.name="wordpress" to see the path of the HTTP requests we have made.
PHASE 5: Recommended Official Dashboards
Sección titulada «FASE 5: Dashboards Oficiales Recomendados»Since Alloy is using standard exporters (Node, Apache, MySQL), we don't have to create the charts from scratch. We can import official templates from the Grafana community.
Import Instructions:
- We will go to our Grafana (
http://172.20.0.166:3000). - In the left menu, we will click on the "+" symbol -> Import.
- In the "Import via grafana.com" box, we will enter one of the IDs listed below and click Load.
- We will select our Data Source (Prometheus/Mimir) from the dropdown below and press Import.
Templates to Import:
Sección titulada «Plantillas a Importar:»- ID
1860(Node Exporter Full): This is the most famous dashboard in the world for monitoring physical server health. It will show us CPU usage, remaining disk space on our WordPress server, RAM usage, network bandwidth, etc. - ID
3894(Apache Overview): It will show us in real time how many web requests are entering our WordPress, the response time, active workers, and bytes transferred. - ID
7362(MySQL Overview): It will give us visibility into the WordPress database engine: queries per second, active connections, cache memory, and InnoDB engine performance.
What about Logs and Traces?
Loki (Logs) and Tempo (Traces) are usually best visualized through the Grafana Explore tab. If we want to add them to a Dashboard, we just need to edit any imported panel, add a new query (Query), select “Loki” as the Data Source, and type {job="apache"} to have the web access logs alongside the CPU graphs.

