Agency Core Infrastructure: centralized telemetry across multi-VPS

Flying blind

Managing multiple VPS environments for different agency clients was an operational nightmare. Each server was a black box. When something broke at 3 AM, we had no centralized logging, no distributed tracing, and no alerting. Mean time to resolution (MTTR) sat at 4-6 hours. Unacceptable.

The goal was a centralized observability stack that surfaces metrics, traces, and logs from every node in one dashboard. Internal management ports had to stay off the public internet.

Hub and spoke

The topology is hub-and-spoke, with a dedicated observability node as the central aggregator.

flowchart BT
  subgraph Hub ["Observability Hub (VPS-0)"]
    direction TB
    Nginx["Nginx<br>(Reverse proxy + mTLS termination)"]
    SigNoz["SigNoz<br>(Traces + Metrics + Logs)"]
    Nginx --> SigNoz
  end
  
  subgraph VPS1 ["Client VPS-1"]
    direction TB
    Docker1["Docker services"]
    Otel1["OTel Collector"]
    Docker1 --> Otel1
  end
 
  subgraph VPS2 ["Client VPS-2"]
    direction TB
    Docker2["Docker services"]
    Otel2["OTel Collector"]
    Docker2 --> Otel2
  end
  
  Otel1 -- "SSH Reverse Tunnel" --> Nginx
  Otel2 -- "SSH Reverse Tunnel" --> Nginx

Each client VPS runs a lightweight OpenTelemetry Collector that ships spans, metrics, and structured logs to the hub over a persistent SSH reverse tunnel, so telemetry never needs a public port.

How it's built

SSH reverse tunnels

Rather than opening OTLP gRPC ports (4317) on a public interface, each spoke node maintains an autossh-managed tunnel:

# On each client node — systemd managed
autossh -M 0 -N \
  -R 4317:localhost:4317 \
  -o ServerAliveInterval=30 \
  hub-user@observability-hub

The client VPS needs zero firewall changes. The tunnel is the transport.

Docker Compose stack

The observability hub runs SigNoz in Docker Compose with persistent volumes:

services:
  clickhouse:
    image: clickhouse/clickhouse-server:24.1
    volumes: [clickhouse_data:/var/lib/clickhouse]
 
  signoz-otel-collector:
    image: signoz/signoz-otel-collector:latest
    ports:
      - "127.0.0.1:4317:4317" # Only localhost — tunnels connect here
    depends_on: [clickhouse]
 
  signoz-frontend:
    image: signoz/frontend:latest
    ports:
      - "3301:3301" # Exposed via Nginx with auth

Nginx + Basic Auth

The SigNoz frontend sits behind Nginx with HTTP basic auth and TLS via Let's Encrypt.

Why these choices

• SSH tunnels over VPN. Simpler to manage per-client. No PKI infrastructure needed.
• SigNoz over the Grafana stack. Self-hosted, single binary deployment, native OTLP support without the complexity of a full Prometheus/Loki/Tempo stack.
• Docker for everything. Reproducible deployments across any Ubuntu/Debian VPS with a single docker compose up -d.

Results

MTTR dropped from 4-6 hours to under 45 minutes. SLA compliance improved, and client-facing downtime incidents fell.