Agency Core Infrastructure: Centralized Telemetry Across Multi-VPS

The Problem

Managing multiple VPS environments for different agency clients was an operational nightmare. Each server was a black box — when something broke at 3 AM, you were flying blind. No centralized logging, no distributed tracing, no alerting. The mean time to resolution (MTTR) for incidents was sitting at 4-6 hours. Unacceptable.

The goal: build a centralized observability stack that surfaces metrics, traces, and logs from every node into a single pane of glass — without exposing internal management ports to the public internet.

Architecture Overview

The solution revolves around a hub-and-spoke topology with a dedicated observability node acting as the central aggregator.

flowchart BT
  subgraph Hub ["Observability Hub (VPS-0)"]
    direction TB
    Nginx["Nginx<br>(Reverse proxy + mTLS termination)"]
    SigNoz["SigNoz<br>(Traces + Metrics + Logs)"]
    Nginx --> SigNoz
  end
  
  subgraph VPS1 ["Client VPS-1"]
    direction TB
    Docker1["Docker services"]
    Otel1["OTel Collector"]
    Docker1 --> Otel1
  end
 
  subgraph VPS2 ["Client VPS-2"]
    direction TB
    Docker2["Docker services"]
    Otel2["OTel Collector"]
    Docker2 --> Otel2
  end
  
  Otel1 -- "SSH Reverse Tunnel" --> Nginx
  Otel2 -- "SSH Reverse Tunnel" --> Nginx

Each client VPS runs a lightweight OpenTelemetry Collector that ships spans, metrics, and structured logs to the hub via a persistent SSH reverse tunnel — no public port exposure required.

Implementation Details

Secure Tunneling

Rather than opening OTLP gRPC ports (4317) on a public interface, each spoke node maintains an autossh managed tunnel:

# On each client node — systemd managed
autossh -M 0 -N \
  -R 4317:localhost:4317 \
  -o ServerAliveInterval=30 \
  hub-user@observability-hub

This approach means zero firewall changes on the client VPS. The tunnel is the transport.

Docker Compose Stack

The observability hub runs a composed SigNoz stack with persistent volumes:

services:
  clickhouse:
    image: clickhouse/clickhouse-server:24.1
    volumes: [clickhouse_data:/var/lib/clickhouse]
 
  signoz-otel-collector:
    image: signoz/signoz-otel-collector:latest
    ports:
      - "127.0.0.1:4317:4317" # Only localhost — tunnels connect here
    depends_on: [clickhouse]
 
  signoz-frontend:
    image: signoz/frontend:latest
    ports:
      - "3301:3301" # Exposed via Nginx with auth

Nginx + Basic Auth

The SigNoz frontend is exposed behind Nginx with HTTP basic auth and TLS via Let's Encrypt, preventing unauthorized dashboard access.

Key Decisions

• SSH tunnels over VPN: Simpler to manage per-client, no PKI infrastructure needed.
• SigNoz over Grafana Stack: Self-hosted, single binary deployment, native OTLP support without the complexity of a full Prometheus/Loki/Tempo stack.
• Docker for everything: Reproducible deployments across any Ubuntu/Debian VPS with a single docker compose up -d.

Results

The reduction in MTTR from 4-6 hours to under 45 minutes translated directly to improved SLA compliance and significantly reduced client-facing downtime incidents.