Mass migration automation: porting 524 domains in under 4 hours

Manual migration doesn't scale

Migrating hundreds of active domains between servers is a bottleneck when done by hand. Manual work introduces human error, unpredictable downtime, and overhead. The goal was an automated pipeline that handles DNS sync across multiple providers, server provisioning, file transfers, and SSL issuance with zero dropped payloads.

A 4-stage idempotent pipeline

To handle the scale without hitting API rate limits or server I/O bottlenecks, I designed a 4-stage pipeline in Node.js using p-limit for controlled concurrency. A custom StateManager tracks the migration lifecycle and makes the system strictly idempotent. If a process fails, the pipeline resumes exactly where it left off without duplicate executions.

The 4 stages

The pipeline splits into 4 idempotent stages:

Tuned to each bottleneck

DNS propagation runs at a concurrency of 20 because its rate-limit threshold is higher. File uploads are strictly capped at 5 concurrent streams to prevent server overload.

Execution sequence

flowchart LR
    Pipeline["Node.js Pipeline"] --> StateManager["State Manager"]
    StateManager --> Read["Read Pending Tasks"]
    
    Read --> S1["Stage 1: DNS Sync<br>(Max 20)"]
    Read --> S2["Stage 2: Provisioning<br>(Max 10)"]
    Read --> S3["Stage 3: File Transfer<br>(Max 5)"]
    Read --> S4["Stage 4: SSL Issuance<br>(Max 10)"]
    
    S1 --> APIs["Cloudflare / Namecheap / Spaceship"]
    S2 --> Hestia["HestiaCP API"]
    S3 --> SFTP["SFTP Stream"]
    S4 --> ACME["Let's Encrypt ACME"]

Each stage writes its completion to the StateManager before advancing. That makes the pipeline resumable at any point.

Engineering decisions

• The StateManager writes to disk atomically, so network failures or unexpected timeouts during large batch runs can't corrupt the target server state.
• Concurrency limits are decoupled by operation type: network, I/O, or CPU. The script maximizes throughput without triggering API rate limits (HTTP 429) from providers like Namecheap or bringing down the target HestiaCP node.
• The pipeline runs either in full or as isolated steps (pnpm step:dns, pnpm step:ssl) for debugging and strict control.

Results

• A multi-day manual migration condensed into under 4 hours of autonomous execution.
• 524 live domains ported with zero configuration mismatch.
• Complete audit trails via state mapping for every migrated domain.
• All domains remained accessible throughout the migration window.