Hybrid Connectivity for a Latency-Sensitive Logiostics Platform

Latency

The VPN measures 25–60 ms with retransmits, so it cannot hold a 30 ms target; packet loss on the internet path is what breaks the latency-sensitive calls. The interconnect claims 15 ms, which leaves headroom. Treat 'stated' as unverified until a pilot measures it - 700 km of fibre at typical speeds should land well under the target, but the SLA and actual jitter need to be seen, not assumed.

Volume - the part that changes the answer

18 TB in three hours is 18,432 GB over 10,800 seconds, about 1.7 GB/s, or roughly 13.7 Gbps. Neither a 1 Gbps interconnect nor a 400 Mbps VPN can move that in the window. No port purchase fixes it; the ETL job's transfer model has to change. The twice-monthly timeouts are a symptom of an impossible window, not of a bad line.

Architecture

• Primary: dedicated interconnect for dispatcher traffic and the 150 Mbps telemetry stream, with two ports or a link aggregation group if the provider supports it.
• Backup: the existing VPN, carried on BGP as a lower-preference path. Sized for the business-critical subset, not for full volume.
• ETL: redesigned to incremental or in-region processing (see the next task); keep bulk transfer off the latency-critical link during business hours.

Sequence

1. Upgrade or stabilise the VPN now so the 12-week lead time is not a blocker; move latency-tolerant traffic first.
2. Order the interconnect and run a measured pilot - latency, jitter, loss - before committing traffic.
3. Cut dispatcher and telemetry traffic to the interconnect, then verify against the 30 ms target.
4. Redesign ETL, then retire bulk transfer from the WAN entirely.

Reduce or relocate the transfer before buying capacity.

• Move the compute to the data. Run the ETL in the cloud region and read from a replicated source, or push only changed records instead of a full 18 TB. Most '18 TB nightly' jobs are full re-pushes of data that barely changed.
• Delta or change-data-capture. Send only inserts and updates. Requires change tracking at the source and a reconciliation job to prove nothing was lost.
• Ingest at the edge. Warehouse telemetry originates outside the colocation. Send it directly to cloud object storage rather than routing it through the colo and back out over the WAN.
• Compress and deduplicate. Structured telemetry often compresses by several times; that is free bandwidth.
• One-time bulk history via a transfer appliance or physical shipment, then incremental after cutover.

Cost comparison

• 18,432 GB at $0.02/GB is roughly $370 per night, around $11,000 a month.
• The same volume at $0.09/GB is roughly $1,650 per night, around $50,000 a month.
• So even after the interconnect, full nightly transfer carries a five-figure recurring bill and still cannot finish in three hours.

Trade-off

The real cost is engineering time and freshness guarantees: incremental sync needs change tracking, a reconciliation process, and a new failure mode to monitor. That is a one-off cost against a permanent reduction in bandwidth, egress, and window risk. Buy the interconnect for latency, not for the ETL.

• Asymmetric routing. If outbound traffic leaves over the interconnect and return traffic comes back through the VPN, stateful firewalls and NAT drop the flow. Make path selection symmetric: consistent BGP local preference, AS-path prepending on the backup, and matching route tables on both sides.
• Failover capacity. The VPN backup at 400 Mbps cannot carry the interconnect's 1 Gbps. Define what runs in degraded mode and confirm the business accepts it; do not discover it during an outage.
• MTU and fragmentation. IPsec overhead on the VPN path causes large packets to drop while the interconnect passes them. Set MSS clamping for tunnel traffic.
• Long-lived sessions. Warehouse telemetry devices hold persistent connections. On a path change those break; require reconnect logic with jitter so 40 sites do not stampede the backup link.
• DNS. Keep private zone resolution inside the cloud network and avoid making on-premises DNS a single point of failure; pair health checks with failover records.
• BGP hygiene. Private ASN allocation, advertised prefix limits, no default route leaking in either direction, and alerting on session flaps.
• Test it. Run a quarterly failover drill - kill the BGP session and a tunnel, and measure how long traffic takes to re-establish and what breaks.