Source-based ISP selection for a vendor API client
An Ubuntu 22.04 gateway has ens3 198.51.100.10/24 (ISP A) and ens4 203.0.113.20/24 (ISP B). LAN 10.20.0.0/24 uses the gateway. All LAN traffic must leave via ISP A by default, but the vendor API client 10.20.5.50 must use ISP B because the vendor accepts only connections from 203.0.113.20. The current default route is via 198.51.100.1. An operator added a second default route via 203.0.113.1, and now outbound LAN sessions fail intermittently. ip route get 192.0.2.80 from 10.20.5.50 still shows ISP A. nftables has a postrouting masquerade rule for oifname 'ens3' only. DNS resolves, but the vendor sees connection attempts from 198.51.100.10 and blocks them. The gateway uses Netplan with systemd-networkd. You must keep the default path for all other LAN hosts on ISP A and make the vendor client's traffic persistent across reboot.
1. Diagnose why adding a second default route caused intermittent failures, and why the vendor client still exits via ISP A.
Read the worked answer
A second default route in the main table creates route selection ambiguity; Linux may prefer one default or alternate behavior depending on metrics and nexthop state. That is the wrong tool for one client. The vendor client also has only its LAN route in the main table, and the masquerade rule matches only ens3, so its packets either follow the main default via ISP A or, if the second default wins, leave with source 198.51.100.10 or with an ISP B source that does not match the vendor whitelist.
Confirm the actual decision path with ip rule show, ip route show table main, ip route show table all, ip route get 192.0.2.80 from 10.20.5.50, and nft list ruleset. Use conntrack -L -s 10.20.5.50 and tcpdump -ni any host 192.0.2.80 during a test. The correct diagnosis is missing policy routing plus missing source NAT on ens4, not a DNS or firewall problem. Remove the second main default before making the per-client policy.
2. Give the policy-routing and NAT design that sends only 10.20.5.50 out ISP B, including verification commands.
Read the worked answer
Use a dedicated routing table for the client and an nftables postrouting rule for source NAT. Runtime form: ip rule add from 10.20.5.50 table 200 priority 100, ip route add default via 203.0.113.1 dev ens4 table 200, and an nftables rule such as ip saddr 10.20.5.50 oifname 'ens4' masquerade in the postrouting chain. Keep the main default via 198.51.100.1. The masquerade is essential: the vendor must see 203.0.113.20 and must be able to return traffic to a public address.
Verify with ip rule show, ip route get 192.0.2.80 from 10.20.5.50, nft list ruleset, and conntrack -L -s 10.20.5.50. Test from the client itself if possible using curl --interface 10.20.5.50 https://192.0.2.80/ and capture with tcpdump -ni ens4 host 192.0.2.80. A successful test shows the SYN leaving ens4 with source 203.0.113.20 and return traffic matching conntrack.
3. How would you make the design persistent with Netplan and avoid breaking the other LAN hosts?
Read the worked answer
Define the default route only in the main table on ens3. Put the ISP B default in table 200 and attach the routing policy to ens4 in Netplan, for example a route with to: default, via: 203.0.113.1, table: 200, and routing-policy: [{from: 10.20.5.50, table: 200}]. Keep ens3 as the only main default with a lower metric. Run netplan try before netplan apply so a mistake does not lock you out, then re-check ip rule and ip route get after reboot testing.
The other LAN hosts remain on ISP A because no rule matches their source addresses. Keep the nftables NAT rule scoped to ip saddr 10.20.5.50 and oifname 'ens4'; do not add a blanket masquerade on ens4, which could hide source addresses for traffic that should use ISP A. Also confirm ip_forward=1 and that the forward chain accepts LAN-to-WAN traffic. Document the vendor whitelist dependency so a future route change does not silently move this client back to ISP A.