Three Hostnames, One VIP: Chains, Protocols and Re-Encryption
One virtual server at 203.0.113.10:443 terminates TLS for three hostnames using SNI: www.example.com and shop.example.com (both covered by a wildcard *.example.com certificate) and api.partner.example (a separate certificate from a public CA). The client SSL profile carries the wildcard as its default certificate; the partner certificate is selected by an SNI-based Client SSL profile on a matching Server Name. Last week a change removed TLS 1.0/1.1 and set the cipher string to DEFAULT:!3DES:!RC4. Since then two groups report failures. First, about 12% of Android 6/7 app users get 'certificate not trusted' from api.partner.example, although the same hostname validates in desktop Chrome. Second, an internal Java 8 batch client pinned to api.partner.example fails with handshake_failure and offers only TLSv1. The backend is re-encrypted: pool members present certificates from an internal CA, and the server SSL profile is set to Server Certificate: require with a trusted certificate bundle containing only Internal-Root-CA. Yesterday a different team provisioned a new pool member using Internal-Root-CA-2024, and roughly one request in eight to the API now returns 502.
1. Diagnose the Android 'certificate not trusted' failures. What do you check first, and what is the likely fix?
Read the worked answer
This is not a TLS-version symptom - the client rejects the certificate, not the protocol. Desktop Chrome on the same hostname succeeds, which points at the certificate chain rather than the leaf's validity dates or hostname coverage.
Most likely cause: the partner certificate is being served as a leaf with no intermediate CA. Chrome either has the intermediate cached from another site or fetches it via the AIA extension embedded in the certificate; Android's trust manager generally does not chase AIA URLs, so the chain cannot be built and the handshake fails. The wildcard is a red herring - *.example.com does not cover api.partner.example, so it is not the certificate in play for those clients.
Prove it before changing anything: openssl s_client -connect 203.0.113.10:443 -servername api.partner.example -showcerts and count what comes back. A single certificate with no issuer chain in the output confirms it. Cross-check the SNI-selected Client SSL profile: the certificate key chain should hold the leaf plus the intermediate.
Fix: rebuild the key chain with the intermediate included and attach it to that profile, then reload the profile rather than restarting. Re-test with the same Android version, since 'not trusted' can also come from an expired root on old devices - check the leaf's notAfter while you are in there.
2. The Java batch client fails with handshake_failure. What is happening, and what are your options?
Read the worked answer
handshake_failure from a client whose protocol list is hard-coded to TLSv1 is a version mismatch, not a cipher problem, and no amount of cipher-string editing will fix it. Removing TLS 1.0/1.1 from the profile means the BIG-IP cannot produce a ServerHello for a ClientHello that offers only TLSv1, so it sends a fatal alert.
Confirm from the wire, not from the error string: capture the handshake (tcpdump -s0 -i 0.0:nnn -w /var/tmp/hs.pcap host <java-host>) and read the ClientHello supported-versions field, or check /var/log/ltm for the SSL alert. You should see the client offering TLS 1.0 and the BIG-IP rejecting it.
Options, in order of preference:
- Have the client team fix their protocol list. This is the only option that preserves the security posture you just adopted.
- Give that client a dedicated virtual server - separate hostname or port - with its own Client SSL profile that permits TLSv1 with an approved cipher, plus a documented expiration date and a compensating control at the application or network layer.
- Accept the outage, only if the job is genuinely non-critical.
What you must not do is re-enable TLS 1.0 on the shared profile. That silently downgrades every browser and mobile app hitting the VIP in order to serve one batch job.
3. Explain the one-in-eight 502s and how you would fix them without weakening backend trust.
Read the worked answer
The server SSL profile is the suspect, and the arithmetic supports it: the API pool appears to have eight members, and one of them is the newly provisioned node. The profile requires a valid server certificate and trusts only Internal-Root-CA; the new member presents a chain signed by Internal-Root-CA-2024, so the BIG-IP rejects it during the re-encrypted handshake. The pool member is not 'down' from a monitor perspective if the monitor does not validate the chain, so the virtual server keeps sending it traffic and the client sees a 502.
Prove it from the BIG-IP rather than from a laptop: check /var/log/ltm for an SSL alert against that member's IP, and compare the issuer of the member's chain with the contents of the trusted certificate bundle. A test openssl s_client run from the BIG-IP shell against member:port with the same CA bundle reproduces the failure.
Fix: merge the new root - and its intermediate if any - into the trusted bundle, re-import, and retest the member. Do not fall back to Server Certificate: ignore to make the 502 disappear; that removes the check that the backend really is your application, which is the whole reason re-encryption is validated. Also record who owns each CA in the bundle: an over-broad trust store is how internal interception goes unnoticed.