The Migration Rule Nobody Removed
Northwind Logistics runs a single R81.20 management server that manages a two-member ClusterXL gateway at HQ, a standalone gateway in a Singapore branch, and a Maestro security group in the data center. The rule base is built from three ordered layers: Corporate, Data Center and Cleanup. Six months ago an engineer added a rule at the very top of the Corporate layer that accepts any source to any destination on TCP 8080 and 8443 for a migration project, and it was never removed. Last week an inline layer for partner API traffic was added under a rule in the Data Center layer, so that partner access to the API server would be dropped except from two approved sources. A penetration test has now shown that partner scanning of TCP 8443 is accepted. In the same change window, policy installation to the Maestro group started taking 22 minutes and twice returned "Installation failed on some of the cluster members", even though no gateway object was modified and no routing changed.
1. Explain why the partner traffic on 8443 is still accepted despite the new inline layer, and describe how you would confirm the cause from evidence rather than assumption.
Read the worked answer
Reasoning. Layers are evaluated in the order they are attached to the gateway's policy, not in the order rules appear to matter to the designer. Corporate sits above Data Center, so a packet is matched against Corporate first. The leftover any-to-any rule for TCP 8080/8443 is the very first rule of the first layer, so an inbound partner packet to the API object on 8443 matches it and is accepted. Evaluation never reaches the Data Center layer or its inline layer. The inline layer only governs traffic that the parent rule sends into it; it cannot retroactively override a rule that already accepted the connection higher up.
Confirmation. Filter SmartLog by destination = API object and service = https/8443, add the columns Matched Rule and Policy Layer, and open the log card to see the full layer path. Note the rule UID and check its hit count over the last 30 days. If the rule has no log track, use fw monitor on the cluster to capture the accept and then re-check after enabling track. Do not delete the rule on the spot: confirm with the project owner that nothing still depends on it, then disable it rather than delete it, and schedule the install in a window.
2. Before retrying the Maestro installation, what would you check to explain both the 22-minute install time and the intermittent failure on some cluster members?
Read the worked answer
Separate the two symptoms. Install duration is a function of rule/object count, number of target gateways, log volume and verification time. Intermittent failure on some members is almost always a per-member condition, not a rule-base problem, so treat them independently.
Per-member checks.
- Confirm every member of the security group runs the same software version and jumbo hotfix level; a single member on a different build fails the policy verification step.
- Check the installation status per member in SmartConsole and read
$FWDIR/log/fwd.elgon the failing member around the install timestamp. - Verify SIC trust and reachability to each member, plus free space on
$FWDIRand/var/log. - For Maestro, verify the security group members are all reported as up and that no recently added member was attached without the matching image.
Duration checks. Look for a large number of objects/rules, many gateways with logging enabled, and verification warnings. Confirm on the management server that the install is actually waiting on the gateways and not on the management database.
Action. Fix the outlier member first, then reinstall; retrying repeatedly only produces more partially installed members and more noise in the audit trail.
3. Write a short change plan that closes the exposure and reduces install time, respecting the constraint that a second long outage in the same week is unacceptable.
Read the worked answer
Step 1 – Immediate, low-risk closure (same window). Disable the 8443/8080 any-any rule at the top of Corporate and install policy to the HQ cluster only. Scope the install to the affected gateway rather than "Install on all", which keeps the 22-minute Maestro job out of the critical path. Validate with an external scan of the API address.
Step 2 – Correct placement. Confirm that the partner inline layer is reached by the intended parent rule and that no earlier layer can match partner traffic. Add explicit drop logging on the partner rule for 24 hours to prove coverage.
Step 3 – Reduce install time structurally.
- Remove or consolidate unused objects and rules; enable the unused-objects report.
- Reduce logging on high-hit rules that are not required for auditing.
- Split the Maestro group's install from the HQ/branch installs so a single slow target does not delay everything.
Step 4 – Prevent recurrence. Require an expiry date and owner on any temporary any-any rule, and add a monthly review of rules with no expiry. Record the change in the audit trail and notify the penetration test owner of the remediation date.