Accepting an OPC UA integration FAT that never tested the real server
A municipal water utility integrates Glacier Fabricant with a 2004-vintage DCS using OPC UA at a 240 ML/d treatment plant. The interface carries 4,000 tags at a 1 s update; 900 of them feed the plant's regulatory flow and turbidity reporting. At factory acceptance test, 8% of monitored tags update slower than 5 s, and the historian shows out-of-order timestamps on 300 tags after a 40 s network interruption. The integrator's OPC UA client uses one subscription with a 1,000 ms publishing interval and all 4,000 monitored items. The DCS server's documented limit is 500 monitored items per session, beyond which it silently drops to a 10 s sampling rate. The utility's network team has also placed the Glacier server on the corporate VLAN "for remote support". The FAT was executed against a simulated server that emits all 4,000 tags in sequence with no contention. SAT is in six weeks, and the regulatory report requires continuous data with a documented maximum gap of 60 s.
1. Do you sign the FAT as passed? Justify your decision against the stated requirements.
Read the worked answer
No. Sign the FAT record as executed with deviations, but do not sign it as passed.
Why the evidence does not support acceptance.
- The requirement is 4,000 tags at 1 s. Eight percent of tags - roughly 320 - miss a 5 s threshold, which is five times the requirement. There is no acceptance criterion in which that is a pass.
- Out-of-order timestamps on 300 tags breach data integrity for the 900 regulatory tags. The 60 s maximum gap requirement cannot be demonstrated when a 40 s outage produces corrupted sequence on recovery, because the recovery behaviour is precisely what determines the real gap.
- The test was run against a simulator with no contention and a sequential emit pattern. It never exercised the constraint that causes the failure: the DCS server's 500-monitored-items-per-session limit. A FAT that avoids the known constraint is not a FAT; it is a demonstration.
What I would sign. The factual test record with each test result and its observed value, a deviation list, and a statement that FAT is incomplete pending re-test. Withholding the signature is not obstruction - the alternative is discovering this at SAT with regulatory reporting already scheduled.
One nuance to note in the record. The 10 s fallback is a documented server behaviour, not a defect. The integrator's design is the defect: it configured a single session that exceeds a published limit and then tested against a server that does not have that limit.
2. Write the re-test plan that would actually expose the integration limits, and state the pass criteria.
Read the worked answer
The plan must test the real server, the real network path and the recovery case.
1. Baseline characterisation (real DCS, non-production window). Connect the Glacier client to the actual DCS server. Confirm the 500-item session limit by stepping item counts: 250, 500, 600, 1,000. Record actual sampling interval reported by the server at each step. Pass: no silent degradation.
2. Redesigned subscription architecture validation. Split into at least 8 sessions of 500 items maximum, or use a gateway that supports the load. For each session set publishing interval, sampling interval, queue size and deadband explicitly, and record them. Pass: every one of the 4,000 tags updates within 1 s under steady state, measured per tag over 30 minutes, reported as a latency distribution not a mean.
3. Priority segregation. Put the 900 regulatory tags in dedicated sessions separate from the balance, then re-run test 2 while inducing load on the non-regulatory sessions. Pass: regulatory tags unaffected.
4. Outage and recovery. Interrupt the link for 40 s and 90 s, plus a DCS restart. Verify buffered values are backfilled with original source timestamps, timestamps are monotonic per tag after recovery, and the measured gap is recorded. Pass: gap is stated and within the 60 s allowance, with a documented worst case that QA and the regulator can see.
5. Network path. Repeat the latency test from the final production VLAN, not from a lab bench.
6. End-to-end reporting. Produce the regulatory report from the integrated system and reconcile it against the DCS historian for the same period. Pass: values match and no gap exceeds 60 s.
Every test needs a named witness, a recorded dataset retained with the FAT documentation, and a fail action. A test with no fail action is a formality.
3. What integration and network design changes do you specify before SAT, and how do you justify them?
Read the worked answer
Three changes, each traceable to an observed failure or a stated constraint.
1. Subscription architecture. Replace the single 4,000-item subscription with multiple sessions inside the 500-item server limit, with explicit publishing and sampling intervals per session, queue sizes sized to the publishing interval, and deadband configured per tag class. This is not tuning - a client that ignores a published server limit is a design error, and the 10 s fallback is the server behaving as documented.
2. Network placement. Move the Glacier server off the corporate VLAN into a dedicated zone with a firewall conduit to the DCS zone, allowing only the OPC UA port and only the specific client/server pair. The corporate VLAN placement is both a cybersecurity exposure (the plant process data path shares a broadcast domain with office traffic and remote support) and a performance risk, and it makes the SAT latency results unpredictable. Justify it as zone/conduit design, and test latency from the final location.
3. Time synchronisation. Specify a single time source for the DCS, Glacier and historian, and test for monotonicity. Out-of-order timestamps on recovery usually indicate either NTP disagreement or client-side re-stamping; both are cheap to fix now and expensive to explain to a regulator.
Additionally, add a monitored-tag health view in Glacier Fabricant that alarms when any session's actual publishing interval exceeds its configured value. The failure mode here was silent degradation, and a silent failure will recur unless something measures it.
Justification to the utility. Every change maps to a requirement they wrote: 1 s updates, 60 s maximum gap, data integrity for reporting. Frame the re-test as protecting the regulatory report, not as schedule slip.