A Green-Screen Flow That Fails One Run in Twenty
A logistics team automates a 3270 terminal emulator used for customs declarations. Text positions shift depending on shipment type, so the developer originally placed UI elements using image recognition and on-screen coordinates. The flow now fails roughly once in twenty runs with 'UI element not found', always mid-transaction - leaving a declaration half-created and blocking later steps. Re-running the flow from the start then creates a second, duplicate declaration. Separately, an auditor noticed the terminal password typed as a plain-text variable and visible in a failure screenshot attached to an email notification. The team wants to fix this without redesigning the whole flow.
1. How do you make the terminal screen interactions reliable, given that coordinates and images drift?
Read the worked answer
Diagnose first: coordinate and image-based targeting fails whenever the emulator's layout or font rendering changes, which is exactly what shipment-type variation does. Move to targeting by content that is stable regardless of position.
- Prefer text-based targeting over image recognition: select the field by its label text ('Consignee', 'Declaration No.') or by the terminal field identifier, using anchors - capture the label element and pick the input field relative to it.
- Where the emulator exposes accessibility or window text, use UI element selectors built from window/control attributes rather than pixels; retain image recognition only as an explicit fallback for elements with no text identity.
- Add a Wait for element / wait for text on screen step before every interaction instead of a fixed sleep, so slow screen paint no longer causes a race. Fixed delays are the usual hidden cause of intermittent failures.
- Use the selector's built-in test/repair capability and store selectors reproducibly: if you must keep an image, capture it at the resolution and theme used on the runtime machine, and pin the machine's display settings.
Reasoning: whichever targeting you choose, the selector should key on business-meaningful, layout-independent attributes, because those are the only things that survive a format change. Where the screen genuinely has no stable identity, that step is a candidate for a manual checkpoint rather than a clever selector.
2. The half-created declaration followed by a duplicate on re-run is the real business problem. Design the error handling and recovery so this cannot happen.
Read the worked answer
Three layers, in order of importance:
- Idempotency at the target. Before creating, the flow searches the customs system for the shipment reference. If a declaration already exists, it either resumes completion or skips. Without this, no amount of retry logic is safe, because a retry after a partial write always risks duplication.
- Block-level error handling. Wrap each logical unit (open session → navigate → fill → submit → confirm) in an On block error container. Set action-level On error behaviour deliberately: Retry with increasing delays for transient screen-paint failures, Continue only where the next step genuinely checks state, and Stop with error where continuing would corrupt data. Avoid a blanket 'continue on error' - that is what turns one failure into a half-created record.
- Compensating action and state. If a create succeeds but submit fails, the handler must either complete the record deliberately or delete/void it, then write a status back to the orchestrating queue row (
Failed, step name, terminal text captured). The cloud flow then decides whether to retry the item, not the whole batch.
Also: capture the emulator screen text at the moment of failure into the error record - for terminal automation, that screen dump is usually the only usable diagnostic.
3. How do you stop the terminal password from appearing in screenshots, logs, and notifications?
Read the worked answer
Use the Sensitive text variable type for anything that carries a secret - passwords, tokens, connection strings. Values held in sensitive text variables are masked in logs, in the flow designer, and in the variable pane, which is the direct fix for the audit finding.
The stronger fix is to not store the credential in the flow at all:
- Keep the password in Azure Key Vault. A cloud flow retrieves it with the Key Vault Get secret action and passes it into the desktop flow as an input variable. The desktop flow only ever holds it as a sensitive text variable.
- Restrict who can read the secret with Key Vault access policies or RBAC, and rotate it centrally without editing the flow.
Separately, review the failure notification: if it attaches a screenshot of the terminal session, that screenshot bypasses variable masking entirely. Either suppress screenshots on the login block or configure the flow to skip screenshot capture for that region, and send failure notifications to a channel rather than emailing images broadly.
Note that if the connector needed for Key Vault is blocked by a data loss prevention policy in that environment, the fallback is the machine's own credential store plus sensitive text variables - decide this with the platform admin before designing the flow.