Guest Shell Alerting on Memory-Constrained Branch Routers
A retail chain runs 400 C1111 routers on IOS XE 17.9. A Python script in the guest shell watches syslog for %LINK-3-UPDOWN messages and posts an alert to an internal webhook. Six weeks after rollout, alerts stop on 70 routers following reloads; on 15 others the script is running but uses 95 percent of the guest-shell memory and the router drops OSPF hellos during busy periods. The script writes its checkpoint file to /bootflash/guest-share, and a recent image upgrade wiped that path. The team also has a legacy EEM Tcl applet that restarts the guest shell when CPU exceeds 70 percent, causing loops. The operations lead wants to keep on-box automation but is open to moving logic off-box. The routers have intermittent WAN links and cannot depend on a central server for local detection.
1. Explain why alerts stop after reload and why the guest shell consumes so much memory and CPU. Distinguish startup, persistence, and resource-limit causes.
Read the worked answer
Alerts stopping after reload is an enablement and persistence problem, not a Python bug. Guest shell is not guaranteed to be enabled and running after every reload unless the configuration explicitly enables it and the application is configured to start. The checkpoint path is also suspect: a path under bootflash can be overwritten or removed by image upgrade, so the script may restart with no state. Verify with show guestshell and the platform's app-hosting state after a reload, and move state to a documented persistent location that survives upgrade.
Memory pressure comes from running a general-purpose interpreter plus syslog buffering on a branch control plane. Tailing syslog in Python, holding lines in memory, and importing large libraries can consume the guest shell's memory cgroup. At 95 percent, the container is near its limit; the router's OSPF hellos suffer because guest shell shares control-plane CPU and scheduling with the network stack.
The EEM applet makes it worse. Restarting guest shell when CPU exceeds 70 percent creates a loop: restart causes CPU spike, the applet fires again, and the container never stabilizes. The applet has no rate limit or max-run guard. Disable it before tuning anything else, then measure baseline CPU and memory with the guest shell idle and under load.
2. Choose an architecture for local link-flap detection under intermittent WAN and tight resources. Compare guest shell Python, EEM, and off-box telemetry, and justify the choice.
Read the worked answer
Use a hybrid: lightweight local detection with EEM, durable local queueing, and off-box forwarding when the WAN is available. Do not make the central server the detection path, because the WAN is intermittent and the requirement is local detection.
EEM is the best fit for the trigger. An EEM applet can match the syslog pattern natively, run with a maxrun limit, write a small record to persistent storage, and optionally invoke a short action. It has a much smaller footprint than a Python interpreter and is designed for control-plane events. Use a rate limit so a flapping interface does not generate a storm, and avoid restarting guest shell from EEM.
Guest shell Python remains useful only for enrichment, such as formatting a JSON payload or doing a local HTTP post when the WAN is up. Keep it small, avoid large imports, cap memory, and run it only when EEM hands it work. Do not let it tail syslog continuously.
Off-box telemetry is for visibility, not local detection. Model-driven telemetry or syslog to a collector gives the NOC rich data, but a disconnected branch cannot depend on it. Buffer locally and forward later. This meets the local-detection constraint while keeping the branch control plane stable.
3. Propose deployment guardrails so the next on-box automation release does not repeat the reload, memory, and loop failures. Include persistence, resource limits, and rollback.
Read the worked answer
Startup and persistence: explicitly enable guest shell in the saved configuration, configure the application to start automatically, and verify after a reload in a canary. Store state and scripts under a documented persistent path and test an image upgrade, not just a reload. If the platform wipes a path on upgrade, the deployment must recreate it.
Resource limits: set guest shell memory and CPU limits below the platform's default if the branch is memory-constrained, monitor the container's usage, and alert before 85 percent. Avoid running syslog tailers and large Python libraries. Set EEM maxrun and rate limits; never let an EEM applet restart the container based on a raw CPU threshold. Use a watchdog that checks application health with hysteresis and a maximum restart count.
Rollout and rollback: deploy to five canaries per hardware and software version, run for a full busy hour, and compare OSPF adjacency stability and control-plane CPU to baseline. Keep a known-good script and configuration archive; if the canary regresses, disable the app and restore the prior config. Stage the change outside the peak window and keep console or out-of-band access. A green show guestshell is not sufficient evidence; verify the end-to-end alert path, including a simulated link flap and a WAN outage with delayed forwarding.