100-490 RSTECH: Border Router in ROMMON After an Image Swap

Border Router in ROMMON After an Image Swap

An engineer upgraded the border router at a distribution center: an ISR 4331 with a single 8 GB flash and no USB drive on site. A new IOS XE image was copied from an HQ TFTP server, boot system flash bootflash:<new-image>.bin was added, show bootvar was checked, the previous image was deleted to free 780 MB, and a reload was scheduled for 02:00. The router came back to rommon 1 >. The site has 180 users. The only out-of-band path is a 4G LTE console server at 10 Mb/s down and 5 Mb/s up, and nobody is on site for six hours. The router's management port is cabled to an access switch that reaches HQ over a separate LTE backup router, so TFTP is reachable once the router has an address. The replacement image is 690 MB and the bootloader (BOOTLDR) in ROM is two releases older than the image. The previous image's file name is still known, and the startup-config in NVRAM is intact and was last saved three weeks ago.

1. The device landed in ROMMON instead of booting the new image. List the causes consistent with the evidence, rank them by likelihood, and name the single command that most quickly narrows it down.

Read the worked answer

Ranked hypotheses

The boot statement was never saved to startup-config. show bootvar shows the running configuration's boot list. If copy running-config startup-config was never issued after adding boot system, NVRAM still points at the deleted old image, the loader finds nothing bootable, and the box drops to ROM monitor. This fits perfectly: the old file was deliberately deleted and the config on NVRAM is three weeks stale.
The TFTP copy was truncated or failed silently. The engineer freed 780 MB and then pulled a 690 MB file over a lossy path; a partial file with a plausible name will not load.
Image/platform or bootloader incompatibility. The BOOTLDR is two releases behind; some IOS XE releases require a minimum ROMMON/BOOTLDR. This would normally print a message before failing, and it is the least likely of the three.

Fastest discriminator

At the ROM monitor prompt, run dir bootflash:. It answers both questions at once: does the new file exist, and is it 690 MB or something smaller? A missing or short file confirms hypothesis 1 or 2 immediately; a full-size file that refuses to load points at hypothesis 3.

2. Give the full recovery sequence from ROM monitor to a verified running configuration, including the ROMMON variables, transfer, verification, and boot commands. Show the arithmetic behind the transfer time.

Read the worked answer

1. Give the router an address and a TFTP target (ROM monitor variable syntax, no set prefix):

IP_ADDRESS=10.20.30.5
IP_SUBNET_MASK=255.255.255.0
DEFAULT_GATEWAY=10.20.30.1
TFTP_SERVER=10.1.1.20
TFTP_FILE=isr4300-universalk9.<new>.bin
TFTP_VERIFY=warning
TFTP_RETRY_COUNT=10
tftpdnld

2. Sanity-check the transfer before booting. dir bootflash: should show exactly 690 MB, and verify bootflash:<file> (or the platform's md5/verify equivalent) should return OK. Do not skip this - a second bad boot costs another outage.

3. Preserve the environment and boot manually. sync writes the ROMMON variables, then boot bootflash:<file>. Booting manually proves the image works before you commit it.

4. Once IOS is up:

conf t
 boot system flash bootflash:<new>.bin
 config-register 0x2102
end
copy running-config startup-config
show bootvar

Then confirm show bootvar reports the new image and that startup-config contains it, and only then reload to prove the change survives.

Transfer arithmetic: 690 MB is roughly 5,520 Mb. At an optimistic 5 Mb/s upstream that is about 1,100 seconds (~18 minutes); TFTP's block-and-ack behaviour, a 4G link, and ROM monitor's slow client make 30-45 minutes realistic. Expect the customers to be down for the better part of an hour, not twenty minutes.

3. Name two controls that would have prevented this outage and are worth adding now, and explain for each what failure mode it removes.

Read the worked answer

1. Keep two bootable images until the new one has actually booted. The old image should have been deleted only after the new image had booted cleanly and survived one scheduled reload. This removes the 'nothing left to fall back to' failure: if the new boot variable is wrong or the file is bad, the loader finds the old file and the site stays up. The 780 MB saving can wait a week.

2. Make the boot change committed and verified, not just configured. After any boot system change: copy running-config startup-config, then show bootvar, then confirm the same output appears in show startup-config | include boot. Optionally keep a configuration archive. This removes the exact failure that happened here - a running-config change that evaporated at reload - and it is the single cheapest habit change available.

Worth adding alongside those two: checksum verification of every image before the old one is touched, and a rule that image deletions happen only in a window with confirmed out-of-band access. The LTE console server did its job; the process, not the tooling, was the weak link. Also confirm the BOOTLDR/ROMMON level before the next upgrade so the incompatibility path is ruled out in advance rather than discovered at 02:00.