Steps for recovering a UC500 from ROMMON

First thing, we want to remember that the UC500 appliances do not have a dedicated onboard, non-removable flash. The persistent storage resides on one removable CompactFlash (CF) card (two, in the case of the UC560) in front-panel slots (see Introduction to the Cisco UC500).

Formatting the CF card

Suppose we want to start with a fresh CF card, with no files in it. We can use an USB to CF adapter like this one:

Or we can also format the CF card from within the UC500:

Note that the only file available on the CF card was the IOS file needed to boot the appliance. Without this file, and since the CF is the only persistent storage, the UC500 cannot boot and enters ROMMON:

The most logical and easy thing to do, is to just copy the IOS file again to the CF card (i.e. from a PC by using a USB-to-CF adapter). That should work and after copying it to the CF, inserting the CF on the UC500 and entering the boot command at the ROMMON prompt should boot the IOS again.

Formatting the CF card from a PC

Now let's complicate things a bit. Instead of formatting the CF card using the format command on the UC500, we will format the CF card in FAT from Windows 10.

Best practice is to never format a CF card to be used on a Cisco device, using another device (i.e Windows PC).

Then, we'll copy the IOS file again to the CF card.

Everything looks good. But, interestingly enough, now the CF card is not recognized by the UC500. The boot command does nothing since the CF card is not recognized:

We are getting some weird open(): Open Error = -66 error followed by boot: cannot open "flash:". It seems like the format of the CF done on Windows corrupted the CF in some way, that it is no more recognized by the UC500.

What really happened here is, the format done by Windows changed the partition scheme from supper-floppy to MBR (however it left unchanged the allocation unit size to 4KB, default).

An allocation unit aka cluster, is the smallest unit of disk space that a file system can use to store a file.

The MBR is a partitioning scheme used to define how the storage device (CF in this case) is divided into partitions. When formatting the device with an MBR, it means the device is partitioned. That means the system expects to find one or more partitions on the device.

Devices like the Cisco UC500 use ROMMON (ROM Monitor). ROMMON is not designed to handle partitioned media. It expects the storage device to be non-partitioned - i.e, it wants the device to be a single volume with no partitions, like a super-floppy disk.

When using MBR to partition the CF card, ROMMON cannot recognize ir properly, because ROMMON doesn't understand MBR or partition tables. Instead, it looks for the entire CF card as one single volume and gets confused by the partition table created by MBR.

For reference, to check the allocation unit size (aka. cluster size) we can use chkdsk:

Recovering from corrupted CF: method #1

So now that we have discovered the problem, we have two ways to recover from it. The easiest way is method 1, by which we will use the tool Rufus on the Windows PC to reformat the CF according to what the ROMMON would expect.

Note the options we are configuring on Rufus:

Partition scheme: Super Floppy Disk
Filesystem: FAT
Cluster size: 4096 bytes (default, and also the cluster size when using the format command on IOS)
Quick format

Again, a supper-floppy disk refers to a disk format that has no partition table (MBR), the entire disk is treated as a single volume and the file system (e.g FAT) is directly applied to the entire disk, not just to a partition within the disk.

Now the CF is recognized again on the UC500. Also, there would be no problem transfering files from Windows using the USB-to-CF adapter.

For reference, at the left there is info about the CF filesystem just formatted from Rufus with the options above. To the right, the filesystem details right after using the format IOS command. Note every value is essentially the same.

Recovering from corrupted CF: method #2

The second method, while longer, is a common procedure for recovering control of Cisco devices. This method uses a TFTP server where the IOS file will be located and booted from.

We will use a Windows PC with a TFTP server on it and the needed IOS file on the TFTP root directory. The file to be used is the latest IOS available for the UC500 - uc500-advipservicesk9-mz.151-4.M6

We will connect this PC to the WAN port of the UC500.

Then we will configure the following IP settings on the PC (a sample network just for the transfer of the file):

IP address: 192.168.1.100
Netmask: 255.255.255.0
Gateway: 192.168.1.254

I will use SolarWinds TFTP Server as my TFTP server. Be sure to check whether the TFTP service is running on port UDP 69, and configure a Windows Firewall exception if necessary.

To replicate the problem, we have reformatted the CF card on Windows (MBR), and we are into ROMMON again without the "corrupted" flash:

Now we will configure the UC500 as a TFTP client and start the TFTP download process from the TFTP server. We will be using an IP address on the same sample network, which will be the default gateway we configured on the PC (192.168.1.254).

With the set command, we can see the actual variables set at the ROMMON mode.

We will assign the IP settings as follows:

Note we are using the PC IP address as the default gateway on the UC500.

Now onto the TFTP configuration commands:

We have configured the TFTP_SERVER variable pointing to the PC IP address and the TFTP_FILE pointing to the exact file name - case sensitive). To begin the download we will use the tftpdnld command.

Here is the full output after executing the command:

rommon 6 > IP_ADDRESS=192.168.1.254
rommon 7 > IP_SUBNET_MASK=255.255.255.0
rommon 8 > DEFAULT_GATEWAY=192.168.1.100
rommon 9 > TFTP_SERVER=192.168.1.100
rommon 10 > TFTP_FILE=uc500-advipservicesk9-mz.151-4.M6
rommon 11 > TFTP_CHECKSUM=0
rommon 12 > tftpdnld

   IP_ADDRESS: 192.168.1.254
      IP_SUBNET_MASK: 255.255.255.0
     DEFAULT_GATEWAY: 192.168.1.100
  TFTP_SERVER: 192.168.1.100
    TFTP_FILE: uc500-advipservicesk9-mz.151-4.M6
 TFTP_MACADDR: a4:93:4c:6e:81:c0
 TFTP_VERBOSE: Progress
    TFTP_RETRY_COUNT: 18
 TFTP_TIMEOUT: 7200
       TFTP_CHECKSUM: No
       FE_SPEED_MODE: Auto Detect

Invoke this command for disaster recovery only.
WARNING: all existing data in all partitions on flash will be lost!
Do you wish to continue? y/n:  [n]:  y
....
Receiving uc500-advipservicesk9-mz.151-4.M6 from 192.168.1.100 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
File reception completed.
Copying file uc500-advipservicesk9-mz.151-4.M6 to flash:.
program load complete, entry point: 0x80015000, size: 0xb700

Format: Drive communication & 1st Sector Write OK...
Writing Monlib sectors.
...............................................................................................
Monlib write complete

Format: All system sectors written. OK...
Format: Operation completed successfully.

Format of flash: complete
program load complete, entry point: 0x80015000, size: 0xb700

rommon 13 >

From the output we can see that the tftpdnld forces a flash format (prompted). After the process is completed, the file is present on the flash: