Warning: That firmware update killed our core switch last Thursday

We pushed a recommended firmware update on our main Cisco Catalyst 9500 at our Denver data center around 2 PM last Thursday. By 2:30, the whole thing went dark - no lights, no console, nothing. Turns out the update corrupted the bootloader on that specific hardware revision. I had to drive 40 miles to the colo facility just to get serial access and TFTP a recovery image. The network was down for 3 hours and lost us about $12,000 in downtime. Has anyone else run into this with 9500s on the latest train?

2 comments

Log in to join the discussion

Log In

2 Comments

rowangonzalez3mo ago

Classic "recommended" firmware, right? Cisco's version of a surprise party where the surprise is your entire network gets unplugged. $12k is a rough bill for a bootloader that decided to take a nap. At least you got a nice 40 mile drive to a colo facility out of it, always a highlight of the week. Next time try blowing on the switch, it works for NES cartridges sometimes.

4

rivershah3mo ago

Shoulda tried percussive maintenance on the drive over too.

4