Troubleshooting: C9200L-48P-4X Fuji code Locked Up

Categories Cisco, Switch
  • A C9200L had locked up. All switchports were dead/no light and the Fiber modules were a solid amber lit color. All ports had hosts on the other end that were still alive and sending electrical signal to their ports. I attempted to power cycle by pulling the power cables and reseating the redundant power supplies. After two attempts the switch was still locked up with fans staying on high after POST should have cleared.

Troubleshooting Information

  • Do any lights at all illuminate? I see the switchports don’t, but is any activity seen on the front panel?
    Sfp ports stayed solid amber – switchports none lit, no activity when reseating connections
  • Do the PSUs appear to power on?
    Yes both PSUs appear to power on, switch fans kick on during POST and stay on full speed
  • Has the device ever powered on?
    Yes device was pulled from production
  • What version of code was running (if known/applicable)?
    Fuji – 16.9.4

While attempting to swap the 9200L with a loaner switch I ran into the following warning messages. NOTE: The fiber and SFP modules were being reseated into different members of the stack until the RMA could come in.

No Big Deal

  • I had never ran into the Duplicate GBIC error before. While attempting to do some research on this I ran into bug reports of this occuring on 3850s.

Solution/Work Around

  • Remove the old switch member
  • no errdisable detect cause gbic-invalid
  • reseat connections
  • admin shut/no shut the module ports.










  • I figured that removing the stack member, reseating the connections would be enough but for some reason the ports were still errdisabled.










  • I had to shut/no shut the ports twice after reseating each connection. Once I did this the ports moved out of errdisable.










 SWITCH-NAME(config)#
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 SWITCH-NAME(config)#
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 SWITCH-NAME(config)#
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 SWITCH-NAME(config)#
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3
 SWITCH-NAME(config)#Warning: [1 51] is dup of [3 50]
 SWITCH-NAME(config)#end
 SWITCH-NAME#sh logg
 Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
 No Active Message Discriminator.
 No Inactive Message Discriminator.
 Console logging: level emergencies, 0 messages logged, xml disabled,                  filtering disabled Monitor logging: level debugging, 173 messages logged, xml disabled,                  filtering disabled     Logging to: vty2(7) Buffer logging:  level debugging, 46694 messages logged, xml disabled,                 filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
 No active filter modules.
 Trap logging: level informational, 46254 message lines logged     Logging Source-Interface:       VRF Name:
 Log Buffer (4096 bytes):
  port Gi1/0/26 and port Gi1/0/25
 *Feb 24 14:25:45.184: %SYS-6-LOGOUT: User pete has exited tty session 2(10.10.16.40)
 *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state
 *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 SWITCH-NAME#ter le 0
 SWITCH-NAME#sh logg
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 SWITCH-NAME#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 SWITCH-NAME(config)#no errdisable detect cause gbic-invalid
 SWITCH-NAME(config)#exi
 SWITCH-NAME#
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 SWITCH-NAME#
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2
 SWITCH-NAME#Warning: [2 50] is dup of [3 52]
 SWITCH-NAME#sh logg
 *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state
 *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state
 *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 SWITCH-NAME#
 *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 SWITCH-NAME#
 *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 SWITCH-NAME#Warning: [1 50] is dup of [3 51]
 SWITCH-NAME#show sw
 SWITCH-NAME#show switch
 Switch/Stack Mac Address : 10b3.d582.9880 - Local Mac Address
 Mac persistency wait time: Indefinite
                                              H/W   Current
 Switch#   Role    Mac Address     Priority Version  State
 1       Standby  4c71.0d81.xxxx     1      V01     Ready
  2       Member   7c21.0e62.xxxx     1      V01     Ready
  3       Member   0000.0000.xxxx     0      V01     Removed
 *4       Active   10b3.d582.xxxx     1      V01     Ready
 SWITCH-NAME#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 SWITCH-NAME(config)#no switch 3 provision
 SWITCH-NAME(config)#
 *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 SWITCH-NAME(config)#
 *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1
 SWITCH-NAME(config)#Warning: [1 49] is dup of [3 49]
 SWITCH-NAME(config)#int ra te 1/1/1 - 2
 SWITCH-NAME(config-if-range)#no shut
 SWITCH-NAME(config-if-range)#do sh logg
 Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
 No Active Message Discriminator.
 No Inactive Message Discriminator.
 Console logging: level emergencies, 0 messages logged, xml disabled,                  filtering disabled Monitor logging: level debugging, 183 messages logged, xml disabled,                  filtering disabled     Logging to: vty2(17) Buffer logging:  level debugging, 46704 messages logged, xml disabled,                 filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
 No active filter modules.
 Trap logging: level informational, 46261 message lines logged     Logging Source-Interface:       VRF Name:
 Log Buffer (4096 bytes):
 PDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state
 *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 SWITCH-NAME(config-if-range)#do sh clock
 *15:33:51.262 UTC Wed Feb 24 2021
 SWITCH-NAME(config-if-range)#shut
 SWITCH-NAME(config-if-range)#no shut
 SWITCH-NAME(config-if-range)#
 *Feb 24 15:34:28.246: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to up
 *Feb 24 15:34:28.259: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/2, changed state to up
 SWITCH-NAME(config-if-range)#
 *Feb 24 15:34:31.578: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/1, changed state to up
 *Feb 24 15:34:31.757: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/2, changed state to up
 SWITCH-NAME(config-if-range)#exi
 SWITCH-NAME(config)#int te 2/1/2
 SWITCH-NAME(config-if)#shut
 SWITCH-NAME(config-if)#no shut
 SWITCH-NAME(config-if)#
 *Feb 24 15:34:54.208: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/1/2, changed state to up
 SWITCH-NAME(config-if)#
 *Feb 24 15:34:57.425: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/2, changed state to up