Troubleshooting: C9200L-48P-4X Fuji code Locked Up
- A C9200L had locked up. All switchports were dead/no light and the Fiber modules were a solid amber lit color. All ports had hosts on the other end that were still alive and sending electrical signal to their ports. I attempted to power cycle by pulling the power cables and reseating the redundant power supplies. After two attempts the switch was still locked up with fans staying on high after POST should have cleared.
Troubleshooting Information
- Do any lights at all illuminate? I see the switchports don’t, but is any activity seen on the front panel?
Sfp ports stayed solid amber – switchports none lit, no activity when reseating connections - Do the PSUs appear to power on?
Yes both PSUs appear to power on, switch fans kick on during POST and stay on full speed - Has the device ever powered on?
Yes device was pulled from production - What version of code was running (if known/applicable)?
Fuji – 16.9.4
While attempting to swap the 9200L with a loaner switch I ran into the following warning messages. NOTE: The fiber and SFP modules were being reseated into different members of the stack until the RMA could come in.


No Big Deal
- I had never ran into the Duplicate GBIC error before. While attempting to do some research on this I ran into bug reports of this occuring on 3850s.
Solution/Work Around
- Remove the old switch member
- no errdisable detect cause gbic-invalid
- reseat connections
- admin shut/no shut the module ports.

- I figured that removing the stack member, reseating the connections would be enough but for some reason the ports were still errdisabled.

- I had to shut/no shut the ports twice after reseating each connection. Once I did this the ports moved out of errdisable.

SWITCH-NAME(config)# *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed SWITCH-NAME(config)# *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2 SWITCH-NAME(config)# *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed SWITCH-NAME(config)# *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3 SWITCH-NAME(config)#Warning: [1 51] is dup of [3 50] SWITCH-NAME(config)#end SWITCH-NAME#sh logg Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled) No Active Message Discriminator. No Inactive Message Discriminator.Console logging: level emergencies, 0 messages logged, xml disabled, filtering disabled Monitor logging: level debugging, 173 messages logged, xml disabled, filtering disabled Logging to: vty2(7) Buffer logging: level debugging, 46694 messages logged, xml disabled, filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
No active filter modules.Trap logging: level informational, 46254 message lines logged Logging Source-Interface: VRF Name:
Log Buffer (4096 bytes): port Gi1/0/26 and port Gi1/0/25 *Feb 24 14:25:45.184: %SYS-6-LOGOUT: User pete has exited tty session 2(10.10.16.40) *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52] *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] SWITCH-NAME#ter le 0 SWITCH-NAME#sh logg *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50] *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state SWITCH-NAME#conf t Enter configuration commands, one per line. End with CNTL/Z. SWITCH-NAME(config)#no errdisable detect cause gbic-invalid SWITCH-NAME(config)#exi SWITCH-NAME# *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed SWITCH-NAME# *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2 SWITCH-NAME#Warning: [2 50] is dup of [3 52] SWITCH-NAME#sh logg *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52] *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50] *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50] *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5) *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5) *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52] SWITCH-NAME# *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed SWITCH-NAME# *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2 SWITCH-NAME#Warning: [1 50] is dup of [3 51] SWITCH-NAME#show sw SWITCH-NAME#show switch Switch/Stack Mac Address : 10b3.d582.9880 - Local Mac Address Mac persistency wait time: Indefinite H/W Current Switch# Role Mac Address Priority Version State 1 Standby 4c71.0d81.xxxx 1 V01 Ready 2 Member 7c21.0e62.xxxx 1 V01 Ready 3 Member 0000.0000.xxxx 0 V01 Removed *4 Active 10b3.d582.xxxx 1 V01 Ready SWITCH-NAME#conf t Enter configuration commands, one per line. End with CNTL/Z. SWITCH-NAME(config)#no switch 3 provision SWITCH-NAME(config)# *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed SWITCH-NAME(config)# *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1 SWITCH-NAME(config)#Warning: [1 49] is dup of [3 49] SWITCH-NAME(config)#int ra te 1/1/1 - 2 SWITCH-NAME(config-if-range)#no shut SWITCH-NAME(config-if-range)#do sh logg Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled) No Active Message Discriminator. No Inactive Message Discriminator.Console logging: level emergencies, 0 messages logged, xml disabled, filtering disabled Monitor logging: level debugging, 183 messages logged, xml disabled, filtering disabled Logging to: vty2(17) Buffer logging: level debugging, 46704 messages logged, xml disabled, filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
No active filter modules.Trap logging: level informational, 46261 message lines logged Logging Source-Interface: VRF Name:
Log Buffer (4096 bytes): PDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50] *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50] *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5) *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5) *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52] *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51] *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49] SWITCH-NAME(config-if-range)#do sh clock *15:33:51.262 UTC Wed Feb 24 2021 SWITCH-NAME(config-if-range)#shut SWITCH-NAME(config-if-range)#no shut SWITCH-NAME(config-if-range)# *Feb 24 15:34:28.246: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to up *Feb 24 15:34:28.259: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/2, changed state to up SWITCH-NAME(config-if-range)# *Feb 24 15:34:31.578: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/1, changed state to up *Feb 24 15:34:31.757: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/2, changed state to up SWITCH-NAME(config-if-range)#exi SWITCH-NAME(config)#int te 2/1/2 SWITCH-NAME(config-if)#shut SWITCH-NAME(config-if)#no shut SWITCH-NAME(config-if)# *Feb 24 15:34:54.208: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/1/2, changed state to up SWITCH-NAME(config-if)# *Feb 24 15:34:57.425: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/2, changed state to up
Leave a Reply