Cisco - // A Networking Blog

Troubleshooting: C9200L-48P-4X Fuji code Locked Up

Cisco, Switch03/01/2021

A C9200L had locked up. All switchports were dead/no light and the Fiber modules were a solid amber lit color. All ports had hosts on the other end that were still alive and sending electrical signal to their ports. I attempted to power cycle by pulling the power cables and reseating the redundant power supplies. After two attempts the switch was still locked up with fans staying on high after POST should have cleared.

Troubleshooting Information

Do any lights at all illuminate? I see the switchports don’t, but is any activity seen on the front panel?
Sfp ports stayed solid amber – switchports none lit, no activity when reseating connections
Do the PSUs appear to power on?
Yes both PSUs appear to power on, switch fans kick on during POST and stay on full speed
Has the device ever powered on?
Yes device was pulled from production
What version of code was running (if known/applicable)?
Fuji – 16.9.4

While attempting to swap the 9200L with a loaner switch I ran into the following warning messages. NOTE: The fiber and SFP modules were being reseated into different members of the stack until the RMA could come in.

No Big Deal

I had never ran into the Duplicate GBIC error before. While attempting to do some research on this I ran into bug reports of this occuring on 3850s.

Solution/Work Around

Remove the old switch member
no errdisable detect cause gbic-invalid
reseat connections
admin shut/no shut the module ports.

I figured that removing the stack member, reseating the connections would be enough but for some reason the ports were still errdisabled.

I had to shut/no shut the ports twice after reseating each connection. Once I did this the ports moved out of errdisable.

 SWITCH-NAME(config)#
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 SWITCH-NAME(config)#
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 SWITCH-NAME(config)#
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 SWITCH-NAME(config)#
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3
 SWITCH-NAME(config)#Warning: [1 51] is dup of [3 50]
 SWITCH-NAME(config)#end
 SWITCH-NAME#sh logg
 Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
 No Active Message Discriminator.
 No Inactive Message Discriminator.
 Console logging: level emergencies, 0 messages logged, xml disabled,                  filtering disabled Monitor logging: level debugging, 173 messages logged, xml disabled,                  filtering disabled     Logging to: vty2(7) Buffer logging:  level debugging, 46694 messages logged, xml disabled,                 filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
 No active filter modules.
 Trap logging: level informational, 46254 message lines logged     Logging Source-Interface:       VRF Name:
 Log Buffer (4096 bytes):
  port Gi1/0/26 and port Gi1/0/25
 *Feb 24 14:25:45.184: %SYS-6-LOGOUT: User pete has exited tty session 2(10.10.16.40)
 *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state
 *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 SWITCH-NAME#ter le 0
 SWITCH-NAME#sh logg
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 SWITCH-NAME#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 SWITCH-NAME(config)#no errdisable detect cause gbic-invalid
 SWITCH-NAME(config)#exi
 SWITCH-NAME#
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 SWITCH-NAME#
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2
 SWITCH-NAME#Warning: [2 50] is dup of [3 52]
 SWITCH-NAME#sh logg
 *Feb 24 14:46:24.069: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 14:46:24.069: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te2/1/2, putting Te2/1/2 in err-disable state
 *Feb 24 14:46:49.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state
 *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 SWITCH-NAME#
 *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 SWITCH-NAME#
 *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 SWITCH-NAME#Warning: [1 50] is dup of [3 51]
 SWITCH-NAME#show sw
 SWITCH-NAME#show switch
 Switch/Stack Mac Address : 10b3.d582.9880 - Local Mac Address
 Mac persistency wait time: Indefinite
                                              H/W   Current
 Switch#   Role    Mac Address     Priority Version  State
 1       Standby  4c71.0d81.xxxx     1      V01     Ready
  2       Member   7c21.0e62.xxxx     1      V01     Ready
  3       Member   0000.0000.xxxx     0      V01     Removed
 *4       Active   10b3.d582.xxxx     1      V01     Ready
 SWITCH-NAME#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 SWITCH-NAME(config)#no switch 3 provision
 SWITCH-NAME(config)#
 *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 SWITCH-NAME(config)#
 *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1
 SWITCH-NAME(config)#Warning: [1 49] is dup of [3 49]
 SWITCH-NAME(config)#int ra te 1/1/1 - 2
 SWITCH-NAME(config-if-range)#no shut
 SWITCH-NAME(config-if-range)#do sh logg
 Syslog logging: enabled (0 messages dropped, 7 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
 No Active Message Discriminator.
 No Inactive Message Discriminator.
 Console logging: level emergencies, 0 messages logged, xml disabled,                  filtering disabled Monitor logging: level debugging, 183 messages logged, xml disabled,                  filtering disabled     Logging to: vty2(17) Buffer logging:  level debugging, 46704 messages logged, xml disabled,                 filtering disabled Exception Logging: size (4096 bytes) Count and timestamp logging messages: disabled File logging: disabled Persistent logging: disabled
 No active filter modules.
 Trap logging: level informational, 46261 message lines logged     Logging Source-Interface:       VRF Name:
 Log Buffer (4096 bytes):
 PDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:50.168: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:46:58.360: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:46:58.360: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state
 *Feb 24 14:47:00.408: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:47:02.420: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:28.930: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:29.942: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:32.982: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:35.463: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:38.714: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:39.922: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 14:50:40.721: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:52.774: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:53.788: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to down
 *Feb 24 14:50:56.717: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:50:58.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/42, changed state to up
 *Feb 24 14:51:02.833: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 *Feb 24 14:51:48.227: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:51:48.227: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in err-disable state
 *Feb 24 14:52:18.181: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 14:52:38.420: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 14:53:07.578: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 14:53:07.578: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/3, putting Te1/1/3 in err-disable state
 *Feb 24 14:56:02.489: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: pete] [Source: 192.168.1.5] [localport: 23] at 14:56:02 UTC Wed Feb 24 2021
 *Feb 24 15:00:25.568: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:00:54.982: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2
 *Feb 24 15:02:52.913: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/3 removed
 *Feb 24 15:04:47.672: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3Warning: [1 51] is dup of [3 50]
 *Feb 24 15:06:07.071: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:11:22.731: %SYS-5-CONFIG_I: Configured from console by pete on vty0 (192.168.1.5)
 *Feb 24 15:12:05.166: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te2/1/2 removed
 *Feb 24 15:12:32.313: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te2/1/2Warning: [2 50] is dup of [3 52]
 *Feb 24 15:14:09.249: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/2 removed
 *Feb 24 15:14:16.391: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/2Warning: [1 50] is dup of [3 51]
 *Feb 24 15:19:14.899: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/1 removed
 *Feb 24 15:19:24.716: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1Warning: [1 49] is dup of [3 49]
 SWITCH-NAME(config-if-range)#do sh clock
 *15:33:51.262 UTC Wed Feb 24 2021
 SWITCH-NAME(config-if-range)#shut
 SWITCH-NAME(config-if-range)#no shut
 SWITCH-NAME(config-if-range)#
 *Feb 24 15:34:28.246: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to up
 *Feb 24 15:34:28.259: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/2, changed state to up
 SWITCH-NAME(config-if-range)#
 *Feb 24 15:34:31.578: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/1, changed state to up
 *Feb 24 15:34:31.757: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/2, changed state to up
 SWITCH-NAME(config-if-range)#exi
 SWITCH-NAME(config)#int te 2/1/2
 SWITCH-NAME(config-if)#shut
 SWITCH-NAME(config-if)#no shut
 SWITCH-NAME(config-if)#
 *Feb 24 15:34:54.208: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/1/2, changed state to up
 SWITCH-NAME(config-if)#
 *Feb 24 15:34:57.425: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/2, changed state to up

Firepower Threat Defense – Interfaces and Zones

Cisco, fmc02/21/2021

FTD is a true zone based firewall.
Security zones are collections of interfaces or sub-interfaces.
Policy rules can apply to source and/or destination zones.
This is NOT an ASA – We do NOT use security levels.

Interface Types

Management Interface
Diagnostic Interface
Physical Interface
Etherchannel
Redundant
Routed
Bridged

WAN/Outside

Our WAN/Outside will be our untrusted traffic.
We will use DHCP to get an IP address leased from the internet service provider.

LAN/Inside

Our LAN interface will be for our internal (inside) traffic.
We will use a static IP address and this port will be a routed interface that is connected to a routed port on our Cisco switch.
You can have different designs and use sub interfaces with dot1q trunking but I prefer to use as much layer 3 that I can.

Routing

We are using OSPF for our routing protocol.
We will advertise all of our inside networks and keep everything in the backbone.
We are going to be a normal router, we don’t need stub or not-so-stubby – we are an internal router.
We don’t need to worry about range or virtual links.

Firepower Threat Defense – Logical Objects

Cisco, fmc02/21/2021

Network Address Objects

A network object represents one or more IP addresses. Network objects are used in various places, including access control policies, network variables, intrusion rules, identity rules, network discovery rules, event searches, reports, and so on.
It is a good idea to create objects for each VLAN
If possible creating objects for individual devices may be required.
It is a good idea to be as granular as possible because this will allow for flexibility and scalability to your rules offering more insights on what is happening within your network.

After the individual objects are created we will create a network group object that will encompass all of our internal network.
This group object will be used throughout our policy creation.

Ports

Port objects or groups represent different protocols. You can use port objects and groups in various places in the systems web interface, including access control policies, identity rules, network discovery rules, port variables, and event searches.
Just like our network objects we can create layer4 tcp/udp objects that will be used in the same manner.
You can also add service ports in the ACP when creating access control policies but they will not have a descriptive name.

Interfaces and Zones

Interface objects segment your network to help you manage and classify traffic flow. An interface object simply groups interfaces. These groups may span multiple devices; you can also configure multiple interface objects on a single device.
FTD is a zone based system creating security zones allows for easier management.

Application Filter

Application filters help you perform application control by organizing applications according to basic characteristics: type, risk, business relevance, category, and tags.

This can be used as a way to block web sites if you don’t have a license for URL filtering (This feature is included with the base license.
Add the selected filters.
You can choose the applications you want selected if you don’t want to choose all.
Add to Rule.
Save the new application filter.

Variable Sets

Variables represent values commonly used in intrusion rules to identify source and destination IP addresses and ports.You can also use variables in intrusion policies to represent IP addresses in rule suppressions, adaptive profile updates, and dynamic rule states.
Now we will edit our HOME_NET variable from ‘any’ to our network group object we created earlier.
This will help give a better insight and more security.

Update the new group object into the Included Networks list to remove the ‘any’ object.

After saving the new object we can see it has been applied and our new variable set has been created.

FMC Health Policies and Alerts

Cisco, fmc02/21/2021

Health policies and alerts are an important part of managing devices.
At the top right corner of FMC we can see that their is currently an alert which is being generated because I am using a demo license.
The FMC is making a ‘call’ every 5 minutes and Cisco is replying that I’m not worthy!
Let’s disable that since this is a lab environment and you would not want to do this for production but I’m annoyed with seeing red.

The monitor gives us a break down of the health monitors that are hitting counters.

Health Policies

Now we go into the policy section of the health tab and create a new policy.
This policy will be applied to our FTD and FMC.

Disable the Smart licensing status monitor by selecting off and then saving the configuration.

SNMP Alert

Next we will continue to create our Syslog and SNMP alerts.

Syslog Alerts

That’s All Folks!

The SNMP/SYSLOG alerts should now be live once we have selected all modules we want, the severity at which they are defined and the alerts we choose.

Firepower Management Console System Configuration

Cisco, fmc, Security02/20/2021

Syslog

Setting up a syslog server prevents allowing another user on the FMC to delete the logs. This also keeps a backup of your logs.
Set the ‘send audit to syslog’ as enabled
enter in the sys log serves IP address for the host.
Change the faility to SYSLOG
Set severity to what you would like.
Test the syslog server and then Save once verified it’s working.

You will be greeted with a Sucess message after the configuration change has took affect.

Login Banner

Create a login banner with whatever login greeting you would like.
It is a good idea to display an authorization warning for the login banner or any message you would like users to see once logged into the FMC.
Save the login banner once completed.

Change Reconciliation

Change reconciliation allows a reported to be generated every x hours:minutes to provide a history of what configurations have changed.

Email Notification

Enter in your information as needed.
The from address can be whatever you like all other information will need to be legitimate

HTTPS Server Certificate

Creating an HTTPS certificate.

Management Interfaces

Hostname of the device
Domain name the device is in.
DNS servers
Remote management port
Don’t forget to hit save!

Remote Storage Device

SNMP

Setup snmpv2
Create an ACL pointing to ISE or whatever you’re using for SNMP.

vWLC with Hotspot Guest Access using ISE 2.7

Cisco, ISE02/19/2021

Hotspots are a portal where users can access an open SSID. Generally, they will need to accept an Acceptable Use Policy before being granted access to the internet. You can have different scenarios than just this lab scenario.
Log into the vWLC. Click the security tab at the top.

Click the New button to add a new AAA server.
Enter the IP address of the ISE server, port number is 1812, and that Support for COA is checked. *** Change of Authorization is a feature that allows a RADIUS server to adjust an active client session. ***
Create a Shared Secret and make note of it as ISE will need to be configured with the same secret. Click Apply.

Next click Accounting from the Security/AAA menu on the left. Hit New and enter the required information.

Next we will log into ISE and configure the WLC as a network device
Go to Work Centers, then Network Resources.

Click Add and fill out the WLC information. Check Radius Auth. Settings and be sure to fill out the Shared Secret we filled out earlier in the WLC.

After you save the network device you can verify it has been added by checking the Network Devices section.

Configuring the Guest SSID

Log into your WLC and click the WLANs tab. Choose Create New from the drop down box and click Go.

Enter a profile name and SSID.

Select Status Enabled, and the correct interface for your guest traffic. *** NOTE: My screen shot doesn’t show the Guest SSID as being enabled ***

Next click the Security tab.
Change Layer 2 Security to None, and check MAC Filtering.

Click AAA Servers, and change the Authentication and Authorization servers to the ISE server via the drop down boxes.

Click the Advanced tab.
Check Allow AAA Override.
Under NAC change the drop down to ISE NAC.
Uncheck Flex Connect Local Switching if enabled.
Check DHCP/HTTP profiling under Radius Client Profiling.

Next we have to create a few ACLs. One for Web Auth Redirect that will allow DNS and traffic to ISE and another ACL for restricting guest access.

You can verify your ACLs have been added to the vWLC from the Access control list section.

ISE Policies

Our policy goals will be:

redirect users who connect to the Guest network to a web portal.
Once the AUP has been accepted they will get a new policy applied to them restricting their access to internet only via the ACL we created earlier.

Click Results and and go to Authorization Profiles.
Click Add to create a new profile.
Give the policy a descriptive name and description.

Scroll down to the Common Tasks and check Web Redirection.
Select Hotspot from the drop down.
Enter WEB_AUTH_REDIRECT as the ACL and the value will be the Hotspot guest portal.
Click Submit.

Click Add again, enter a new name and description. This policy will apply the guest restriction ACL we created on the WLC.
Scroll down into the Common Tasks and find Airespace ACL, enter the name Guest_ACL
Click Submit.

Now, go to Work Centers, Guest Access, Policy Sets.

Create a new policy set

Add a new profile above the one we just created
This will be for applying the Guest ACL for the user once going through the portal. Conditions will be Wireless_MAB, IdentityGroup = GuestEndpoints, and Guest_Flow. Result will be the Guest_Access policy we created which applies the ACL we created on the WLC.

Lastly, use whatever wireless device to verify you can connect to the new SSID.
Save all configurations and backup if needed.

Cisco Switch Configuration for ISE

Cisco, ISE, Switch02/15/2021

Switch Configuration

Example configuration used

conf t
 radius server ISE_RADIUS
 address ipv4 10.0.30.40 auth-port 1645 acct-port 1646
 key Temp1234!@#$
 exit
 aaa group server radius ISE
 server name ISE_RADIUS
 ip radius source-interface vlan 30
 exit
 aaa authentication dot1x default group ISE
 aaa authorization network default group ISE
 aaa authorization exec default group ISE local if-authenticated
 aaa accounting update perdiodic 3
 aaa accounting dot1x default start-stop group ISE
 aaa server radius dynamic-author
 client 10.0.30.40 server-key Temp1234!@#$
 radius-server attribute 6 on-for-login-auth
 radius-server attribute 8 include-in-access-req
 radius-server attribute 25 access-request include
 end

Explanation

Explain commands

ISE Configuration

Add in the name, description, IP address, Device profile, Model name, Software version.
Enable the RADIUS Authentication Settings and input the shared secret then submit to add the switch in.

To verify after you click on submit you will now see the device listed under the Network Devices section.

Authorization Profile

Policy Set

Verification

On the switch you can issue:

show cdp neighbor
show authentication session
show aaa server

On ISE GUI you can review the live session and logs under the RADIUS section in Operations:

Troubleshooting

No authentication sessions are showing on the network device.

Run an authentication test using the network device and review the logs in ISE.
Verify the aaa server configuration

test aaa group radius test-user test-password new-code

In ISE GUI:

Verify NTP is matching for Logs

Verify the NTP server matches on all devices.
Cisco Switch:

ISE Server:

Cisco ISE 2.7 on ESXi

Cisco, ESXi, ISE02/14/2021

What is ISE?

Cisco Identity Services Engine (ISE) is a solution to streamline security policy management and reduce operating costs. You can see users and devices controlling access across wired, wireless, and VPN connections to the corporate network.

Image

System Requirements

Clock speed: 2.0 GHz or faster
Number of CPU cores: 4 CPU cores
16 GB memory
300 GB Storage
1 NIC interface required (two or more NICs are recommended; six NICs are supported).
***Cisco ISE supports E1000 and VMXNET3 adapters.***

License

The Cisco ISE image comes with a 90-day evaluation license already installed, so you can begin testing all Cisco ISE services when the installation and initial configuration is complete.

Transfer the .OVA file to your datastore on the ESXi server and follow the installation steps.
Once the upload and import have completed start the virtual machine.

Initial Setup

Boot the image.
To begin configuration enter ‘setup’ as the username and then follow along the prompts for initial setup.
***This process can take 30 minutes to multiple hours depending on your hardware resources***

Verify the Installation

After ISE starts login with your new credentials and begin to verify the installation.

show application
show application status ise

Web GUI

Initial Login

CLI Admin Vs GUI Admin

The username and password that you configure when using the Cisco ISE setup program are intended to be used for administrative access to the Cisco ISE CLI and the Cisco ISE web interface.
You can initially access the Cisco ISE web interface by using the CLI-admin user’s username and password that you defined during the setup process. There is no default username and password for a web-based admin.
The CLI-admin user is copied to the Cisco ISE web-based admin user database.
Only the first CLI-admin user is copied as the web-based admin user.
You should keep the CLI- and web-based admin user stores synchronized, so that you can use the same username and password for both admin roles.
The Cisco ISE CLI-admin user has different rights and capabilities than the Cisco ISE web-based admin user and can perform other administrative tasks.

Create a CLI Admin

Cisco ISE allows you to create additional CLI-admin user accounts other than the one you created during the setup process. To protect the CLI-admin user credentials, create the minimum number of CLI-admin users needed to access the Cisco ISE CLI.You can add the CLI-admin user by using the following command in the configuration mode:

username <username> password [plain/hash] <password> role admin

Please note the password complexity and requirements.

Create a Web-Based Admin

For first-time web-based access to Cisco ISE system, the administrator username and password is the same as the CLI-based access that you configured during setup.

Choose Administration > System > Admin Access > Administrators > Admin Users.
Choose Add > Create an Admin User.
Enter the name, password, admin group, and the other required details.
Click Submit.

Reset a Disabled Password Due to Administrator Lockout

An administrator can enter an incorrect password enough times to disable the account. The minimum and default number of attempts is five.
Use these instructions to reset the administrator user interface password with the application reset-passwd ise command in the Cisco ISE CLI. It does not affect the CLI password of the administrator. After you successfully reset the administrator password, the credentials are immediately active and you can log in without having to reboot the system.
Cisco ISE adds a log entry in the Administrator Logins window. The navigation path for this window is Operations > Reports > Reports > Audit > Administrator Logins. The credentials for that administrator ID is suspended until you reset the password associated with that administrator ID.

Joining a 2702i Cisco AP to Virtual WLC on 8.10

Cisco, Wireless02/13/2021

Issue

From the WLC GUI the AP is stuck in downloading state.

While consoled into the access point we can see the following errors in the console output.

<!--  /* Font Definitions */  @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:3 0 0 0 1 0;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-469750017 -1073732485 9 0 511 0;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536869121 64767 1 0 415 0;}  /* Style Definitions */  p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin-top:0in; margin-right:0in; margin-bottom:8.0pt; margin-left:0in; line-height:107%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} .MsoPapDefault {mso-style-type:export-only; margin-bottom:8.0pt; line-height:107%;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} --> 
 ERROR: Image is not a valid IOS image archive.
 Download image failed, notify controller!!! From:7.6.100.0 to 0.0.0.0, FailureCode:3  

<!--  /* Font Definitions */  @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:3 0 0 0 1 0;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-469750017 -1073732485 9 0 511 0;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536869121 64767 1 0 415 0;}  /* Style Definitions */  p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin-top:0in; margin-right:0in; margin-bottom:8.0pt; margin-left:0in; line-height:107%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} .MsoPapDefault {mso-style-type:export-only; margin-bottom:8.0pt; line-height:107%;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} --> 
 *Feb 13 15:48:34.219: %CAPWAP-5-SENDJOIN: sending Join Request to 10.0.30.50perform archive download capwap:/ap3g2 tar file
 *Feb 13 15:48:34.223: %CAPWAP-6-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.
 *Feb 13 15:48:34.223: Loading file /ap3g2...

Resolution

The root cause of the issue is 8.5 and later versions need to use c3700 / cx700 to join ME. If your AP version is lower than 8.5 you need to upgrade to ap3g2 and then upgrade to c3700.
After that, you can successfully join the WLC.

TFTP Server

For my TFTP server I am using a Synology NAS and running the TFTP Service:
How to setup Synology NAS TFTP

Cisco AP Process

Power the AP off and hold down the mode button.
While the mode button is pressed down hold it and power the AP back on.
Continue to hold the mode button down until the AP’s light goes solid red.
Once the AP is red lit, let go and console into the AP using your console cable.

Commands

set IP_ADDR <DEVICE IP ADDRESS>
set NETMASK <SUBNET>
set DEFAULT_GATEWAY <GATEWAY IP>
tftp_init
ether_init
flash_init
tar -xtract tftp://<TFTP SERVER IP>/<.TAR FILE> flash:
set BOOT flash:/<.TAR FILE>

Example

set IP_ADDR 10.0.30.53
 set NETMASK 255.255.255.0
 set DEFAULT_ROUTER 10.0.30.1
 tftp_init
 ether_init
 flash_init
 tar -xtract tftp://10.0.30.14/ap3g2-rcvk9w8-tar.153-3.JF10.tar flash:
 set BOOT flash:/ap3g2-rcvk9w8-mx

My TFTP server is 10.0.30.14
The subnet is a /24
The gateway is 10.0.30.1
initialize tftp, ether and flash services on the AP
Extract the tar file in the TFTP servers root directory (the file is named ap3g2-rcvk9w8-tar.153-3.JF10.tar ) and we are extracting the file into our flash directory.
***DURING THE TAR EXTRACT PROCESS CONTINUE TO HIT SPACE BAR EVERY 1 – 2 SECONDS*** if not than you can receive a premature error.
Set the boot file location and let the AP sit for 5 minutes after finished booting new image you should be able to see it on the WLC and it will be in a Registered State.

WLC GUI – AFTERMATH

Now we can update the AP name, location, set a static IP for the device.

Hardcode the controllers.

The Access Point has now been successfully upgraded and is now registered to the Virutal wireless LAN controller.

Automating Cisco IOS using NSO 5.3 on Ubuntu Linux 20.04 – Part 4 (NSO Services)

Automation, Cisco, Devnet, NSO01/18/2021

A network service refers to a collection of configurations across devices on a network.
NSO Services are used to create instances of services to deploy across your network.
You can define any set of configuration templates and sets of variables for those templates.
NSO will keep track of the service instances and only update or remove services.
Services can be a customer, tenant, site id, or anything you want to define.
Services provide assurance and verification that prove the intended outcome on the network before applying the configuration into a production environment. Services can be used along side other applications or workflow engines.

Creating the Service Package

NSO uses a built in package management system to handle NEDs and custom service packages.
ncs-make-package is a CLI tool that will auto generate a skeleton structure of files needed for creating the service package.
The tree is a tiny, cross-platform command-line program used to recursively list or display the content of a directory in a tree-like format. It outputs the directory paths and files in each sub-directory and a summary of a total number of sub-directories and files.

Review the contents of the loopback-service.yang and loopback-service-template.xml files using cat to output to the terminal.

Launch the ncs_cli
Create a loopback interface with an IP schematic.
go to the top directory and verify the pending configuration changes.

Output the changes using XML formatting and copy the interface configuration section between the </config> tags

Using a text editor such as gedit, nano, vi, vim or whatever you are comfortabl with paste in the XML configuration you copied between the </config> tags
rename the IP address and give it a variable of {/dummy}
Save the configuration after verifying the config contents are proper.

make the package
Get back into the ncs_cli
reload the packages in ncs.

Creating a Service Instance

From configuration mode in the NSO CLI use the new loopback-service command.
Name the instance test and add variables for the device and IP scheme.

Verify the configuration on the a0 device.