File this one under “things I’ve missed so many times I should write a blog article about them.”
I’ve Been Here Before
Here’s the scenario: You’ve ordered a new Ethernet-delivered circuit from your ISP to connect to equipment in your cabinet at a colo facility. The carrier has dropped the circuit, the colo staff has done the cross-connect and left you a fiber pigtail in your cabinet. You’ve configured your port, inserted your SFP optical module, and plugged in the fiber. You “no shut” your port and….. nothing. Nada. Zilch. Not even a link light.
Now what? Maybe the fiber pair is swapped. You try swapping it or do a quick check to make sure there is light output on the correct side. You suspect maybe there’s a bad component. You swap SFPs, move your SFP (and config) to another port, then spend a while messing with fiber loopbacks. Everything checks out and the link still doesn’t come up. Maybe the provider didn’t really hand the circuit off as 1000BaseSX. You start going through your order info, then emailing your account team, while your port sits dark. They confirm the handoff type. Now you’re wondering if your signal level is too low. Maybe there were too many cross-connects or a bad patch cable or strand. You beg the colo staff to dig out their level checker or shoot the fiber to verify signal. Nothing’s adding up.
Exasperated (and under increasing management pressure since this link was supposed to be up last week and you’re spending another day at the colo troubleshooting), you put a call in to the ISP’s NOC. They insist the link is up. It’s just not passing traffic. It must be your switch, as they’re not seeing any frames from your side. Finally you get escalated to an engineer that has been around the block, and he simply asks “Did you configure ‘speed nonegotiate’?” You try this obscure command, and…
*Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up *Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
What Was That About?
If you’re like me, you wonder just what there is to negotiate about a fiber-delivered, gigabit link. The optics don’t work at anything but gigabit. And there are two independent fiber strands for TX and RX. What is there to negotiate?! (I’m talking about speed/duplex here, there are actually a number of parameters that have to be negotiated for any gigabit link related to a master/slave relationship and other things)
Well, it turns out that things like speed and duplex actually are negotiated, even on fiber links. According to the Cisco Catalyst 6500 Configuration Guide (emphasis mine):
On Gigabit Ethernet ports, link negotiation exchanges flow-control parameters, remote fault information, and duplex information. Link negotiation is enabled by default. The ports on both ends of a link must have the same setting. The link will not come up if the ports at each end of the link are set inconsistently (link negotiation enabled on one port and disabled on the other port).
The source link goes on to provide a table:
|Link Negotiation State||Link Status|
|Local Port||Remote Port||Local Port||Remote Port|
The table explains the situation with our hypothetical circuit turn-up. Carriers love to hardcode links (if I don’t get around to ranting about this, Greg Ferro did a fine job of it a couple years ago). And for better or worse, whereas hardcoding a copper link typically causes a duplex mismatch and the resultant horrible performance, a mismatch with “hardcoding” a fiber link makes the link fail to come up.
Notice that from the CLI, on a copper port (Fa0/1 in my lab) there are options for the actual speed and duplex values. Again, those inclined to hardcode those are familiar with setting “speed 100” and “duplex full” :
SW1-SP(config)#int fa0/1 SW1-SP(config-if)#speed ? 10 Force 10 Mbps operation 100 Force 100 Mbps operation auto Enable AUTO speed configuration SW1-SP(config-if)#duplex ? auto Enable AUTO duplex configuration full Force full duplex operation half Force half-duplex operation
But on the fiber Gi0/1 interface, there is no option to set speed and duplex explicitly (because as I stated above there really isn’t anything else you can set the port to when you’re talking about a fiber physical connection into a 1000BaseSX optical transceiver). All we have is the mysterious “speed nonegotiate” option:
SW1-SP(config)#int gi0/1 SW1-SP(config-if)#speed ? nonegotiate Do not negotiate speed SW1-SP(config-if)#dupl? % Unrecognized command
So the “speed nonegotiate” command is what disables the auto-negotiation behavior on the port, and if the two ports disagree on whether or not to do that negotiation what you get is the side that is set for “nonegotiate” will actually show “up/up” and the side that is using the default “no speed nonegotiate” (awkward…) will show “down/down.” Good times.
You can even see in the “show interface” output that the port knows it’s being forced up. Observe:
SW1-SP#show int gi0/1 status Port Name Status Vlan Duplex Speed Type Gi0/1 disabled 1 auto auto 1000BaseSX SFP SW1-SP#conf t SW1-SP(config)#interface GigabitEthernet0/1 SW1-SP(config-if)#speed nonegotiate SW1-SP(config-if)#no shut *Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up *Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up SW1-SP(config-if)#do sh int gi0/1 | i line proto|link type GigabitEthernet0/1 is up, line protocol is up (connected) Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseSX SFP
Meanwhile, on the other end of the link, things are not as rosey:
SW2-CPE#sh int gi0/1 status Port Name Status Vlan Duplex Speed Type Gi0/1 disabled 1 auto auto 1000BaseSX SFP SW2-CPE(config)#int gi0/1 SW2-CPE(config-if)#no shut *Mar 1 01:18:48.431: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down SW2-CPE#show int gi0/1 | i line proto|link type GigabitEthernet0/1 is down, line protocol is down (notconnect) Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP
And now we can see why the first-tier NOC guys at the SP thought the link was fine:
SW1-SP(config-if)#do sh int gi0/1 status Port Name Status Vlan Duplex Speed Type Gi0/1 connected 1 full 1000 1000BaseSX SFP SW2-CPE(config-if)#do sh int gi0/1 status Port Name Status Vlan Duplex Speed Type Gi0/1 notconnect 1 auto auto 1000BaseSX SFP
So there we are. Got a fiber link to a provider that just won’t come up? Try disabling the gigabit autonegotiation. It’s faster than hunting down a guy from the colo team to have him check the cross-connects 🙂