File this one under “things I’ve missed so many times I should write a blog article about them.”
I’ve Been Here Before
Here’s the scenario: You’ve ordered a new Ethernet-delivered circuit from your ISP to connect to equipment in your cabinet at a colo facility. The carrier has dropped the circuit, the colo staff has done the cross-connect and left you a fiber pigtail in your cabinet. You’ve configured your port, inserted your SFP optical module, and plugged in the fiber. You “no shut” your port and….. nothing. Nada. Zilch. Not even a link light.
Now what? Maybe the fiber pair is swapped. You try swapping it or do a quick check to make sure there is light output on the correct side. You suspect maybe there’s a bad component. You swap SFPs, move your SFP (and config) to another port, then spend a while messing with fiber loopbacks. Everything checks out and the link still doesn’t come up. Maybe the provider didn’t really hand the circuit off as 1000BaseSX. You start going through your order info, then emailing your account team, while your port sits dark. They confirm the handoff type. Now you’re wondering if your signal level is too low. Maybe there were too many cross-connects or a bad patch cable or strand. You beg the colo staff to dig out their level checker or shoot the fiber to verify signal. Nothing’s adding up.
Exasperated (and under increasing management pressure since this link was supposed to be up last week and you’re spending another day at the colo troubleshooting), you put a call in to the ISP’s NOC. They insist the link is up. It’s just not passing traffic. It must be your switch, as they’re not seeing any frames from your side. Finally you get escalated to an engineer that has been around the block, and he simply asks “Did you configure ‘speed nonegotiate’?” You try this obscure command, and…
*Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
What Was That About?
If you’re like me, you wonder just what there is to negotiate about a fiber-delivered, gigabit link. The optics don’t work at anything but gigabit. And there are two independent fiber strands for TX and RX. What is there to negotiate?! (I’m talking about speed/duplex here, there are actually a number of parameters that have to be negotiated for any gigabit link related to a master/slave relationship and other things)
Well, it turns out that things like speed and duplex actually are negotiated, even on fiber links. According to the Cisco Catalyst 6500 Configuration Guide (emphasis mine):
On Gigabit Ethernet ports, link negotiation exchanges flow-control parameters, remote fault information, and duplex information. Link negotiation is enabled by default. The ports on both ends of a link must have the same setting. The link will not come up if the ports at each end of the link are set inconsistently (link negotiation enabled on one port and disabled on the other port).
The source link goes on to provide a table:
| Link Negotiation State | Link Status | ||
|---|---|---|---|
| Local Port | Remote Port | Local Port | Remote Port |
| Off | Off | Up | Up |
| On | On | Up | Up |
| Off | On | Up | Down |
| On | Off | Down | Up |
The table explains the situation with our hypothetical circuit turn-up. Carriers love to hardcode links (if I don’t get around to ranting about this, Greg Ferro did a fine job of it a couple years ago). And for better or worse, whereas hardcoding a copper link typically causes a duplex mismatch and the resultant horrible performance, a mismatch with “hardcoding” a fiber link makes the link fail to come up.
Notice that from the CLI, on a copper port (Fa0/1 in my lab) there are options for the actual speed and duplex values. Again, those inclined to hardcode those are familiar with setting “speed 100” and “duplex full” :
SW1-SP(config)#int fa0/1
SW1-SP(config-if)#speed ?
10 Force 10 Mbps operation
100 Force 100 Mbps operation
auto Enable AUTO speed configuration
SW1-SP(config-if)#duplex ?
auto Enable AUTO duplex configuration
full Force full duplex operation
half Force half-duplex operation
But on the fiber Gi0/1 interface, there is no option to set speed and duplex explicitly (because as I stated above there really isn’t anything else you can set the port to when you’re talking about a fiber physical connection into a 1000BaseSX optical transceiver). All we have is the mysterious “speed nonegotiate” option:
SW1-SP(config)#int gi0/1
SW1-SP(config-if)#speed ?
nonegotiate Do not negotiate speed
SW1-SP(config-if)#dupl?
% Unrecognized command
So the “speed nonegotiate” command is what disables the auto-negotiation behavior on the port, and if the two ports disagree on whether or not to do that negotiation what you get is the side that is set for “nonegotiate” will actually show “up/up” and the side that is using the default “no speed nonegotiate” (awkward…) will show “down/down.” Good times.
You can even see in the “show interface” output that the port knows it’s being forced up. Observe:
SW1-SP#show int gi0/1 status
Port Name Status Vlan Duplex Speed Type
Gi0/1 disabled 1 auto auto 1000BaseSX SFP
SW1-SP#conf t
SW1-SP(config)#interface GigabitEthernet0/1
SW1-SP(config-if)#speed nonegotiate
SW1-SP(config-if)#no shut
*Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
SW1-SP(config-if)#do sh int gi0/1 | i line proto|link type
GigabitEthernet0/1 is up, line protocol is up (connected)
Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseSX SFP
Meanwhile, on the other end of the link, things are not as rosey:
SW2-CPE#sh int gi0/1 status
Port Name Status Vlan Duplex Speed Type
Gi0/1 disabled 1 auto auto 1000BaseSX SFP
SW2-CPE(config)#int gi0/1
SW2-CPE(config-if)#no shut
*Mar 1 01:18:48.431: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
SW2-CPE#show int gi0/1 | i line proto|link type
GigabitEthernet0/1 is down, line protocol is down (notconnect)
Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP
And now we can see why the first-tier NOC guys at the SP thought the link was fine:
SW1-SP(config-if)#do sh int gi0/1 status
Port Name Status Vlan Duplex Speed Type
Gi0/1 connected 1 full 1000 1000BaseSX SFP
SW2-CPE(config-if)#do sh int gi0/1 status
Port Name Status Vlan Duplex Speed Type
Gi0/1 notconnect 1 auto auto 1000BaseSX SFP
So there we are. Got a fiber link to a provider that just won’t come up? Try disabling the gigabit autonegotiation. It’s faster than hunting down a guy from the colo team to have him check the cross-connects 🙂
Herding Packets 
The “speed nonegotiate” command is indeed helpful, especially with providers who persist in using it on their equipment, but beware – as I recall, turning off negotiation on a fiber port makes it so that a port status of UP UP simply means “I see light on my Rx port”. It doesn’t guarantee that what it sees on the Rx is valid, nor does it imply that the Tx link is functioning, because there’s no two way communication going on.
So as suggested, auto-negotiation all the way (by choice). There are instances where it just won’t work, but my default stance us to negotiate.
That’s correct, when ‘speed nonegotiate’ is enabled, the port goes “up/up” basically as soon as it seems some light. One more reason auto is good — it’s smarter. In fact, if you create a unidirectional link while the port is at auto, it will be noticed and the port brought down right away. But with ‘nonegotiate’ on, a unidirectional link may not cause the port to go down, making UDLD much more important.
I may update the post to include this info. Thanks for reading, and for bringing this up!
This is one of those things you run into sooner or later if you work with/at an ISP 🙂 The negotiation stuff is quite advanced if you read through the IEEE 802.3 specification. It’s strange really that you can disable auto negotation when IEEE says that it is mandatory.
[…] nonegotiate” — As I recently described on this very […]
Note that many (all?) active CWDM and DWDM devices require the link negotiation to be disabled, so this is not just some ISPs requiring some obscure configurations, there is device-dependency as well.
Thanks for the info. I’m into attempt number 3 for cutting over our ATT link to our new 4451 with GLC-LH-SM SFP. The worst part was that when I hooked up the sfp of our new 4451 border router to another sfp on a nexus fiber module, it came right up. It made me believe there was a negotiation parameter problem, especially after light levels were within spec.
Hi Ken, yes, Nexus and IOS devices sometimes behave differently with regard to things like that. I haven’t tested a Nexus interface in the “nonegotiate”scenario, but it’s certainly possible it doesn’t have the same problem. Good luck with your turn-up.
HI, this is common problem on Nexus that sometimes 1000G-SX does not come up even you check everything(polarity, cleaning connectors, checking cable atenuation, replacing cable), after that try no negotiate auto
js