Disabling Gigabit Link Negotiation on Fiber Interfaces

File this one under “things I’ve missed so many times I should write a blog article about them.”

I’ve Been Here Before

Here’s the scenario: You’ve ordered a new Ethernet-delivered circuit from your ISP to connect to equipment in your cabinet at a colo facility. The carrier has dropped the circuit, the colo staff has done the cross-connect and left you a fiber pigtail in your cabinet. You’ve configured your port, inserted your SFP optical module, and plugged in the fiber. You “no shut” your port and….. nothing. Nada. Zilch. Not even a link light.

Now what? Maybe the fiber pair is swapped. You try swapping it or do a quick check to make sure there is light output on the correct side. You suspect maybe there’s a bad component. You swap SFPs, move your SFP (and config) to another port, then spend a while messing with fiber loopbacks. Everything checks out and the link still doesn’t come up. Maybe the provider didn’t really hand the circuit off as 1000BaseSX. You start going through your order info, then emailing your account team, while your port sits dark. They confirm the handoff type. Now you’re wondering if your signal level is too low. Maybe there were too many cross-connects or a bad patch cable or strand. You beg the colo staff to dig out their level checker or shoot the fiber to verify signal. Nothing’s adding up.

Exasperated (and under increasing management pressure since this link was supposed to be up last week and you’re spending another day at the colo troubleshooting), you put a call in to the ISP’s NOC. They insist the link is up. It’s just not passing traffic. It must be your switch, as they’re not seeing any frames from your side. Finally you get escalated to an engineer that has been around the block, and he simply asks “Did you configure ‘speed nonegotiate’?” You try this obscure command, and…

 *Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
 *Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

What Was That About?

If you’re like me, you wonder just what there is to negotiate about a fiber-delivered, gigabit link. The optics don’t work at anything but gigabit. And there are two independent fiber strands for TX and RX. What is there to negotiate?! (I’m talking about speed/duplex here, there are actually a number of parameters that have to be negotiated for any gigabit link related to a master/slave relationship and other things)

Well, it turns out that things like speed and duplex actually are negotiated, even on fiber links. According to the Cisco Catalyst 6500 Configuration Guide (emphasis mine):

On Gigabit Ethernet ports, link negotiation exchanges flow-control parameters, remote fault information, and duplex information. Link negotiation is enabled by default. The ports on both ends of a link must have the same setting. The link will not come up if the ports at each end of the link are set inconsistently (link negotiation enabled on one port and disabled on the other port).

The source link goes on to provide a table:

Link Negotiation State Link Status
Local Port Remote Port Local Port Remote Port
Off Off Up Up
On On Up Up
Off On Up Down
On Off Down Up

The table explains the situation with our hypothetical circuit turn-up. Carriers love to hardcode links (if I don’t get around to ranting about this, Greg Ferro did a fine job of it a couple years ago). And for better or worse, whereas hardcoding a copper link typically causes a duplex mismatch and the resultant horrible performance, a mismatch with “hardcoding” a fiber link makes the link fail to come up.

Notice that from the CLI, on a copper port (Fa0/1 in my lab) there are options for the actual speed and duplex values. Again, those inclined to hardcode those are familiar with setting “speed 100” and “duplex full” :

SW1-SP(config)#int fa0/1
SW1-SP(config-if)#speed ?
 10 Force 10 Mbps operation
 100 Force 100 Mbps operation
 auto Enable AUTO speed configuration

SW1-SP(config-if)#duplex ?
 auto Enable AUTO duplex configuration
 full Force full duplex operation
 half Force half-duplex operation

But on the fiber Gi0/1 interface, there is no option to set speed and duplex explicitly (because as I stated above there really isn’t anything else you can set the port to when you’re talking about a fiber physical connection into a 1000BaseSX optical transceiver). All we have is the mysterious “speed nonegotiate” option:

SW1-SP(config)#int gi0/1
SW1-SP(config-if)#speed ?
 nonegotiate Do not negotiate speed
SW1-SP(config-if)#dupl?
% Unrecognized command

So the “speed nonegotiate” command is what disables the auto-negotiation behavior on the port, and if the two ports disagree on whether or not to do that negotiation what you get is the side that is set for “nonegotiate” will actually show “up/up” and the side that is using the default “no speed nonegotiate” (awkward…) will show “down/down.” Good times.

You can even see in the “show interface” output that the port knows it’s being forced up. Observe:

SW1-SP#show int gi0/1 status
Port  Name         Status   Vlan Duplex Speed Type
Gi0/1              disabled 1    auto   auto  1000BaseSX SFP

SW1-SP#conf t
SW1-SP(config)#interface GigabitEthernet0/1
SW1-SP(config-if)#speed nonegotiate
SW1-SP(config-if)#no shut
*Mar 1 01:18:49.228: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Mar 1 01:18:50.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

SW1-SP(config-if)#do sh int gi0/1 | i line proto|link type
GigabitEthernet0/1 is up, line protocol is up (connected)
 Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseSX SFP

Meanwhile, on the other end of the link, things are not as rosey:

SW2-CPE#sh int gi0/1 status
Port  Name         Status   Vlan Duplex Speed Type
Gi0/1              disabled 1    auto   auto  1000BaseSX SFP

SW2-CPE(config)#int gi0/1
SW2-CPE(config-if)#no shut
*Mar 1 01:18:48.431: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

SW2-CPE#show int gi0/1 | i line proto|link type
GigabitEthernet0/1 is down, line protocol is down (notconnect)
Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP

And now we can see why the first-tier NOC guys at the SP thought the link was fine:

SW1-SP(config-if)#do sh int gi0/1 status
Port  Name              Status     Vlan  Duplex Speed Type
Gi0/1                   connected  1     full   1000  1000BaseSX SFP

SW2-CPE(config-if)#do sh int gi0/1 status
Port  Name              Status     Vlan  Duplex Speed Type
Gi0/1                   notconnect 1     auto   auto  1000BaseSX SFP

So there we are. Got a fiber link to a provider that just won’t come up? Try disabling the gigabit autonegotiation. It’s faster than hunting down a guy from the colo team to have him check the cross-connects 🙂

Advertisements
Tagged , , ,

8 thoughts on “Disabling Gigabit Link Negotiation on Fiber Interfaces

  1. John says:

    The “speed nonegotiate” command is indeed helpful, especially with providers who persist in using it on their equipment, but beware – as I recall, turning off negotiation on a fiber port makes it so that a port status of UP UP simply means “I see light on my Rx port”. It doesn’t guarantee that what it sees on the Rx is valid, nor does it imply that the Tx link is functioning, because there’s no two way communication going on.

    So as suggested, auto-negotiation all the way (by choice). There are instances where it just won’t work, but my default stance us to negotiate.

    • bobmccouch says:

      That’s correct, when ‘speed nonegotiate’ is enabled, the port goes “up/up” basically as soon as it seems some light. One more reason auto is good — it’s smarter. In fact, if you create a unidirectional link while the port is at auto, it will be noticed and the port brought down right away. But with ‘nonegotiate’ on, a unidirectional link may not cause the port to go down, making UDLD much more important.

      I may update the post to include this info. Thanks for reading, and for bringing this up!

  2. Daniel Dib says:

    This is one of those things you run into sooner or later if you work with/at an ISP 🙂 The negotiation stuff is quite advanced if you read through the IEEE 802.3 specification. It’s strange really that you can disable auto negotation when IEEE says that it is mandatory.

  3. […] nonegotiate” — As I recently described on this very […]

  4. Note that many (all?) active CWDM and DWDM devices require the link negotiation to be disabled, so this is not just some ISPs requiring some obscure configurations, there is device-dependency as well.

  5. Ken Vescovi says:

    Thanks for the info. I’m into attempt number 3 for cutting over our ATT link to our new 4451 with GLC-LH-SM SFP. The worst part was that when I hooked up the sfp of our new 4451 border router to another sfp on a nexus fiber module, it came right up. It made me believe there was a negotiation parameter problem, especially after light levels were within spec.

    • bobmccouch says:

      Hi Ken, yes, Nexus and IOS devices sometimes behave differently with regard to things like that. I haven’t tested a Nexus interface in the “nonegotiate”scenario, but it’s certainly possible it doesn’t have the same problem. Good luck with your turn-up.

      • js says:

        HI, this is common problem on Nexus that sometimes 1000G-SX does not come up even you check everything(polarity, cleaning connectors, checking cable atenuation, replacing cable), after that try no negotiate auto
        js

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

@greatwhitetec

Virtualization, Storage, and other techy stuff

The Stupid Engineer

I ask those questions you're too clever to.

Sunay Tripathi's Blog

Pluribus Networks Founder's Blog on OS, Networking, Virtualization, Cloud Computing, Solaris Architecture, etc

Ed Koehler's Blog

Just another WordPress.com weblog

JGS.io

Data networking, stray thoughts, nerdy fun...

Network Heresy

Tales of the network reformation

The Borg Queen

Jottings on the intersection of tech and humanness

Networking From The Trenches

Ramblings about my thoughts, experiences, and ideas.

In Search of Tech

Looking for the next big thing.

Packet Maniac

A day in the life of a maniac packet

Fryguy's Blog

A Network Blog by a Network Engineer

Networking 40,000

Attaining my CCIE with the help of Warhammer 40k

stubby router

just another networking blog

Ronnie Angello

Network Architect . CCIE 17846 . CCDE 2012::1

The Peering Introvert

The sundry interests of Ethan Banks including books, cars, hiking in New Hampshire, religion, music, home theater, technology, geek culture, and social media. And maybe cats.

%d bloggers like this: