Can't seem to get 10gig negotiated

Networking
1

I have IPFire core 146 running on a Dell Poweredge R620 and the following Ethernet controllers:

  • Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
  • Intel Corporation I350 Gigabit Network Connection (rev 01)

Green and Red are using the x540, and use Cat 6 cables to connect to an HP 5406R ZL2 switch with 10Gig module. For some reason, I can’t seem to get 10gig connections to green or red. Other devices use that switch and successfully autonegotiate 10gig.

I’ve tried using other cables, an identical switch, manually forcing 10gig using ethtool and on the switch. Probably a couple other things I can’t think of right now. Any suggestions on what else I can try, or why I can’t get 10 gig?

Here’s the ethtool output for green (red is identical):

[root@fw ~]#  ethtool green0
Settings for green0:
	Supported ports: [ TP ]
  	Supported link modes:   100baseT/Full
  	                        1000baseT/Full
  	                        10000baseT/Full
  	Supported pause frame use: Symmetric
  	Supports auto-negotiation: Yes
  	Advertised link modes:  100baseT/Full
  	                        1000baseT/Full
  	                        10000baseT/Full
  	Advertised pause frame use: Symmetric
  	Advertised auto-negotiation: Yes
  	Speed: 1000Mb/s
  	Duplex: Full
  	Port: Twisted Pair
  	PHYAD: 0
  	Transceiver: internal
  	Auto-negotiation: on
  	MDI-X: Unknown
  	Supports Wake-on: umbg
  	Wake-on: g
  	Current message level: 0x00000007 (7)
  			               drv probe link
  	Link detected: yes

Thanks!

2

Any output in the kernel log?

This should easily work. Maybe not enough power from the PSU? Sounds more like a hardware than a software problem to me.

3

Here are relevant kernel logs. I don’t see anything noteworthy in there, except that I failed to mention that orange is using one of the 1gig nics.

15:13:06 kernel:  ixgbe 0000:01:00.0 green0: NIC Link is Up 1 Gbps, Flow Control: None
15:13:03 kernel:  ixgbe 0000:01:00.0 green0: NIC Link is Down
15:13:01 kernel:  ixgbe 0000:01:00.1 red0: NIC Link is Up 1 Gbps, Flow Control: None
15:13:01 kernel:  igb 0000:03:00.0 orange0: igb: orange0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
15:13:01 kernel:  ixgbe 0000:01:00.0 green0: NIC Link is Up 1 Gbps, Flow Control: None
15:12:58 kernel:  ixgbe 0000:01:00.1: registered PHC device on red0
15:12:58 kernel:  ixgbe 0000:01:00.0: registered PHC device on green0

This system is pretty much idle in terms of resource utilization; I’d be surprised if the psu was having any trouble keeping up.

4

We have some of the same Dell PowerEdge 620 systems in production but with the Broadcom 10g SFP+ / 1g ethernet interfaces. They are connected to Dell 10g switches. With IPFire, it just works and the CPU utilization never goes over 5 percent even with heavy OpenVPN and internet traffic.

This sounds like a driver or a switch issue. Is the switch set to auto negotiate? Which NIC mezzanine card is in your R620? The Dell part number is clearly marked on the card and maybe the key to getting to the bottom of your problem.

5

You may have bought a fake or faulty cheap chinese 10G intel card. There are lots of posts in the net.

The X540 needs perfect cables, otherwise you will have unstable links even with a few meters. I even had problems with 20 meters of cat7 cables with cat6a connectors. The bad transmitter output was fixed with the X550 series.

1 Like
6

The switch is set to autonegotiate. I’m not sure which mezzanine card is in use, and I won’t be able to check until the next maintenance window, 1 week from today.

7

The Intel card came with the Dell, I hope they have a reliable source. :slight_smile:

I’ve been able to test with the same cables used in successful 10gig connections and a x540.

8

More testing has shown that 10gig is successfully negotiated if I connect to an HP FlexFabric 5940 or a different HP 5406R. Apparently the problem is somewhere in the 5406R I was originally using. The two 5406r have different software versions, so that may be the culprit, and upgrades need to happen anyway.

9

Upgrading the 5406R did the trick. 10gig is now successfully negotiated.

1 Like