Gigabit NICs

[Gizmo]

Got a question for all you networking gurus out there.

I have a customer who has a system using dual Gigabit NICs. The configuration is somewhat complex, but it goes like this:

There are three servers. Each server has two NICs. One NIC connects to a dedicated network shared just between the three servers. This is used for internal data replication between the servers, so that if one goes down, the others have the data necessary to take the load. The other NIC is used to connect the server to the outside world.

The NICs in question are Intel EtherPro/1000 using the 82465 chipset.

The customer is complaining that the system is intermittently slow to respond. In this particular case, I've got a custom web server application I wrote which implements a very small portion of the W3C standard. I use it because the mainstream web server apps are just too fat. When I look in my applications logs, everything appears normal. However, because my application is working at the TCP layer, I don't see anything from the network until the TCP connection is completed. In other words, if there is something going on with the NIC driver or the network itself that is resulting in delays or retries, it would be invisible to my app. The symptoms are that when this problem occurs, the client app makes a request and several seconds (sometimes several 10s of seconds) later it gets a response. During this period of time the server does not show unusual disk, cpu, or network traffic.

I am about to deplay a laptop machine with a gigabit adapter and a copy of Ethereal to sniff the network so that I can try to see what is going on, but I have the potential to end up digging through a LOT of data without finding anything.

Have any of you seen anything like this before?

[Áedán]

I've seen something similar before, but I'm not sure it'd apply to the setup you're having problems with.
When dealing with some Sun blades and Cisco switches, there was a negotiation problem where the Ciscos would negotate full duplex, and the Suns would negotiate half duplex. The end result was a somewhat intermittent drop in response time. It didn't help that we were load balancing across about four web servers, as it made it difficult to tell where the problem was.

[XeroHouR]

this isn't due to massive collisions and jamming is it??

[Gizmo]

Originally Posted by XeroHouR

this isn't due to massive collisions and jamming is it??

Not as far as I can tell. I have been told that the switches (they are using managed Cisco switches) have not reported anything abnormal.

[Gizmo]

Originally Posted by Áedán

I've seen something similar before, but I'm not sure it'd apply to the setup you're having problems with.
When dealing with some Sun blades and Cisco switches, there was a negotiation problem where the Ciscos would negotate full duplex, and the Suns would negotiate half duplex. The end result was a somewhat intermittent drop in response time. It didn't help that we were load balancing across about four web servers, as it made it difficult to tell where the problem was.

My first thought was something like this, but I've been told the switches are configured for full duplex, and according to the drivers, so are the NICs.

[XeroHouR]

collision isn't abnormal on a normal ethernet, that is unless you're running token ring. So yea, its prolly not that. Can you determine where as far as the layer the problem is? If one of your nics is messed up then it could be sending out an unnecissarily large amount of frames and overloading the network traffic

[Gizmo]

That's one of the things I'm hoping to answer with the sniffer. I need to try to establish the relationship between when the traffic arrives on the wire and when my app processes it.

[Toro]

and its nothing mad like it going on to power saving mde or anything completely insane like that?


yes i know nothing, used to, dont really know where it went, nevermind

[Áedán]

Originally Posted by XeroHouR

collision isn't abnormal on a normal ethernet, that is unless you're running token ring.

Remember we're talking a fully switched network running in full duplex mode here. It would be unusual to be seeing any collisions on such a network. In fact, if you did see collisions, it would indicate that there was a problem somewhere.

[XeroHouR]

I was pretty sure that on a broadcast network, duplex or not, switched or not, there is always some level of collision. I am in my first semester of Cisco however, so I'm no expert *bow*

[Gizmo]

In a fully switched network, there are only ever two devices on a segment; the computer and the switch. In other words, a fully switched network is basically a logical point-to-point configuration, although it will usually be physically arranged in some variation of a star topology. Under such conditions, there should never be any opportunity for collisions on a given segment, and the switching fabric itself will take care of ensuring that collisions do not occur upstream, where packets from multiple sources will share the same wire. Because of this, collisions should be nearly non-existent and the presence of collision alarms indicates a problem.

The nature of a switched network also presents some difficulties for packet sniffers like what I am trying to deploy. Why? Because the whole premise of a switched network is that the only traffic that should ever appear on a given segment is the traffic intended for a machine on that segment. If there is only one machine on any given segment, then how do you sniff traffic for an arbitrary machine and be certain that you are getting ALL of it?

There are two possibilites:

1) Insert a hub between the machine you want to sniff and the switch. Since a hub replicates ALL of its traffic to ALL of its nodes, this means that you can sniff the traffic of the machine you are interested in, PROVIDED you are on the same hub as the machine in question. Unfortunately, there's no such thing as a gigabit hub at this stage in the game.

2) Get a MANAGED switch that allows you to do port mirroring. This is what I am doing. This type of switch allows you to essentially take two of the switch ports and configure them as a 'mini hub', thus allowing me to monitor the traffic to the server in question.

[cadaveca]

It's not negotiating across the wrong network, is it?

[Gizmo]

I can't say for 100% certain that it isn't, but given the intermittent nature of the problem, I tend to doubt it. The internal network is on a completely different subnet that is unused anywhere else in the facility, and all internal communications is done by ip address to ensure that traffic gets routed appropriately. The Computer Browser service is also disabled in order to ensure that none of the servers becomes a Master or Backup Browser.