Enabling jumbo frames = less performance/slower?

I've been using onboard LAN on my file server since forever. It had a Realtek 8168 and I generally saw good performance during gigabit transfer. Iperf shown around 110MB/s in both directions and FTP/Samba backed that.

I decided this Christmas to move to an Intel Gigabit card, PCIe 1x (2.0) - card is EXPI9301CTBLK. Shows up as Intel Corporation 82574L Gigabit Network Connection in linux (Ubuntu Server 10.04.3 x64). It' using the e1000e driver, I've updated it to the latest release by Intel to resolve any issues.

This card supports 9k jumbo frames, so does my computer and my switch supports 16k jumbo frames. Previously I had 9k set on my computer, and 7k set on the Realtek - which worked great. However with this Intel, if I leave default MTU of 1500 - I get pretty great performance, about 113MB/s in iperf, both directions. If I set 9k - it drops to about 81MB/s in both directions, FTP/Samba both support this. If set to default, 1500 - FTP/Samba and generally in the 100MB/s range.

Normally I'd just stfu and use 1500 - except I'd like to get the most out of my network - why is setting 9k jumbo frames causing so many issues? What can I do to narrow down my issue? Or is it an issue with the PCIe card?

You aren't going to see much more than 113 MB/sec since the theoretical max is 125 MB/sec and you have overhead. There isn't really anything more to gain.

Count yourself lucky you are getting those speeds. I've tried nearly everything to get Linux to go more than 30 MB/sec read and 60 MB/sec through Samba.

Ahh...I guess it's just really strange that enabling jumbo frames actually hurts performance. My SMB config is very generic...looking at yours, I tried doing socket options such as TCP NO DELAY and such - it caused me to go from good performance (80 ~ 100mb/s) to 5 ~ 10mb/s. My install isn't anything special, just clean, updated drivers and what not. My client is Windows 7 though...so I'm not sure if that makes a difference in your situation.

Edit: Slightly interesting, I'm trying 4k jumbo frames and I'm not seeing a massive performance hit like before. I wonder if it's not the intel card but my computers onboard that is acting as a client.

Gigabit doesn't have much more than what you're getting out of it. When you turn on 9K, you might end up with so much data TRYING to go down the wire, that it gets lost at some point.

CompuTamer said:

Gigabit doesn't have much more than what you're getting out of it. When you turn on 9K, you might end up with so much data TRYING to go down the wire, that it gets lost at some point.

Click to expand...

Didn't even think that was possible, guess you learn something new everyday. When will 10gb be affordable

When enabling jumbo frames you need to make sure everything between the server and host can support the large packets, including other switches (and routers/firewalls should they traverse it). Otherwise you will end up with heavy fragmentation and/or packet loss, which leads to the poor performance you're seeing. Try connecting two systems together with both of them running jumbo frames and see how quicker it is.

From the server to my computer and my wifes we are all capable of jumbo frames, except my switch leads up to a router which can do jumbo frames but DD-WRT doesn't seem to like setting it. What your saying is that router being on my network is causing enough fragmentation to hurt performance?

Unless the data is being routed through the router itself (it isn't if you are using a switch), that won't affect it.