Small Linux network issues

Automata · Aug 1, 2011

I've been searching online for months, but I just can't find answers to my questions. I have two issues, which are completely unrelated.

This is for the computer in my signature. My desktop OS is Fedora 14 with 2.6.35.13-92.fc14.x86_64. The server is running CentOS 5.6 with 2.6.18-238.19.1.el5, also x86_64.

First issue:
My transfer rate across the network is slow for Samba, roughly 33 MB/sec. In Windows on the exact same hardware and the same server, I max out my network connection (~117 MB/sec) copying a file to the server's RAID array. I'm using Samba on the server and using this as my benchmark. If I start multiple copies from the server, it still tops out at 33 MB/sec.

I initially thought this was because I had offloading enabled, but disabling it didn't help at all.

Code:

[root@thideras-main ~]# ethtool --show-offload eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: on

I then decided to get clever and see if the network itself was running slow. If I run "nc" on the server and use "pv" piped to "nc" on my computer, I can push 117 MB/sec to the server all day long.

Code:

[root@thideras-main ~]# pv -r /dev/zero | nc 192.168.0.10 2222
[ 112MB/s]

So, this tells me that the network itself is fully capable of doing gigabit speeds, but Samba is running slow. I'm not sure what to change.

--------------------------------------------------------------------
Second issue:
I have my drives mounted in fstab as such:

Code:

//192.168.0.10/hitachi    /mnt/hitachi        cifs    credentials=/root/.smbcredentials,defaults    0 0

If the server becomes unresponsive, either due to a crash or power loss, my desktop computer goes completely unresponsive until I can either umount the share or restart with a modified fstab. From the logs, it repeats this many times:

Code:

Jul  3 23:33:07 thideras-main kernel: [  305.008010] CIFS VFS: No response to cmd 46 mid 4636
Jul  3 23:33:07 thideras-main kernel: [  305.008015] CIFS VFS: Send error in read = -11
Jul  3 23:33:11 thideras-main kernel: [  309.651009] CIFS VFS: No response for cmd 50 mid 4637

Is there an option I can pass so that it doesn't keep trying to contact the server? This isn't so much of an issue because the server is up nearly all the time, but it gets old when I need to shut it down.

It feels weird asking for help, but I simply can't find the answers.

turbohans · Aug 2, 2011

I am looking around for info for you, and myself as I would love to get Samba working on my net. So please bare with me.

For your first problem you have listed I have found that Samba has a lot of tuning features to play with.
The pdf in the following link here outlines some of the 'performance' settings that you can add/tweak. http://oreilly.com/catalog/samba/chapter/book/appb.pdf
And this is their recommended configuration.
-I think this might be outdated though. :shrug:

(smb.conf)

Code:

[global]
log level = 1                      # Default is 0
socket options = TCP_NODELAY IPTOS_LOWDELAY
read raw = yes                     # Default
write raw = yes                    # Default
oplocks = yes                      # Default
max xmit = 65535                   # Default
dead time = 15                     # Default is 0
getwd cache = yes
lpq cache = 30
[okplace]
veto oplock files = this/that/theotherfile
[badplace]
oplocks = no

Also there is the send/receive buffers, I guess these get mixed results though. Could help with a larger number like what these have or might work better disabled.

Code:

SO_RCVBUF=65536 SO_SNDBUF=65536

Hopefully that helps somthing, I am going to set Samba up and play with it myself and look around some more though.

----------------

From what I have gathered on your second problem "mount.cifs" might need to have the "soft" option added? However it seems like the default option (soft) would have prevented hangs on devices accessing the files.

hard
The program accessing a file on the cifs mounted file system will hang when the server crashes.
soft
(default) The program accessing a file on the cifs mounted file system will not hang when the server crashes and will return errors to the user application.

got that here. http://linux.die.net/man/8/mount.cifs

Soo, I think this might be what you need.

Code:

   # device             mountpoint  fs-type    options    dump fsckord
   ...
   master.foo.com:/home  /mnt/home   nfs      rw,soft,intr  0     0
   ...

Automata · Aug 2, 2011

I've tried quite a few options in my configuration file and I know that I've played with buffers. Here is my current configuration.

Code:

[thideras@thideras-server ~]$ cat /etc/samba/smb.conf
#======================= Global Settings =====================================
[global]
        workgroup = WORKGROUP
        server string = Thideras File Server %v
        netbios name = thideras-server
        interfaces = eth0 lo
        bind interfaces only = yes

#This is to stop the error messages in /var/log/messages
        domain master = yes
        local master = yes

        log file = /var/log/samba/log.%m
        # max 50KB per log file, then rotate
        max log size = 50

        security = user
        passdb backend = tdbsam
        encrypt passwords = yes

        max open files = 65536
#       socket options = TCP_NODELAY
        socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072

        wins support = yes
        hostname lookups = yes
#       hosts equiv = /etc/hosts

#--------------------------- Printing Options -----------------------------
        load printers = yes
;       cups options = raw

        printcap name = cups
        printing = cups

Changing these options changed nothing in Linux. Windows ran the same speed. This is why I'm thinking it isn't an option on the server.

-------------------------------------

I added that now and see if I can test this later.

Code:

#Hitachi server share
//192.168.0.10/hitachi  /mnt/hitachi            cifs    credentials=/root/.smbcredentials,defaults,soft 0 0

I appreciate the help thus far.

turbohans · Aug 2, 2011

Yea, these "SO_RCVBUF=131072 SO_SNDBUF=131072" do not do anything unless the kernel alows them somewhat.

I guess there are some limitations there you can tweak too
http://blog.eukhost.com/webhosting/how-to-optimize-tcp-ip-stack-in-linux/

There are big differences in between the Linux versions, but now we will be considering the common issues only. To change the settings of TCP, you need to add following lines to the file /Etc/sysctl.conf and simply run the sysctl-p command.

As in all other operating systems, the size of buffers in Linux is precisely very small. You need to apply the following settings:

Code:

# Increase TCP max buffer size setable using setsockopt ()
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# Increase Linux autotuning TCP buffer limits
# Min, default, and max number of bytes to use
# Set max to at least 4MB, or higher if you use very high BDP paths
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Automata · Aug 2, 2011

Just tried that as well.

Code:

[root@thideras-server ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
[root@thideras-server ~]# service network restart
Shutting down interface br1:                               [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:
Determining IP information for eth0... done.
                                                           [  OK  ]
Bringing up interface eth1:                                [  OK  ]
Bringing up interface br1:
Determining IP information for br1... done.
                                                           [  OK  ]

Noting blew up, so let's try to copy a file.

Code:

[thideras@thideras-main Linux]$ time cp CentOS-5.5-x86_64-bin-DVD-1of2.iso /mnt/

real    1m0.468s
user    0m0.023s
sys     0m4.183s

69.97 MB/sec, not bad. My hard drive might be holding me back, so let's try something else as well.

Code:

[thideras@thideras-main hitachi]$ dd if=/dev/zero of=/mnt/hitachi/test.iso bs=1G count=5
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 66.1717 s, 81.1 MB/s
[thideras@thideras-main hitachi]$ dd if=/dev/zero of=/mnt/hitachi/test.iso bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 16.5596 s, 64.8 MB/s
[thideras@thideras-main hitachi]$ dd if=/dev/zero of=/mnt/hitachi/test.iso bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 138.422 s, 77.6 MB/s

That is certainly much better than I was expecting. I'm going to try these settings on my desktop as well.

Automata · Aug 2, 2011

Did the same on my workstation and I'm getting the same speeds. That is a lot better than what I was getting.

turbohans · Aug 2, 2011

Sweet! forgive me if I missed it but how much of a difference did that make? LOL I am tired I see now, more than double hehe.

I bet that can be tweaked out as fast a Windows pretty easy.

Automata · Aug 2, 2011

Tried to copy once I got back my GUI. Still getting slow speeds.

turbohans · Aug 2, 2011

I did leave this out on accident.

Code:
Also, you need to check if the following parameters are set to the default value of 1.

Code:

sysctl net.ipv4.tcp_window_scaling sysctl net.ipv4.tcp_timestamps sysctl net.ipv4.tcp_sack

Note: You must leave tcp_mem alone. Default values, and so beautiful.

Another thing you may want to try that may help increase the capacity of TCP, should increase the size of the queue interface. In order to do this, enter the following command.

Code:

ifconfig eth0 txqueuelen 1000

range for this is 1000-20000

You will get good increase in speed, making this setting in the wide channels. Doing this makes sense for channels Gigabit Ethernet, but may have side effects such as uneven sharing between multiple streams.

And I think both of those pieces of material I got this from are older than dirt, so the sizes may need adjusting.
For the send/recieve they have this formula to get the correct size, (buffer size = bandwidth * RTT)
I booted over to Ubuntu and started setting everything up. I have not done any transfers yet though. I imagine it still might take you some testing of different values to find a sweet-spot to set it at.
Just noticed you might need to do somthing like this for the code above though.

http://www.cyberciti.biz/faq/gentoo-centos-rhel-debian-fedora-increasing-txqueuelen/

Code:
Setting the txqueuelen permanently

Code:

Edit /etc/rc.locale, enter: vi /etc/rc.local

Append the following setting per interface:

Code:

/sbin/ifconfig eth1 txqueuelen 10000 /sbin/ifconfig eth0 txqueuelen 10000

Automata · Aug 2, 2011

Those three are set to "1" already. Txqueuelen was already at 1000.

turbohans · Aug 2, 2011

thideras said:
Those three are set to "1" already. Txqueuelen was already at 1000.

max value on that is 20,000 and that is the transmit que.

Automata · Aug 2, 2011

Well, that is interesting. I can read at 33 MB/sec and write at 71.7 MB/sec. Raising the txqueuelen did nothing for either.

It is also peaking at 104+ MB/sec when I cancel a transfer.

turbohans · Aug 3, 2011

hmm, yea i got mine working good. I only have a 10/100 router though, so I can't really test much. I get like 9.8mb/s both ways real consistent though, small or big files.

I wonder if I could set up a direct connection between the two machines I have to get a gigabit connection as they both have gb nic's.

Automata · Aug 10, 2011

Well, one thing is fixed. I took down my server shares to move RAID arrays and my computer didn't explode when I shutdown Samba. "Soft" in my fstab file fixed it. Thanks!

When I stopped the service, any windows I had open that were in the network share stopped working, but it let me kill them gracefully.

Code:

#Hitachi server share
//server/hitachi    /mnt/hitachi        cifs    credentials=/root/.smbcredentials,defaults,[COLOR=Red]soft[/COLOR]    0 0

Now I just need to nail down these network transfer issues.

turbohans · Aug 10, 2011

Sweet! Glad that worked! Kinda strange that is supposed to be the default setting (soft) from what it said but seems like it was really defaulting to hard for some reason.

I am still looking into that network tuning though, and have found a few things out after playing with all this also lately. I think the txqueuelen is the same thing as MTU in most routing configurations soo if it was at 1000 or 100 you might want to try 1500.

reading a little on that here right now.

Automata · Aug 11, 2011

Thanks for the link, I'll try that a bit later. I'm making large changes to the internals of my network, which includes formatting the server. I want to make sure that nothing I've done thus far has cause it to run slow. A clean slate is the perfect testbed for this.

turbohans · Aug 12, 2011

thideras said:
Thanks for the link, I'll try that a bit later. I'm making large changes to the internals of my network, which includes formatting the server. I want to make sure that nothing I've done thus far has cause it to run slow. A clean slate is the perfect testbed for this.

Exactly what I was thinking with testing it on Vmware, still trying to figure out if there are virtual network limitations in doing that though.

It's no fun that most of the information for this stuff is soo old though, but I guess that's just the way it is with Linux...

I really want to squezze more details out of this though, as from what I have read in a few places, Samba should be as fast as FTP if tweaked correctly. And 30mb/s on a Gbit network is unacceptable to me LOL

I have done a lot of playing with those settings though, they are pretty much harmless in terms of keeping a system running, but if anyone would like to try there are only two files to backup and play with editing. "/etc/sysctl.conf" & "/etc/samba/smb.conf"

But yea, the information in this thread is a little hard to folow, what I put on my forum is easier to folow but has no results. So I will leave that up to you to decide if it's worth reproducing here in the same format. (as in if you find better values and want to throw up some results)
That's totally up to you though, if you feel like doing the legwork organizing results.

But yea, I am definitively still looking at this, trying to find that fine print written in stone so-to-say.
LOL I posted the thread I made on reddit and got like 500 hits in a few hours,, soooo could do pretty good here!

Automata · Sep 11, 2011

This is making less and less sense the more I transfer data. I have made no changes to the network or any settings since my last post here. Here are two screenshots of me transferring rendered blu-ray files to my file server. The top graph is inbound traffic and the bottom graph is outbound.

The first I found odd. Near the end of the file, it hits 110 MB/sec for several seconds and always at the end of the file. For most of the file, it stays at a steady ~61 MB/sec.

This one is a lot more annoying. During a transfer, it suddenly drops to exactly 11.2 MB/sec and stays there until the next file. This is very close to a 100 Mbps network line, which I'm a bit suspicious of.

I'm completely lost at this point. To make it even stranger, my laptop did the 11.2 MB/sec transfer thing as well. It was on a completely different portion of the network, even. I'm thinking it has to be something with Fedora, since both the laptop and desktop share the exact same version.

Automata · Sep 11, 2011

Hmm, maybe this is a server configuration issue. My Windows box just did the same thing. What is odd is that the Windows box didn't do this for the 700+ GB that I moved a week ago, but it does now.

Since the server is going to be formatted soon, I'm not too worried about it.

turbohans · Sep 11, 2011

Very interesting. I did notice that 1500 seems to be the magical number with the txqueuelen. Looks to me like that is synonymous with the MTU used on most routing/switching firmware. You could set that on all your interfaces individually using a command like this. Substituting eth0 or eth1 for whatever you are using.

Code:

ifconfig eth0 txqueuelen 1500

I would try that before anything else.

Small Linux network issues

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Member

Destroyer of Empires and Use

Destroyer of Empires and Use

Member

Similar threads