Small Linux network issues

[FX4]

I went back and read the thread. I think you need to look at TCP sliding windows. Either the ack isn't coming or Fedora isn't ramping up the TCP window. You should be able to tune the stack in Fedora.

 

[su root]

I went back and read the thread. I think you need to look at TCP sliding windows. Either the ack isn't coming or Fedora isn't ramping up the TCP window. You should be able to tune the stack in Fedora.

Window scaling looks fine... it ramps up relatively quickly and plateaus at 24119 without dipping or scaling back down.



However, it doesn't appear to actually use the window... at any given point, I don't see more than about 5-7kB (one or two packets) in flight, then the client ACKs it before anything else continues. (Specifically, the ACKs in the screencap are ACKing the packet directly above).



This has got me thinking a bit... I live-cd booted one of my clients and did some testing... the out-of-the-box setup gave the same performance problem. Next is to livecd-boot both my server and my clients and see what the performance is from a completely out-of-the-box environment. Maybe I've done this to myself by tinkering with something I shouldn't have.

 

[Automata]

Maybe I've done this to myself by tinkering with something I shouldn't have.

This is what I'm thinking for mine, as well. My Fedora 14 install is old and I don't exactly remember what I've done so far. I installed the day it came out.

 

[FX4]

Well I still think we are on the right track here. You have an agreed to window size but you are not using it. So it seems you need to look at the Fedora stack and see why the transmit is not scaling up. It could be as simple as a setting in the stack. Or it could be the stack is broken. Meaning you have an agreement to transmit more at the TCP level but some setting or register is set wrong so it never gets more data to transmit. This is the fun stuff. What exact version of Fedora are you guys using? I may set up a PC in the next few days to see if I can duplicate your problem.

 

[su root]

Hmm... well that's interesting....
I booted both the server and the client using a Ubuntu 11.10 amd64 disc. Nothing between them except a gig switch and 2x CAT6 STP cables.

Did a simple nfs export (no options), and recreated the performance problem, 33MB/s
Installed samba and did a simple samba share with no other changes, 33MB/s

I also took a win7 system and connected to the samba share, pulled 80+MB/s, then realized I had requested a file that I used earlier that was in cache on the server. Chose another file and pulled the same 33MB/s.

Back to the drawing board for me. I'd love to know what this problem is, but I'll probably end up fixing it with a hardware upgrade on my server.

 

[FX4]

Ubuntu and Fedora use the same IP stack I think.

 

[FX4]

I'm doing a little digging:
http://www.ehow.com/how_5998481_increase-ubuntu-buffers-tcp_ip.html

 

[su root]

Ubuntu and Fedora use the same IP stack I think.

Yeah, it's the standard Linux kernel TCP stack.
I've done some toying with those kernel parameters before, to no effect.

The issue is that we can prove it's not a problem with either the TCP stack or the IO stack. Any network transfers that do not involve Disk IO run at full speed. Local IO, independently of Network runs at full speed. A good example of this is when I transferred a file, then tried the same file transfer from Win7... since it was in memory cache on the server, it didn't need to touch the disk, and went at full speed.

So the problem only occurs when both Disk IO and Network IO happen simultaneously.

 

[FX4]

Hey I just want to make sure, there is a thread over on Ubuntu forums where the NICs were not negotiating correctly and actually settled on 100Mbps rather than 1000Mbps, they hard coded the NICs on both ends and the problem was solved.

 

[Automata]

It is running correctly. I get <30 mb/sec down and <60mb/sec up, which is way more than 10/100.

[   10.507924] sky2 0000:02:00.0: eth0: enabling interface
[   10.508836] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   12.866461] sky2 0000:02:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
[   12.867323] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

 

[su root]

Hey I just want to make sure, there is a thread over on Ubuntu forums where the NICs were not negotiating correctly and actually settled on 100Mbps rather than 1000Mbps, they hard coded the NICs on both ends and the problem was solved.

Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair

33MB/sec = 264Mbit/sec

 

[Automata]

To jump off the topic that I originally posted, I've been playing with bonding tonight on my server. I have 4x gigabit Intel NICs in a bond mode of 4 (802.3ad). I'm seeing a sizable performance increase, but not as much as I'd expect. I'm sitting at 224 mb/sec with 3 servers hitting the bond with netcat. Two will sit at 112 mb/sec and 106 mb/sec, the third is at 6.27 mb/sec. If I stop either of the "fast" ones, the third will suddenly jump to full gigabit speeds. For whatever reason, it won't go past 224 mb/sec, regardless of the number of servers hitting the interface.

[root@ruby ~]# cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 4
    Actor Key: 17
    Partner Key: 4
    Partner Mac Address:

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr:
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr:
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr:
Aggregator ID: 1
Slave queue ID: 0

[root@ruby network-scripts]# cat ifcfg-bond0 ifcfg-eth*
DEVICE="bond0"
USERCTL="no"
BOOTPROTO="none"
ONBOOT="yes"
IPADDR="192.168.0.10"
NETMASK=
NETWORK=
BROADCAST=
GATEWAY=
BONDING_OPTS="miimon=100 mode=4"


DEVICE="eth0"
ONBOOT="yes"
BOOTPROTO="none"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"


DEVICE="eth1"
ONBOOT="yes"
BOOTPROTO="none"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"


DEVICE="eth2"
ONBOOT="yes"
BOOTPROTO="none"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"


DEVICE="eth3"
ONBOOT="yes"
BOOTPROTO="none"
USERCTL="no"
MASTER="bond0"
SLAVE="yes"

If I can't get it to the speed of 4 NICs, I'll just run the two onboard Intel NICs in 802.3ad and leave the other ones to virtual machines. I'd really like to get it working, though.

 

[Automata]

Everyone that has been helping me, my main issue has been resolved. What was it? Something with Fedora 14, I don't know.

I moved to Arch Linux and everything works fine. Ignore the black parts in the screenshot, I'm not sure why it is showing like that. I think I'm missing a Gnome package (using XFCE).



I appreciate the help!

------


Back to my first post, I had another question and now I have one more. If my server gets disconnected, the clients will wait forever and lock up everything on the client unless I umount the drive. For my desktop, I don't really care because it never disconnects. But my laptop will be frequently connected and disconnected from the wireless network, so it may suddenly lose access to the share. Having to manually mount and umount this drive is starting to be a pain. I've looked into automounters before, but I've never had luck.

My other question is newer. Fedora would mount my drives after the network was started. Arch Linux does no such thing. It will fail to mount the network at boot because the service isn't running or the wireless network isn't connected and I have to invoke the mount manually. I know that I could technically add a delay, but I'd rather have it wait until it is available and then connect. This sort of ties into my previous question.

Not sure where to start with this.

 

[turbohans]

This thread keeps getting tastier and tastier. yummy
*Runs off to go play with some unix sockets*

 

[su root]

If you use NFS, a lot of that is automatic. Depending on the options, if the network is not up in time, it will just retry and mount when available. If connectivity is interrupted, it will reconnect when it can. In the mean time, you have the option of how to treat applications.. either tell them that the disk isn't available and fail any I/O, or let them block and wait indefinitely.

On my laptop, I will regularly flip between wireless and wired, and the mount survives the switch (with it taking about a minute to figure out what's going on). Though I typically do this prior to transferring files, not while I'm transferring.

In the client's mount options, set the "bg" option, and the system will try to mount the NFS export once, then continue to boot and retry in the background. (vs. "fg", where it will wait indefinitely, which is better in the case where you are booting off of the network).

Default retry schedule is 10,000 attempts, or approximately 1 week worth of retrying each mount.

Set the option "soft", and if there is an interruption in the connectivity, the system will wait a major timeout (typically 60 seconds) for a response, then fail any I/O requests in progress. Some applications may not like this, but most modern ones will deal with it without crashing. (vs. "hard", which will block I/O until connectivity is restored).

 

[Automata]

I didn't think about that option, thank you. The only question I have is permissions for NFS. I'm pretty sure you can password protect shares, so I'm not worried about that (plus, I have permissions set correctly). The only problem that I could run into is my Samba and NFS shares would basically be the same folder. Depending on the share, I force my username and the group "samba" to keep permission from breaking badly. Since I wouldn't be using NFS, what would be the best way to keep permissions straight without having to do it manually or through CRON? Here is my Samba configuration, if that helps. Basically, I want this to "just work" without having to interfere with it.

[thideras@ruby ~]$ cat /etc/samba/smb.conf
#======================= Global Settings =====================================

[global]
        workgroup = MYGROUP
        server string = Samba Server Version %v
        hosts allow = 127. 192.168.0.
        log file = /var/log/samba/log.%m
        max log size = 50
        security = user
        passdb backend = tdbsam
        local master = yes
        preferred master = yes
        wins support = yes
        socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
        max open files = 65536

# --------------------------- Printing Options -----------------------------
        load printers = yes
        #obtain list of printers automatically on SystemV

#============================ Share Definitions ==============================

[printers]
        comment = All Printers
        path = /var/spool/samba
        browseable = no
        guest ok = no
        writable = no
        printable = yes
        valid users = thideras

[Hitachi]
        comment = Hitachi root
        path = /mnt/hitachi
        browseable = yes
        public = no
        writable = yes
        valid users = thideras
        create mode = 770
        directory mode = 770
        force user = thideras
        force group = samba

[Media]
        comment = Media
        path = /mnt/hitachi/media
        browseable = yes
        public = no
        writable = yes
        valid users = @samba
        create mode = 770
        directory mode = 770
        force user = thideras
        force group = samba

[Backups]
        comment = Backups
        path = /mnt/hitachi/backups
        browseable = yes
        public = no
        writable = yes
        valid users = thideras
        create mode = 770
        directory mode = 770
        force user = thideras
        force group = samba

[Pathways]
        comment = Pathways
        path = /mnt/hitachi/CCCApps
        browseable = yes
        public = no
        writable = yes
        valid users = thideras,pathways
        create mode = 770
        directory mode = 770
        force user = pathways
        force group = samba

[Ram]
        comment = Ram disk
        path =/mnt/ram
        browseable = yes
        public = no
        writable = yes
        valid users = thideras
        create mode = 770
        directory mode = 770
        force user = thideras
        force group = thideras

[HTPC]
        comment = HTPC Share
        writable = no
        path = /mnt/hitachi/media
        valid users = Media
        browsable = yes

[Web]
        comment = Shared Web Folder
        writable = yes
        path = /mnt/hitachi/www
        valid users = web
        browsable = yes
        create mode = 770
        directory mode = 770
        force user = web
        force group = thideras

 

[su root]

You can't password-protect NFS shares. It's a very different access control scheme, it's host-based security instead of user-based (which makes a lot more sense than allowing everyone on the system to use a remote share authenticated using a single username). Once you allow a remote system to connect to a NFS share, they rely on regular filesystem permissions only. If you have a user with UID=1000 on the client, they have equivalent access to user with UID=1000 (typically the same user) on the server. With NFSv4 and up, you have the option to Kerberos-authenticate each user, but requires a bit more setup. Root (UID=0) has a security option, where you can tell the server to treat any incoming UID=0 request as the "nobody" user.

Typically the best way to give a group of users access to the same files like this is to make them all members of the same primary group, and change their default umask to 0002. This way, any files or directories they create automatically give write permissions to the owning group (in this case, 775 or 664 are the resulting permissions).

If they need different primary groups, then make them all a part of the same group, then SetGID the directory (chmod g+s directoryname/), and make the directory owned by the common group. So if the "samba" group owns the SetGID directory "test", this means that any new file or directory created within the "test" directory will be group-owned by the "samba" group (and not the default group of the user).

So applying this to your case, do this:

chgrp -r samba /mnt/hitachi

Make sure everything is group-owned by the "samba" group. (optional)

find /mnt/hitachi -type d -exec chmod g+s "{}" \;

For every directory under /mnt/hitachi, add the SetGID option (don't set this on executable files).

Then on your clients, check your umask by running the command "umask". If it begins with "000" then you are all good. If not, look at /etc/profile for the system default. Log out & back in for the change to take effect.

You will also want to make sure that your users have the same UID across all systems. This can be tricky if you need to change a UID, and are not familiar with it.

So now, if user "suroot" creates a file or directory from a client, you will see:

[FONT="Courier New"]-rw-rw-r-- 1 suroot samba   0 Jun 11 17:47 file
drwxrwsr-x 2 suroot samba  48 Jun 11 17:47 directory[/FONT]

However, your user, being in the "samba" group will still be able to read/write/delete/etc the directories and files, because of the group permissions. The SetGID is inherited (so any directories created will automatically get the SetGID.

If you end up with files that don't respect the umask, you can set a cron job to find and fix them:

44 4 * * * root find /mnt/hitachi ! -perm /020 -exec chmod o+rw,g+rw,o+r,o-w "{}" \;

Crontab line to search for any files under /mnt/hitachi that are not group-writeable, and set the permissions to Owner=Read/Write, Group=Read/Write, Others=Read(No Write). Execute will be left alone.

 

[Automata]

Interesting and good to know. I might play with this and change up my sharing scheme. Thanks for the help, that clears up a lot!