Is my CPU holding back my mdadm raid speed?

[ziggo0]

The box is strictly a file server and my first NAS/File Server build ever. It consists of the following hardware.

AMD X2 3800+
3gb DDR2
AM3 mobo w/ 6 SATA II Ports (AHCI Mode)
PCI Gigabit Card
5x 2tb WD Green Drives in mdadm raid5
Running Ubuntu Server 10.04.3 x64

Each drive passes WD's tools. Each drive is running fine (individually tested) and performs correctly. I generally achieve 80 ~ 90mb/s write speed and 90 to 95mb/s read speed from across the network. Iperf shows that I'm getting around 840Mbits/sec, or 99.5MBytes/sec (there AND back). Jumbo frames are at 7k and my switch does support it. Here are some benchmarks to the raid it's self.

ziggo0@nas01:~$ sudo hdparm -tT /dev/md0
[sudo] password for ziggo0:

/dev/md0:
 Timing cached reads:   2228 MB in  2.00 seconds = 1114.40 MB/sec
 Timing buffered disk reads:  1170 MB in  3.00 seconds = 389.52 MB/sec

ziggo0@nas01:~$ cd /srv/storage

ziggo0@nas01:/srv/storage$ dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 10.7154 s, 100 MB/s

ziggo0@nas01:/srv/storage$ dd if=/dev/zero of=test bs=64k count=64k conv=fdatasync
65536+0 records in
65536+0 records out
4294967296 bytes (4.3 GB) copied, 39.1444 s, 110 MB/s

My problem is when writing, even reading from the array the CPU is maxed out on both cores pretty hard. This is causing some random slow down I believe, because every now and then when copying a file to the server it will get this random slow spike and gradually pick back up, but not to max speed - generally around 45 ~ 50mb/s.

Could my issue simply be that the older slower 3800+ cannot keep up with the parity calculations and is causing a choke up? I know that the CPU is a bit older...but if that is where my problems are centered around I'm considering upgrading the box to something more robust (and would serve a larger purpose than a file server).

 

[I.M.O.G.]

You probably need to look closer at local file operations, before looking too hard at operations over the network. I don't think its normal for mdadm to be causing that sort of utilization, unless the array is rebuilding or something... But I've seen crazy CPU usage for things like SMB and network related file operations.

You should benchmark local file operations, then try to do something similar over the network and compare - then you'll know where to look further.

 

[ziggo0]

Good idea, here is some info I came up with.

Copying a file from the server to a desktop results in smbd at 20% to 25%, load average at 0.26.

Copying a file from the desktop to the server results in smbd at 30%, md0_raid5 at 20% ~ 24%, flush-9:0 10% ~ 12%, load average 1.60 to 1.90.

Using dd to create a 10gb file on the raid results in md0_raid5 at about 30%, flush-9:0 at 13% ~ 16%, and dd is sitting anywhere from 25 to 75%, load average is above 2.00, sometimes dipping below.




If you have anything you can recommend that I should specifically do I would enjoy the guidance.

 

[I.M.O.G.]

Sorry, I was trying to be helpful, but I'm not sure from your data that it did any good, except maybe ruled something out.

Free and I'll be watching for further advice.

 

[ziggo0]

Haha thanks for trying. I very recently did a fresh install of the OS and noticed the usage is about 1/3 less than it used to be. I'm not sure why that would be but I think I'm going to be upgrading the box regardless.

 

[ziggo0]

Well this is interesting. I did some googling and found that stripe_cache_size by default is very low. 256 is the value. I changed it to 8192 as per others recommendation and my write speed went from 89mb/s to 255mb/s.........that is a huge increase. With this the CPU usage while writing at above gigabit speeds are threw the roof! I guess it is time for an overall upgrade.