Linux HD performance - should I see more?

Hi folks,
I'm in the process of migrating my NAS RAID (running Linux on a small dedicated system) and copying data from one drive to the other is going slower than expected. In the long run I'm more concerned about this happening correctly rather than speedily, but I'm also concerned that if something is not right now, it will not be right down the road.

At present the system boots off a laptop drive and uses a 2TB and a 3TB drive for storage. Over a year ago I replaced a 2TB Seagate that had thousands of remapped sectors with a 3TB WD Red. At the time I configured the 3TB as a 2TB match for the other drive. Since then I purchased another three TB drive and with the other Seagate starting to developed remapped sectors, it seems like a good time to put that into service. I failed and removed the 3TB from the raid, repartitioned it and created a new RAID (all are mirrored sets) in degraded mode. In other words, both RAIDs are operating on only one drive.

After formatting and mounting the new RAID I'm using rsync to copy files from one to the other. The exact command is:


This forks a couple more instances of rsync (a reader and writer I presume.)

There is about 1.6TB of data to copy and the internal transfer rate I get from the WD web site is 147 MB/s (http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-800002.pdf) An approximate calculation of the time to write the data is:
1,600,000 MB/(147*60*60) => 3 hours

It's been running over 8 hours and is less than half way done. At the rate so far, it will take about 20 hours. I understand that there will be some overhead but a factor of 7 seems excessive. Does the internal data rate not reflect the ability of the drive to write data or just read? Did I screw up my calculation? Does the overhead for the file system and RAID add that much? I would suppose of filesystem operations require the drive heads to move back and forth across the platter, that could slow things a lot. Incidentally the drive being read from is rated for sustained throughput of 95 MB/s so I suppose that will determine overall speed. Still not a factor of 7.


Here are the particulars of the system:

Both drives are formatted as ext4:

Is everything OK or do I need to start looking for problems?

Hmmm... Here is the partition table from the 'from' drive:
Starting at 1049 looks not so good, no? The drive reports a sector size of 512 but is really a 4096 byte/sector drive. Maybe not. That's the same as the 3TB drive which does report the correct physical sector size and should be correctly partitioned.

Thanks!

It's a very slow CPU, so it's possible that that's the limiting factor. In this case your CPU usage should be 100% (on at least one rsync process).

If you have a lot of small files instead of few large files, that will slow down the copy as well.

Or if the read disk is erroring and retrying.

There's an iotop utility that can show you read and write bandwidths per process, if you don't know that already.

cyberfish said:

It's a very slow CPU, so it's possible that that's the limiting factor. In this case your CPU usage should be 100% (on at least one rsync process). Possible, but I'm skeptical. The rsync processes were running about 25% so they spent a lot of time waiting on I/O.

If you have a lot of small files instead of few large files, that will slow down the copy as well. This is definitely the situation. It's a backup of working directories and with some development and code libraries mixed in, there are a lot of small files. The Android SDK is probably the worst for this and there are others.

Or if the read disk is erroring and retrying. Also possible, It had developed 7 remapped sectors and seemed to stabilize there (unlike the previous one I replaced which grew the remapped sector count several times/week.)

There's an iotop utility that can show you read and write bandwidths per process, if you don't know that already.I'll have to take a look at that. I've looked at several I/O activity monitors and never found one that seemed to tell me what's going on at a glance (like top.)

Click to expand...

Thank you for the reply. The original rsync was still running when my backups ran in the wee hours. I reran it in the morning to capture the results of the previous night's backups. This time it ran a bit more than 4 hours which seems pretty reasonable. That was mostly reading and comparing and writing a few files.

Out with the old and in with the new. I've now paired an HGST Deskstar NAS with the WD Red and it is busy resyncing.

I think. The " [2/1] [U_]" doesn't look right. Maybe that's temporary until the resync is complete.

Edit: Next afternoon and all is well. First the resync had to complete and then it ran a check. Now all seems to be settled.

One thing to consider is the difference between a Mebibye (multiples of 1024) and a Megabyte (multiples of 1000). Rsync uses Mebibytes (as per the man page) I dont know which factor the 1.6TB is or what the tool you used to bench mark the disk reports in. I suspect part of the problem is that the bench tool reports in 1000 byte chunks and that makes calculations go screwy.... especially when dealing with Gibibytes or Tebibytes.

Math aside, file systems can play a part, CPU can be a factor, raid can also play a factor as well... though I wouldn't expect a factor of 7 either

At first, I thought this thread was about video!

And the terabyters easily can do more than 95 MB/s That's measly bandwidth for such sizes, when a 320 GB comes real close. Thus, it does look like possibly a bad HDD and just have to wait it out to copy files...

The new terabyters apparently can easily do 140 MB/s and that's the platter alone!

Apparently gone are the days of needing RAID 0 to get 140 MB/s from HDDs...

I still remember the days of needing RAID 0 to get what a current single 500 GB Barracuda gets!

(For STR)

Stratus_ss said:

One thing to consider is the difference between a Mebibye (multiples of 1024) and a Megabyte (multiples of 1000). ...

Math aside, file systems can play a part, CPU can be a factor, raid can also play a factor as well... though I wouldn't expect a factor of 7 either

Click to expand...

Right - 1000 vs. 1024 would make a difference if I was splitting hairs but I was just looking at order of magnitude calculations.

RJARRRPCGP said:

At first, I thought this thread was about video!

And the terabyters easily can do more than 95 MB/s

Click to expand...

I should have used the abbreviation "HDD." I guess I was thinking "Hard Drive."

The Seagate drive is rated by Seagate "Sustained Data Rate OD (MB/s)" as 95. It's an ST32000542AS and I suppose not as fast as the latest drives. I'm not sure what the OD means. Would that be the outer cylinder? Drives used to transfer faster at the outer cylinders but that may no longer be true with the variable cylinder layouts used by modern drives.

HankB said:

The Seagate drive is rated by Seagate "Sustained Data Rate OD (MB/s)" as 95. It's an ST32000542AS and I suppose not as fast as the latest drives.

Click to expand...

Sounds like it's the slowest version, probably 5,400 RPM or 5.900 RPM.
Even those would easily be as fast as a 7,200 RPM 320 GB.