[O/C]Realities of Hard Drive Fragmentation

i'm getting ready to relax for the night.

Thanks Pettey. I will definitely look into that this weekend after my daughter goes back to her mother (it's my weekend with my daughter and i plan to enjoy every second of it with her).

I believe that in order to do this again, I need to rebuild the fragmented image to include the other defraggers listed in response to this, plus the IOZone software, i'll also leave in HDTune just to see if it really is showing the true speed difference, and I'm looking for one more benchmarking software for this, though i'm leaning towards PCMark, i'm not sure which version i should use (one of the free versions, obviously) that would measure what i'm looking for, raw speed before and after.

File-Copy Test might be worth looking at.

TollhouseFrank said:

...the built in defragmenter that comes standard with WindowsXP...

Click to expand...

There are two (not one) different disk defragmentation software programs that come with Windows XP:You should clarify which "built in" defragmentation software was used.

It is also important to know what file system was defragmentation attempted on.

The above herein this post notwithstanding, the, not so unfamiliar, latency (access time)/throughput (average read speed) correlation seems to exist. (That is, latency can be sacrificed (increased) to acquire greater throughput; throughput can be sacrificed (decreased) to acquire lesser latency)

4.2.8

MyDegrag is now 4.2.8, any chance of running it and seeing if they improved?

Thanks for the article, gave me the motivation to defrag!

AlucardCasull said:

MyDegrag is now 4.2.8, any chance of running it and seeing if they improved?

Thanks for the article, gave me the motivation to defrag!

Click to expand...

Same here.

zzzzzzzzzz said:

There are two (not one) different disk defragmentation software programs that come with Windows XP:

The Disk Defragmenter snap-in (Dedfrg.msc)
defrag.exe​

You should clarify which "built in" defragmentation software was used.

It is also important to know what file system was defragmentation attempted on.

The above herein this post notwithstanding, the, not so unfamiliar, latency (access time)/throughput (average read speed) correlation seems to exist. (That is, latency can be sacrificed (increased) to acquire greater throughput; throughput can be sacrificed (decreased) to acquire lesser latency)

Click to expand...

two different commands that lead to the same exact program as far as i can tell, which honestly, looks like a ripoff of diskeeper from about the time that XP came out.

However, just for the sake of making sure I don't miss anything, I'll use both of those in the next round of tests.

Those utilities are identical, just 2 different ways of getting to the same thing.

I haven't read it yet since it's blocked by websense on my current network, but Donn who posted earlier has some material that may be helpful, though his goals with his article were different:
http://donnedwards.openaccess.co.za/2007/09/great-defrag-shootout-winners.html

i'm gonna let this sit for the weekend. I need to unwind and destress from the week at work. Starting monday, I'll used some spare hardware here at the shop to build a machine specifically for these tests, and to build an image of said machine once I'm done so that I can get results that are outside of VirtualBox, which ought to get rid of about 50% of what most readers have led me to believe they think affects my tests the most.

The other 50% is finding software that will measure what I'm trying to measure and testing it to see if it works the way I want it to work.

I like that plan Frank, and I'm looking forward to your results.

thanks IMOG. I got a spare 80gb drive here at the shop that I can throw a shop copy of XP onto and use the bench machine to get it setup to test this again. Once I have it ready to go, I just need to find something I can use to make backup images of the drive so that I can restore it over and over.

It's one thing to make a backup of a VM. It's another to do a backup of a whole OS disk. I need to make sure that whatever I use to do the backups is free and doesn't mess with the dataset.

Using a Linux LiveCD, you could dd if=/dev/sda of=/dev/sdb bs=4096 (assuming the source drive is at /dev/sda, and you're copying to another drive, rather than a file). You could also use /dev/sda1 and /dev/sdb1 to copy just the partition (again, assuming Windows partition is /dev/sda1, and you've partitioned /dev/sdb with the same partititions), rather than the whole disk, or use a file in of= to store it to a file rather than another disk. To write it back to disk, just reverse the if and of parameters. bs=4096 tells it to copy 4K at a time.

before i do that pettey, I need to get the image built again, and have a way that i am 100% sure of to back it up and restore it so that each defragger gets the same chance with the same dataset.

That's the point of dd. Copy the disk, not the files.

Fragmentation in a file?

I hope I don't point out something someone else has already brought up, but here goes...

If you are using a VM to test with you are essentially using a single file as the virtual file system. This in effect would be similar to short stroking a drive. So if that VM file is 15 gigs on an 80 gig drive then you are going to be limited to the a theoretical max transfer speed depending on where that 15 gig file resides on the 80 gig drive... (and if you are deleting and coping it back every time you start a new test that might cause issues with fragmentation of the file itself...)

I would think in order for the test to be valid you could not use it inside a VM. Or at best if you did, you would be testing the impact of fragmentation on the VM itself. (And either way this is what needs to happen in order to form the baseline image: )

Like several others have said you have to copy the bits. Or more exactly you need to sector copy the initial image. This would give you a consistent starting point to measure from each time. If you simply copies the VM file back to the drive you may end up with file fragmentation. Or if you made an image of the system it would probably defrag itself during the creation/restore process. (I know Ghost used to work like that for a standard disc clone.) I would also install all of the test programs first to minimize changes to the initial image as much as possible.

If we going more for a copy/move/creation type of test setup, having a second drive as the frag drive might be a good idea. Instead of using/copying a whole OS, just have a secondary data drive be the fragmented one. While it wouldn't give you as tactile results as restoring the OS each time it would be somewhat easier.

Don't want to rain on any parades, but just wanted to point out that you could have fragmentation inside fragmentation....

bcsizemo said:

If you are using a VM to test with you are essentially using a single file as the virtual file system. This in effect would be similar to short stroking a drive. So if that VM file is 15 gigs on an 80 gig drive...

Click to expand...

This should not matter when the results are used only for relative comparisons. However, due to rounding or truncating decimal values, the accuracy of the relative comparisons might be decreased.

bcsizemo said:

just wanted to point out that you could have fragmentation inside fragmentation....

Click to expand...

This is something I quite dislike when working with a virtualization software VMWare Workstation.

When using a file to represent a storage device, much time is spent . When the represented information becomes fragmented, it often is necessary to defragment three times to defragment a virtual machine and its represented information in a timely manner: first the container file is defragmented, then the represented information (operating system, virtual files, etc.) are defragmented, then the container file is defragmented again (after becoming fragmented by the defragmentation of the virtual content).

A VMWare virtual machine may be set to use a physical disc instead of a file to store the virtual content.

I have never used nor have examined VirtualBox.

zzzzzzzzzz said:

When using a file to represent a storage device, much time is spent . When the represented information becomes fragmented, it often is necessary to defragment three times to defragment a virtual machine and its represented information in a timely manner: first the container file is defragmented, then the represented information (operating system, virtual files, etc.) are defragmented, then the container file is defragmented again (after becoming fragmented by the defragmentation of the virtual content).

Click to expand...

If you are using a virtual disk for your VM, you should allocate all required space on the host for the virtual disk from the start. The situation you propose could only occur if disk space is not preallocated on the host. Defragmenting a virtual disk within vmware is actually only possible if your using a growable disk.

Assuming a properly configured virtual disk on the host, there are 2 possible fragmentation scenarious when using a VM with a virtualized disk. This file can be fragmented on the host, which can be resolved by defragmenting the host system. The guest can experience fragmentation within this file, which is resolved by defragmenting from within the guest.

I.M.O.G. said:

If you are using a virtual disk for your VM, you should allocate all required space on the host for the virtual disk from the start. The situation you propose could only occur if disk space is not preallocated on the host. Defragmenting a virtual disk within vmware is actually only possible if your using a growable disk.

Click to expand...

I use fixed sized disks. You had described

I.M.O.G. said:

...there are 2 possible fragmentation scenarious when using a VM with a virtualized disk. This file can be fragmented on the host, which can be resolved by defragmenting the host system. The guest can experience fragmentation within this file, which is resolved by defragmenting from within the guest.

Click to expand...

There are three levels of fragmentation that may affect a virtual guest file system for VMWare virtual disks:

Fragmentation of the host system file system (including files to store virtual machine information). This fragmentation may be defragmented on the host system by using a defragmentation tool.

Fragmentation of the content of host file storing a guest (virtual) system. Defragmenting the contents of the host file may be performed using a utility provided by the virtualization software vendor/owner (may be incorporated into main virtualization software).

Fragmentation of the guest (virtual) system file system. This fragmentation may be defragmented by deframenting the guest (virtual) system file system from the running guest system. Defragmenting a guest file system fragments the host file storing the guest (virtual) system.

It is possible for the virtual (guest) system host file to be unfragmented on the host system, but fragmented within.

That is true for sparse/growable disks, but not for preallocated disks.

You cannot defragment a virtualdisk file on the host, unless you are using growable/sparse disks on vmware.

This document reflects the idea you have:
http://www.vmware.com/support/ws55/doc/ws_disk_defrag.html

What it does not state, is that you cannot defragment a preallocated virtual disk file:

Remember, you cannot defragment a virtual disk if you allocated all the disk space when you created the virtual disk.

Click to expand...

Source: http://www.vmware.com/support/ws55/doc/ws_disk_manager_examples.html

Additionally, notice how this defragmentation tip for vmware focuses on growable disks (or sparse virtual disks, same thing, different terminology):
http://blogs.vmware.com/teamfusion/2008/10/tip-defragmenta.html

So if you are using a sparse virtual disk, which is default in vmware fusion, then your defragmentation is a 3 part process. The only advantage of using sparse virtual disk is that your VM's will take up less space on the host system - the tradeoff is non-optimal performance and more complicated defragmentation. For anyone else who is preallocating the disk space for the VM, a 2 step defragmentation on the host and guest is all that is required.

I.M.O.G. said:

That is true for sparse/growable disks, but not for preallocated disks.

You cannot defragment a virtualdisk file on the host, unless you are using growable/sparse disks on vmware.

This document reflects the idea you have:
http://www.vmware.com/support/ws55/doc/ws_disk_defrag.html

What it does not state, is that you cannot defragment a preallocated virtual disk file:

Source: http://www.vmware.com/support/ws55/doc/ws_disk_manager_examples.html

Additionally, notice how this defragmentation tip for vmware focuses on growable disks (or sparse virtual disks, same thing, different terminology):
http://blogs.vmware.com/teamfusion/2008/10/tip-defragmenta.html

So if you are using a sparse virtual disk, which is default in vmware fusion, then your defragmentation is a 3 part process. The only advantage of using sparse virtual disk is that your VM's will take up less space on the host system - the tradeoff is non-optimal performance and more complicated defragmentation. For anyone else who is preallocating the disk space for the VM, a 2 step defragmentation on the host and guest is all that is required.

Click to expand...

The attachment to this post shows settings I usually use when creating a VMWare virtual machine in VMWare Workstation 6.5.3.

I have noticed that Virtual Machine host file is defragmentable when the options are used.

I note that I do sometimes make virtual machine snapshots. This seems to add large files for which the size of the aggregate is less than the size of the original virtual host file(s). Perhaps this may be considered a "growth".

You might want to look into using diskeeper 2010 as well. I swear by this form of De-fragmenting. But for a free solution I use Defraggler as well.. very Slow , but it works..

I never knew that there was a myth about defragging. It has always been proven since windows 98, that defragging speeds up file access and booting... Just wish there was a way to track it and test each file...for real world proof...