how to make a drive image.

[Stratus_ss]

suppose we just have different ways of doing things.

MRD's way is good for a single migration

what I would do is this,

1) either using the command line or something like GParted, shrink the root partition so there is little or no unused space (you may need a gparted boot disk, or if you go the command line route, i prefer gentoo minimal disk)
2) dd if=/dev/your_hardrive of=my_boot_sector_image bs=1M count=100 (so for me it would be dd=/dev/sda of=boot.sda bs=1M count=100)
3) dd if=/old/partition of=my_old_partition.image (dd if=/dev/sda5 of=root_partition.image because my root is on sda5)

on the new hard drive

4) dd if=/location/of/boot_image of=/dev/sba (so for me it would be dd if=/temp/boot.sda of=/dev/sdb)
5) mkdir /temp_ new
6) mount /dev/your_root_partition /temp (mount /dev/sdb5 /temp)
8) dd if=/location/root_image of=/temp (dd if=/temp/root_partition.image of=/temp)
9) e2fsck -f /dev/sdb5 && resize2fs -p /dev/sdb5

viola your done. In some cases you may also have to rebuild grub, depending on whether you want your new harddrives unused space all thrown into the root partition. My preference is to have a partition for /boot one for / and one for /home with the /home getting all the extra space

Now for a single migration MRD's is probably a bit quicker and may, in this case be more advantageous for you. However, for sake of completeness for those who may follow this thread, for long term ease of use, the DD way stores your image. Burn it off and dont worry if your hard drive takes a dump on you

 

[MRD]

I don't see how your method is faster even for a large migration... it still requires significant manual work, and the unnecessary step of shrinking and growing partitions. Also, your method alters the source disk, which is always riskier than mounting the source read-only, especially for a relative newbie who is afraid he might lose his data. All that partition resizing is asking for trouble and totally unnecessary. Also, low level copies are much slower than high level copies (dd vs cp). Finally, you don't deal with GRUB at all, and you cannot just copy a partition table between totally different drives with different geometries, you must repartition.

dd would be better for migrating windows or other proprietary os's that are picky about where on the drive each file is physically located, but I can't see why on earth you'd do this for Linux.

 

[Stratus_ss]

well first of all because this is the way that i am told to do it

secondly it was designed for deployment over a network. I see your point about how low level copies are slower, but unless you are doing say 40 gigs (which i suppose is possible) it shouldnt make that much of a difference.

Thirdly i'm not sure if your method would work with LVMs which is way outside of the scope of the OP, but that is how i would set things up.

and finally, while the risk is there of dumping data while resizing partitions, I havent encountered it yet (unless of course someone kicks a cord or something and then you are in trouble)

after re-reading the OP I think i will defer to MRD's suggestion for the purpose of this post... in the interest of KISS his way is superior. Just not the way I would do it

 

[MRD]

Yes, it would work with LVM's, except that you might have to partition differently. (You don't use fdisk to partition with LVM iirc...) I forget exactly how I set up my LVM2 setup... it was a while ago.

If I were doing network deployment, I'd make tarballs and then untar those for deployment rather than copying from an actual partition (it's faster anyways, since the compression speeds up the transfer of information, unless of course you have a slow cpu and a fast network connection, then it may be not... and compression can be used with dd too if you want... you just pipe it through one of the compression programs).

Doing a full dd of even about 20gb over a gig-e wired connection can easily take 4-16 hours, depending on a number of factors.

I'll be doing a lot of copying in the near future (getting rid of my large number of disks forming my LVM2 file server disk and replacing it with a 1 TB SATA II and setting up 3 1 TB disks in a RAID 5 array on a backuppc server. I was going to work on it this weekend, if only ZZF were as fast as Newegg at shipping... Why is it that Newegg 3 day shipping is always 1 day, and everyone ZZF's is a week?

 

[th3]

Have moved a few Linux installs myself and totally agree with MRD. cp -ax will save you lots of time, i would not consider dd for this purpose.

 

[Stratus_ss]

Yes, it would work with LVM's, except that you might have to partition differently. (You don't use fdisk to partition with LVM iirc...) I forget exactly how I set up my LVM2 setup... it was a while ago.

If I were doing network deployment, I'd make tarballs and then untar those for deployment rather than copying from an actual partition (it's faster anyways, since the compression speeds up the transfer of information, unless of course you have a slow cpu and a fast network connection, then it may be not... and compression can be used with dd too if you want... you just pipe it through one of the compression programs).

Doing a full dd of even about 20gb over a gig-e wired connection can easily take 4-16 hours, depending on a number of factors.

I'll be doing a lot of copying in the near future (getting rid of my large number of disks forming my LVM2 file server disk and replacing it with a 1 TB SATA II and setting up 3 1 TB disks in a RAID 5 array on a backuppc server. I was going to work on it this weekend, if only ZZF were as fast as Newegg at shipping... Why is it that Newegg 3 day shipping is always 1 day, and everyone ZZF's is a week?

might i enquire as to what those factors may be? I do not have nearly the experience that you do on such things just what I do on the job. When we do 8 gigs it takes maybe 5-8 mins over gig-e which would put the math at roughly 25-40 mins for 40 gigs.

I am just curious what sort of things would see that much of a drastic slowdown

 

[MRD]

Use or non use of compression, algorithm and level of compression, speed of network connection (wired or wireless, 10/100/1000/10000), size of ethernet frames used (jumbo can speed things up if supported from end to end), pci latency, system bus type/speed, speed of disk drives, speed of disk drive interfaces, network congestion, length and quality of ethernet cabling (fiber, cat5, cat5e, cat6) and number of relay points (switches/routers/repeaters), use of raid or lvm, speed of processors on each end, configuration of the TCP stack and send/receive buffers (sysctl stuff), method of transferring the dd info (nfs, netcat, etc.), bad sectors on the drive, compressibility of data and type of data (bzip2 won't do much with a jpeg, but it'll do a lot on a big file full of 0's), cpu speed on each end, how much verification you do (checksumming etc), offloading of checksumming to net cards, and lots of other things.

gig-e is virtually never really gig-e. The theoretical max is about 100MB/s, maybe a tiny bit more, of actual, useful data transferred. So in theory, gig-e can move a gigabyte in about 10s or so, which would move 10 gigabytes in about 1m 40s... but this never actually happens... or even anything close to it.

Low level disk access generally eliminates buffering, which really slows things down.

Note that many of the above factors apply to any means of moving data over a network, including cp, while some apply exclusively, or more strongly, to dd.

Another thing to note is that if there is any filesystem corruption or fragmentation (and yes, all filesystems fragment to some degree, including Linux... just not as badly as Windows filesystems, especially FAT32), dd will copy this, while cp will remove any filesystem corruption (new filesystem) and will defragment the files as it will write each sequentially. cp also lets you change filesystems easily and without risk if you want to.

 

[splat]

use rsync http://linux.die.net/man/1/rsync

 

[Zerix01]

Only if the drives are identical in size. It can clone from one to another, but you can't really clone a 40gb to a 200gb with it.

Why not? You just need to expand the file system afterward to fill the rest of the disk.

 

[MRD]

It's not quite that simple. The partition table copied over will also not work right, and will be basd on incorrect physical geometry parameters. You can pobably make it work... but it's not exactly how it's supposed to work.

 

[Zerix01]

It's not quite that simple. The partition table copied over will also not work right, and will be basd on incorrect physical geometry parameters. You can pobably make it work... but it's not exactly how it's supposed to work.

If you copy from one partition to another rather than copying the whole disk with all partitions, such as dd if=/dev/sda1 of=/dev/sdb1 vs dd if=/dev/sda of=/dev/sdb I don't think it will copy any partition data, only the file system contained in that partition. At that point if you use what ever utility needed to expand that file system to fill the partition then the appropriate data will be correctly updated.

But like mentioned before, it might just be easier to use cp to copy all the data from one to the next.

 

[MRD]

It doesn't copy the partition table itself if you use only cp, but that's the point. Unless you really know what you are doing (like the details of how a partition table is stored on a PC), you could corrupt it by copying over the partition table and then expanding things. It might look ok and blow up later, or it might just look completely wrong. The safe and clean way to do it is to repartition using fdisk and copy from the old to the new partition. You might even do an ls and mount the filesystem and it will look fine, but some data will be missing.

If you have 2 identical drives, then dd works great, but this is rarely the case. (I don't mean 2 80GB drives, I mean 2 80GB drives of the exact same model number from the same manufacturer with identical physical geometry patterns.)

 

[Stratus_ss]

If you have 2 identical drives, then dd works great, but this is rarely the case. (I don't mean 2 80GB drives, I mean 2 80GB drives of the exact same model number from the same manufacturer with identical physical geometry patterns.)

which just happens to be the case in our (and most) mass deployment situations. But i suppose to the OP since its a harddrive migration, it wouldnt really matter too much either way. Apparently cp works fine so thats probably easiest

 

[MRD]

It's not coincidence that most businesses doing mass deployments want a large quantity of identical machines. They don't want to spend time configuring each machine manually.

 

[gsrcrxsi]

yea, i just got my new drive so im going to be attempting this soon. i think ill follow your steps for cp. im going from a 320gb to a 640gb, but im gonna break it up into 3 200gb partitions i think

 

[MRD]

The other advantage of cp is a lot of flexibility in placement and sizing of partitions.

 

[gsrcrxsi]

the only thing im wondering how is how to migrate my windows partition. i will be moving from a 100GB to 200GB

 

[MRD]

I think this should work, but keep the old drive unaltered until you know it does.

Just use dd to copy the old partition to a new partition (note that dd can take as if/of options either a device name, e.g. /dev/hda, or partition names, e.g. /dev/hda1... these mean totally different things). You can set up the partitions using fdisk in Linux, define the type as NTFS/HPFS, make the partition primary, and flag it as bootable.Then use gparted to enlarge the ntfs filesystem to fill the whole partition (I think this is possible...). Finally, set up your boot loader on the mbr.

Partitions have to have the same number/device, as Windows is VERY picky about this and inflexible and will get seriously borked if you change them (it's hard coded). Data partitions don't matter so much, but OS partitions, and even partitions where programs reside, do matter a lot.

 

[gsrcrxsi]

im following your instructions, (with slight modification for easy-ness) for linux

what ive done:

using GParted - made all partions that i want on the new drive
-sdd1 - ext3 bootable
-sdd2 - extended - sdd5 - swap
-sdd3 - ntfs bootable
-sdd4 - xfs (storage)

im about to start copying the data from my old ext3 partiton to my new one on the new drive, but after i do that, wont i have to edit the fstab file? because afaik, and what it shows in my current fstab file is that it mounts my file system by mounting the HDD's UUID to the / mount point. to me this says that i will have to edit the fstab file on the new drive to the new drives UUID, otherwise it wouldnt boot. correct?

 

[gsrcrxsi]

huston, we have a problem

i edited the fstab file, (it makes sense to me to do so). load it up, didnt boot. i popped in my super grub disc. and booted it, but it just hangs on the loading splash screen. and never loads.

any ideas?