Ancient texts - Virtualizing CentOS 5 physical servers with LVM configuration

Related articles

Somewhere on the internet it was written:

Virtualizing a physical Linux server is a combination of knowledge, experience and luck.

Recently I had a the task for a client to virtualize a CentOS 5 Linux server. After searching on Google about it, I was disappointed to see so many posts of people having problems with it, especially those who had physical Linux servers with LVM configurations. If you research about a bit, you will be surprised on how many production CentOS 5 servers exist which run with their default installations with one large physical volume and one swap partition. The things are even worse if the boot partition is also on that same logical volume.

Unfortunately there are still many forgotten production Linux servers today which run CentOS 5. The things are even worse, because in smaller companies, those servers are set up and forgotten, and even those admins who once administered those servers are gone and not working for that company for years. What happens is when you try to virtualize Linux server with similar configuration described above with the VMware converter, the process usually terminates at 97% with error:

FAILED: An error occurred during the conversion:
'Partition number must be set for the boot volume'

Another common example is also found in this VMware KB article.


When you try to power on this crippled VM you'll end up with one of the two things.

  1. The VM will behave as it is an empty VM (just created with no OS)
  2. The system will try to boot, you will see the GRUB loading screen, and after a while when the kernel starts you will see a kernel panic message.

Both of this problems are completely solvable. Since you already have the main volume with everything inside (databases, software etc..) you just need to make sure that now you have to repair the key parts from which every Linux operates boots and operates - the boot partition.


Let's begin... and always remember:
The devil dwells in the details!

NOTE:

With the exception of a few things in this guide, most of this steps also apply if you attempt to recover any other crippled Linux system.


Things that I am taking into consideration:
  1. You are attempting to repair a CentOS 5 virtual machine. After converting it from a physical server the conversion process was terminated with one of the errors above.
  2. You know your way with VMware vSphere.
  3. The CentOS 5 was installed with the default LVM configuration, although your configuration may be different I don't see a reason why all of these steps (reproduced to match your config) might not work for you. After all you are only repairing a Linux system.
  4. Although I am going with a full example here, it will be very unlikely that your setup will be the same as mine. But the steps described here will be more or less the same in any system.
  5. Know that converting CentOS5 and even CentOS6 is not officially supported by VMware.

This is how my example configuration looks like:
[root@centos5 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       28G  3.7G  230G   1% /
/dev/sda1              99M   30M   65M  32% /boot
tmpfs                 942M     0  942M   0% /dev/shm
[root@centos5 ~]# uname -a

Linux centos5 2.6.18-406.el5 #1 SMP Tue Jun 2 17:25:57 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@centos5 ~]# ls -l /dev/VolGroup00/
total 0
lrwxrwxrwx 1 root root 31 Sep 18 02:37 LogVol00 -> /dev/mapper/VolGroup00-LogVol00
lrwxrwxrwx 1 root root 31 Sep 18 02:37 LogVol01 -> /dev/mapper/VolGroup00-LogVol01

I am simply using the above server server as a model to write this guide. As I said, if you know your stuff around linux os, then you'll understand what I am trying to do here. If you are a newbie, than it's good that you have the enthusiasm to learn. Feel free to learn, but take precautions not to brake anything even more and try to in-depth understand what each of the bellow commands does.


Steps for everything

  1. Power off the cloned VM;
  2. From the original server make note of the CentOS version and the kernel version you are converting:
    $ uname -a
  3. Obtain that same installation image. Go to http://vault.centos.org/ and find your CentOS version. If possible try to locate that same kernel which you obtained from the uname command. Make note of the URL of the kernel as you may be needing to hard type that whole URL later. You'll be working on the console of the vSphere. No SSH and all those fancy GUIs. Just a hardcore console.
  4. At the vSphere, open the VM Settings. It doesn't matter how many disks the cloned VM has. What is important here is to add two more disks. One disk with 1GB size. The other the same size or bigger as your main logical volume. In my case the logical volume was 230G so I added 250G just in case. Make the second disk thick provisioned, to avoid unnecessary complications.
  5. Position those disks exactly like this:
    • Make the smaller disk at position SCSI(0:0). This will be our /dev/sda and here will be our new /boot partition.
    • The larger disk at position SCSI(0:1). This will be our /dev/sdb and here will be the other large /(root) partition.
    • Move the cloned drive on position SCSI(0:3).
    • Take note of this as you will need to remember the positions for later when you will be formatting the disks.

  1. Apply the disc onto the VM and power on the VM with the installation CD. As soon as the VM powers on, Initiate Linux rescue mode
    Note: at this step you must be really quick. When booting the VM you have only half a second after you power on the VM, to press ESC at the post sequence in order to open the "Boot" dialog.
  2. After the recovery mode boots, you will be asked if you want the recovery installer to scan your system for existing Linux installations. Allow the installer to perform the scan and the Linux installation it finds will be mounted to /mnt/sysimage.
  3. You can use parted to initialize and format the newly added disks. Formatting disks and creating partitions is not a part of this guide. You know how to do that right? In any case if you like to learn about parted go here:
    http://www.gnu.org/software/parted/manual/html_chapter/parted_2.html
    In any case, I've made mine disks and partitions to be:
    • /dev/sda will be our new /boot partition. Make sure you make this to be flagged as boot when you use parted. Then using mkfs.ext2 format the partition new /dev/sda1 as ext2.
    • /dev/sdb will be our new / partition. Format the partition /dev/sdb1 using mkfs.ext3.
    • You can use $ fdisk -l to verify what you have done with the partitions.
    • Do not mount the partitions yet!

  1. Since we are not in a physical server any more we don't need the LVM configuration. I've attempted to fix the system with the existent LVM setup but it didn't work. There are modules missing and you will have much problems afterwards. But we don't need to use logical volumes anymore. What we need to do is to convert the logical volume to a normal partition. That process is very simple, but a long lasting one. We'll be using dd to clone the LV.
    • Using dd (direct dump), the program will copy the entire logical volume into our newly created partition and we will have our whole file system into a new single normal partition.
    • The process is simple:
      $ dd if=/dev/VolGroup00/LogVol00 of=/dev/sdb1 bs=4M
    • I've waited 3-4 hours for this process since my volume was very large.

  1. After the process is done, power off the VM and remove the original disk from the VM with the LV setup. In our case that is SCSI(0:3) disk.
    Don't remove the disk from the datastore yet. You might need this disk in case something goes wrong.
  2. Power on the VM again and go to rescue mode. You may attempt to scan the disks for existing linux installations but the scan will fail. At least it did in my case. It doesn't matter. We will mount the partition for ourselfs.
    $ mkdir /mnt/sysimage
    $ mount /dev/sdb1 /mnt/sysimage

    At this point you should be able to see the contents of /mnt/sysimage. That should be your main partition with everything inside. Check with the original physical server if there are any differences. The /boot partition on your VM will probably be empty, but everything else should be here. If something is missing, go back and find out what might have gone wrong in the steps before.

    Continue mounting the required partitions:
    $ mount /dev/sda1 /mnt/sysimage/boot

    These last three partitions are important because they pull important files for the future steps:
    $ mount --bind /dproc /mnt/sysimage/proc
    $ mount --bind /dev /mnt/sysimage/dev
    $ mount --bind /sys /mnt/sysimage/sys

  1. Chroot into our new configuration
    $ chroot /mnt/sysimage

  1. We now have to update the required files in order to reflect the new partitioning scheme of our system:
    Use vim or nan`o to do that.

Update /etc/fstab At this point probably /etc/fstab looks something like this:

$ more /etc/fstab
/dev/VolGroup00/LogVol00    /        ext3    defaults    1 1
LABEL=/boot                 /boot    ext2    defaults    1 2
tmpfs                       /dev/shm tmpfs     defaults 0 0
devpts                      /dev/pts devpts    gid=5,mode=620 0 0
sysfs                       /sys     sysfs     defaults    0 0
proc                        /proc    proc      defaults    0 0
/dev/VolGroup00/LogVol01    swap     swap      defaults    0 0

Asuming your system is identical as mine make sure that it looks like this:

$ more /etc/fstab
/dev/sdb1                   /        ext3    defaults    1 1
LABEL=/boot                 /boot    ext2    defaults    1 2
tmpfs                       /dev/shm tmpfs     defaults 0 0
devpts                      /dev/pts devpts    gid=5,mode=620 0 0
sysfs                       /sys     sysfs     defaults    0 0
proc                        /proc    proc      defaults    0 0

Don't worry about swap partition yet. Linux can work without swap just fine. You can still add swap later, but that is out of the context of this guide. You can also try and put the /boot partition with its correct name /dev/sda1 instead of LABEL=/boot.


Next go to /etc/mtab and update that file as well:

$ more /etc/mtab
/dev/sdb1 / ext3 rw 0 0
/dev/sda1 / ext2 rw 0 0

Note: The contents in this file will vary depending on your configuration. Make sure you update the context about /dev/sdb1 and /dev/sda1. If everything is done correctly when you run df -h it should output the correct partitions and sizes of your system.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             230G  171G   59G  90%          /
/dev/sda1               1G  4.6M  949M   1%      /boot
Mark the boot partition /dev/sda1 with the label /boot
e2label /dev/sda1 /boot
  1. Mount the CDRom (if you will be reinstalling from the CDROM, or you can skip this step if you will be installing from the archives)
    $ mkdir /mnt/cdrom
    $ mount /dev/hda /mnt/cdrom

Note: You don't have to use the installations from cdrom. If you have internet connection (the rescue installer will ask you to configure networking at the beginning), than you can use wget to locate and download the installation binaries that suit for you. Just make sure if you use wget to download into the /tmp directory

  1. Here you just need to make sure that you satisfy the following:
    • Reinstall the kernel (I downloaded it from the archives):
      $ cd /tmp
      $ wget http://mirror.centos.org/centos/5/centosplus/x86_64/RPMS/kernel-2.6.18-406.el5.centos.plus.x86_64.rpm
      $ rpm -ivh --force kernel-2.6.18-406.el5.centos.plus.x86_64.rpm
    • I know that it is such a pain to manually type all that URL string into the console. But hey, patience is a virtue.

    • Reinstall the GRUB (From the CDROM):
      $ cd /mnt/cdrom/CentOS
      $ rpm -Uvh --force grub-0.97-13.5.i386.rpm

    • Reinstall the RedHat-Logos (From the CDROM):
      $ rpm -Uvh --force redhat-logos-4.9.99-11.el5.centos.noarch.rpm

After these steps your boot partition should now be populated:

$ ls -l /boot
total 23966
-rw-r--r-- 1 root root   67854 Jun  2 23:57 config-2.6.18-406.el5
drwxr-xr-x 2 root root    1024 Sep 20 17:36 grub
-rw------- 1 root root 3658692 Sep 18 02:35 initrd-2.6.18-406.el5.img
-rw-r--r-- 1 root root  118892 Jun  2 23:57 symvers-2.6.18-406.el5.gz
-rw-r--r-- 1 root root 1284620 Jun  2 23:57 System.map-2.6.18-406.el5
-rw-r--r-- 1 root root 2127980 Jun  2 23:57 vmlinuz-2.6.18-406.el5
  1. Install the GRUB bootloader
    $ grub-install /dev/sdb
    Note: Sometimes that could fail with the error: Could not locate device for
    In that case you can use this procedure to install the grub from the grub shell. To do that you have to first exit the chroot-ed enviorment.

  1. Now we have to create our GRUB config file. Understanding the GRUB and its associated files is also out of the scope of this tutorial. Use vim or nano to create your grub.conf file which should be located at /boot/grub/grub.conf. The grub file bellow is the my grub file and merely an example. Take note of the drives (sda1 should be hd0,0) make sure to match yours according to your distribution, kernel version etc.:
$ cd /boot/grub
$ more grub.conf
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-406.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-406.el5 ro root=/dev/sdb1 rhgb quiet 
        initrd /initrd-2.6.18-406.el5.img
  1. At the end we need to create a simlink to our grub.conf file.
    $ cd /boot/grub
    $ ln -s grub.conf menu.lst

  1. Reboot your server. Press ESC to interupt the POST and enter the BIOS (Enter Setup). Go to the Boot section and adjust the drives in that order so that the SCSI(0:0) is the first drive to attempt booting. We made SCSI(0:0) to be our boot /sda1 partition, remember? After that reboot. If everything is done correctly your cloned VM should boot into the GRUB bootloader and than start the kernel and continue with the boot process.
Hey?! WTF? Kernel Panic after all this trobule?

Yes. A number of times that could happen even after all of the steps above. If you receive kernel panic even after this process than you have to regenerate the initrd image.
Use this article from the CentOS Tips and Tricks to regenerate a new initrd image. After that your system should boot fine.

Creating a New Initial RAM Disk


Hey, but I've done all that and my system still doesn't boot :((((

Well than, bad luck for you. Try to do all of this again and be more careful. Learn what each of those steps does in detail. As I said at the beginning - Converting physical Linux servers to virtual is not yet fully supported by the vendors.


Even so CentOS 5 P2V is not officially supported by VMware. That is why the process to do this is so long and daunting and not always worth the effort.


At the end however, a broken server, the knowledge and the newly acquired experience may be the only things you could be left with.


Resources used to put all of this stuff together: