I've never been so exited about a hardware failure in my server room as I was last week. I'm running a VMware Virtual Infrastructure 3 cluster. Really, it's a pair of Dell PowerEdge 1950 servers running ESX with all the goodies turned on (HA, DRS, VMotion, etc). At 9:45PM one of the PE1950s crashed hard with some sort of BIOS/CPU problem. There were 11 or 12 virtual servers running on this host. Because of the VMware High Availability features, I didn't notice the failure until late the next day when the Network Admin noticed an amber error light on the front panel. All the virtual servers had quickly rebooted on the other ESX server and went back to work as if nothing had happened. It was so cool. This is how HA is suppose to work. So I took my time getting the bios/machine code on the dead PE1950 patched up. Once I got it fixed and added it back to the cluster, the DRS feature automatically migrated some VMs over to it to balance the cluster.
I'm really glad I took the time to do a proper High Availability design in my latest generation of server infrastructure. Higher uptime and lower stress for the win!!
Tuesday, January 22, 2008
Wednesday, January 09, 2008
VMWare ESX disk-fu
In my VMware VirtualCenter cluster I have a number of Linux VMs running CentOS5. When I set them up originally, I gave them 512MB physical ram and 768MB swap. The swap is configured as a logical volume on the system disk, which is a virtual disk file in ESX. This disk is has two partitions, /boot, and an LVM pv.
I need to run some large rsync processes on these VMs, so I need to "upgrade" the ram to 2G and wanted to add swap space as well in order to maintain the same physical:swap ratio. I could have just added another virtual disk and added that to the volume group. But I really wanted expand the original virtual disk, and the PV on it. Here is how I did this.
1) Shutoff the VM and expand the virtual disk. On my esx server, I issued these commands:
# vmare-cmd machinename.vmx stop
# vmkfstools -X 8224M machinename.vmdk
# vmware-cmd machinename.vmx start
All this does is tell esx to grow the virtual disk file to the requested size. Note that the -X flag takes the desired size of the disk, not the amount by which to grow the disk.
2) Grow the partition in the virtual disk used as the system pv. This was a bit trickier. I needed to move the end of the second partition, which has type 8e (lvm) to the new end of the disk. Fdisk and Sfdisk won't let your move the end of a partition and parted will only move a partition for which it can understand the filesystem. Since parted doesn't know lvm, that was out. The solution was to use fdisk and delete and recreate the partition. This sounds destructive. Because the partition is the last on on the disk and we are recreating it's entry in the partition table almost identical to the old entry it works. You would not be able to do this with a partition not at the end of the disk. Also, you'd have to be very careful if it was an extended partition. I'm not sure of the implications of that. After I recreated the partition table entry with a new end of disk, I rebooted so the kernel would know about the changes.
3) Tell lvm about the disk changes. First grow the physical volume:
# pvresize /dev/sda2
This grows the PV, creating new PEs in the volume group, now just grow the volume.
# lvextend -l +72 /dev/vg0/swap
The -l (lowercase Ell) tells the volume to grow by the given number of physical extents.
4) Finally, remake the swap space so it picks up the new space:
# swapoff /dev/vg0/swap
# mkswap /dev/vg0/swap
# swapon /dev/vg0/swap
This method worked well, but was very specific to my configuration. I'm glad I was able to do it without adding new virtual disks or volumes to my VM.
I need to run some large rsync processes on these VMs, so I need to "upgrade" the ram to 2G and wanted to add swap space as well in order to maintain the same physical:swap ratio. I could have just added another virtual disk and added that to the volume group. But I really wanted expand the original virtual disk, and the PV on it. Here is how I did this.
1) Shutoff the VM and expand the virtual disk. On my esx server, I issued these commands:
# vmare-cmd machinename.vmx stop
# vmkfstools -X 8224M machinename.vmdk
# vmware-cmd machinename.vmx start
All this does is tell esx to grow the virtual disk file to the requested size. Note that the -X flag takes the desired size of the disk, not the amount by which to grow the disk.
2) Grow the partition in the virtual disk used as the system pv. This was a bit trickier. I needed to move the end of the second partition, which has type 8e (lvm) to the new end of the disk. Fdisk and Sfdisk won't let your move the end of a partition and parted will only move a partition for which it can understand the filesystem. Since parted doesn't know lvm, that was out. The solution was to use fdisk and delete and recreate the partition. This sounds destructive. Because the partition is the last on on the disk and we are recreating it's entry in the partition table almost identical to the old entry it works. You would not be able to do this with a partition not at the end of the disk. Also, you'd have to be very careful if it was an extended partition. I'm not sure of the implications of that. After I recreated the partition table entry with a new end of disk, I rebooted so the kernel would know about the changes.
3) Tell lvm about the disk changes. First grow the physical volume:
# pvresize /dev/sda2
This grows the PV, creating new PEs in the volume group, now just grow the volume.
# lvextend -l +72 /dev/vg0/swap
The -l (lowercase Ell) tells the volume to grow by the given number of physical extents.
4) Finally, remake the swap space so it picks up the new space:
# swapoff /dev/vg0/swap
# mkswap /dev/vg0/swap
# swapon /dev/vg0/swap
This method worked well, but was very specific to my configuration. I'm glad I was able to do it without adding new virtual disks or volumes to my VM.
Tuesday, December 04, 2007
In Exile
Well Tweaker.TV finally expired with the registrar (register.com) and I don't know if I can make myself pay the Fifty Bucks again for a Tuvalu domain. So I think I'll try posting here and be another independent sucked into the black hole that is Google. To many domains, too little time.
-Tweak Out
-Tweak Out
Subscribe to:
Posts (Atom)
