fsck on KVM raw device disks

If you have a small company resource optimization is a must. Therefore, I use KVM virtualization to provide several (virtual) servers on a KVM host. The hard disks are provided as raw devices under the control of the KVM "virtio" driver. My virtual machines have been running very reliably for two years now.

However, some days ago one of the servers - a LAMP server called "vm1" - could not be booted any more. Not even its virtual console displayed any messages - the boot process stopped at the very beginning. I took this as a sign to check for corruptions of the file system. In the case of "vm1" the raw device is located on a LVM partition "/dev/volgrpk/vm1_hd1" of a LVM volume group on the KVM host. The internal structure of this device consists of a SWAP partition plus a data partition. The latter one gets mounted on "/" on the virtual guest server.

Because it was such a long time ago that I had to use fsck on a virtual device I had forgotten that you cannot deal directly with KVM raw devices from the KVM host. My naive trial to use fsck directly produced warnings as the KVM host cannot handle KVM raw image devices :

kvm-host:~ # fsck /dev/volgrpk/vm1_hd1 
fsck from util-linux 2.23.2
e2fsck 1.42.8 (20-Jun-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/mapper/volgrp1-vm1_hd1

So, I needed a tool which provides access to the partitions of a KVM raw device with some disk image on it.

kpartx to access partitions on KVM raw devices

The easiest way to access a raw device from the KVM host is to use "kpartx". For Opensuse a relatively up to date RPM is provided in the Update repository "http://download.opensuse.org/update/13.1/".

For the usage of kpartx see :
KVM-Host – Mounten “virtueller” Gast-Partitionen aus logischen LVM-Volumes
http://linux.die.net/man/8/kpartx
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Troubleshooting_Xen-Accessing_data_on_guest_disk_image.html

Before you apply kpartx you should have a look at "/proc/partitions/" to be able to distinguish the newly attached virtual partitions later on. In my case I get something like:

kvm-host:~ # cat /proc/partitions           
major minor  #blocks  name

   8        0 3906228224 sda
   8        1    2088450 sda1
   8        2  104856576 sda2
   8        3  125829112 sda3
   8        4  125837312 sda4
   8        5  104855552 sda5
   8        6  104856576 sda6
   8        7    2096128 sda7
   8        8  629145600 sda8
   8        9  419433472 sda9
   8       10  146809856 sda10
  11        0    1048575 sr0
 253        0  125829120 dm-0
 253        1   62914560 dm-1
 253        2   62914560 dm-2
 253        3   83886080 dm-3
 253        4   41943040 dm-4
 253        5  104857600 dm-5
 253        6  188743680 dm-6
 253        7  104857600 dm-7
  43        0   62914560 nbd0

Now, we first make sure that the KVM guests supposed to use your raw image devices are offline. More precisely: ... that any KVM guests using image partitions on the raw device you want to access by kpartx are offline. I avoid sharing or NFS-, SMB-mounting partitions on raw devices - then it is clear which KVM guests to stop.

To get access to the partitions of a raw device we use kpartx with the option "-a". In the following example this is shown for for two raw devices used by two different virtual servers:

kvm-host:~ # kpartx -a /dev/volgrpk/vm1_hd1 
kvm-host:~ # kpartx -a /dev/volgrpk/vm2_hd1 
kvm-host:~ # cat /proc/partitions
major minor  #blocks  name

   8        0 3906228224 sda
   8        1    2088450 sda1
   8        2  104856576 sda2
   8        3  125829112 sda3
   8        4  125837312 sda4
   8        5  104855552 sda5
   8        6  104856576 sda6
   8        7    2096128 sda7
   8        8  629145600 sda8
   8        9  419433472 sda9
   8       10  146809856 sda10
  11        0    1048575 sr0
 253        0  125829120 dm-0
 253        1   62914560 dm-1
 253        2   62914560 dm-2
 253        3   83886080 dm-3
 253        4   41943040 dm-4
 253        5  104857600 dm-5
 253        6  188743680 dm-6
 253        7  104857600 dm-7
  43        0   62914560 nbd0
 253        8    2095104 dm-8
 253        9   60818432 dm-9
 253       10    2095104 dm-10
 253       11   60818432 dm-11
kvm-host:~ # file -s /dev/dm-8
/dev/dm-8: Linux/i386 swap file (new style), version 1 (4K pages), size 523775 pages, no label, UUID=23b50bc8-dbeb-4b2d-be10-75b9f390377e
kvm-host:~ # file -s /dev/dm-9
/dev/dm-9: DOS/MBR boot sector
kvm-host:~ # file -s /dev/dm-10
/dev/dm-10: Linux/i386 swap file (new style), version 1 (4K pages), size 523775 pages, no label, UUID=83220483-e419-42d8-88b6-e6dfe202aebc
kvm-host:~ # file -s /dev/dm-11
/dev/dm-11: DOS/MBR boot sector; GRand Unified Bootloader, stage1 version 0x3, LBA flag 0x1, 1st sector stage2 0x1448200, GRUB version 0.97
kvm-host:~ #

Clearly four additional partitions dm-8 until dm-11 are recognized. Two of them are SWAP partitions. "/dev/dm-9" and "/dev/dm-11" are data partitions each of which is mounted on "/" on my virtual KVM guest servers.

The dm-9 partition is the relevant one for the planned fsck check, which we now can perform in the usual way:

kvm-host:~ # fsck -f  /dev/dm-9
fsck from util-linux 2.23.2
e2fsck 1.42.8 (20-Jun-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/volgrpk-vm1_hd1p2: 444945/3808512 files (0.2% non-contiguous), 14281802/15204352 blocks

By the way: A look into "/dev/mapper" shows the available devices and partitions and their device names automatically provided by kpartx:

kvm-host:~ # ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Oct 27 07:46 control
....
....
lrwxrwxrwx 1 root root       7 Oct 27 12:09 volgrpk-vm1_hd1 -> ../dm-1
lrwxrwxrwx 1 root root       7 Oct 27 12:10 volgrpk-vm1_hd1_part1 -> ../dm-8
lrwxrwxrwx 1 root root       7 Oct 27 12:21 volgrpk-vm1_hd1_part2 -> ../dm-9
lrwxrwxrwx 1 root root       7 Oct 27 12:10 volgrpk-vm1_hd1p1 -> ../dm-8
lrwxrwxrwx 1 root root       7 Oct 27 12:21 volgrpk-vm1_hd1p2 -> ../dm-9
lrwxrwxrwx 1 root root       7 Oct 27 08:43 volgrpk-vm2_hd1 -> ../dm-2
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1_part1 -> ../dm-10
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1_part2 -> ../dm-11
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1p1 -> ../dm-10
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1p2 -> ../dm-11
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd0 -> ../dm-3
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd1 -> ../dm-4

I omitted some lines and devices in the list above. The last 2 entries stem from a running virtual IMAP server. Note that one and the same device may be presented with different names. again
Now, we can test our virtual server again by booting it - which in my case worked perfectly ...

Before testing the virtual LAMP sever with the repaired file system we should not forget to detach the devices still controlled by kpartx:

kvm-host:~ # kpartx -d /dev/volgrp1/vm1_hd1 
kvm-host:~ # kpartx -d /dev/volgrp1/vm2_hd1

A look into "/dev/mapper" again shows that the raw image disk devices are still shown. However, the partitions have disappeared.

kvm-host:~ # ls -l /dev/mapper 
total 0
crw------- 1 root root 10, 236 Oct 27 07:46 control
lrwxrwxrwx 1 root root       7 Oct 27 13:18 volgrpk-vm1_hd1 -> ../dm-1
lrwxrwxrwx 1 root root       7 Oct 27 08:43 volgrpk-vm2_hd1 -> ../dm-2
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd0 -> ../dm-3
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd1 -> ../dm-4 

Now, we can test our virtual server (here vm1) by booting it - which worked perfectly in my case ...