fsck on KVM raw device disks

If you have a small company resource optimization is a must. Therefore, I use KVM virtualization to provide several (virtual) servers on a KVM host. The hard disks are provided as raw devices under the control of the KVM “virtio” driver. My virtual machines have been running very reliably for two years now.

However, some days ago one of the servers – a LAMP server called “vm1” – could not be booted any more. Not even its virtual console displayed any messages – the boot process stopped at the very beginning. I took this as a sign to check for corruptions of the file system. In the case of “vm1” the raw device is located on a LVM partition “/dev/volgrpk/vm1_hd1” of a LVM volume group on the KVM host. The internal structure of this device consists of a SWAP partition plus a data partition. The latter one gets mounted on “/” on the virtual guest server.

Because it was such a long time ago that I had to use fsck on a virtual device I had forgotten that you cannot deal directly with KVM raw devices from the KVM host. My naive trial to use fsck directly produced warnings as the KVM host cannot handle KVM raw image devices :

kvm-host:~ # fsck /dev/volgrpk/vm1_hd1 
fsck from util-linux 2.23.2
e2fsck 1.42.8 (20-Jun-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/mapper/volgrp1-vm1_hd1

So, I needed a tool which provides access to the partitions of a KVM raw device with some disk image on it.

kpartx to access partitions on KVM raw devices

The easiest way to access a raw device from the KVM host is to use “kpartx”. For Opensuse a relatively up to date RPM is provided in the Update repository “http://download.opensuse.org/update/13.1/“.

For the usage of kpartx see :
KVM-Host – Mounten “virtueller” Gast-Partitionen aus logischen LVM-Volumes
http://linux.die.net/man/8/kpartx
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Troubleshooting_Xen-Accessing_data_on_guest_disk_image.html

Before you apply kpartx you should have a look at “/proc/partitions/” to be able to distinguish the newly attached virtual partitions later on. In my case I get something like:

kvm-host:~ # cat /proc/partitions           
major minor  #blocks  name

   8        0 3906228224 sda
   8        1    2088450 sda1
   8        2  104856576 sda2
   8        3  125829112 sda3
   8        4  125837312 sda4
   8        5  104855552 sda5
   8        6  104856576 sda6
   8        7    2096128 sda7
   8        8  629145600 sda8
   8        9  419433472 sda9
   8       10  146809856 sda10
  11        0    1048575 sr0
 253        0  125829120 dm-0
 253        1   62914560 dm-1
 253        2   62914560 dm-2
 253        3   83886080 dm-3
 253        4   41943040 dm-4
 253        5  104857600 dm-5
 253        6  188743680 dm-6
 253        7  104857600 dm-7
  43        0   62914560 nbd0

Now, we first make sure that the KVM guests supposed to use your raw image devices are offline. More precisely: … that any KVM guests using image partitions on the raw device you want to access by kpartx are offline. I avoid sharing or NFS-, SMB-mounting
partitions on raw devices – then it is clear which KVM guests to stop.

To get access to the partitions of a raw device we use kpartx with the option “-a”. In the following example this is shown for for two raw devices used by two different virtual servers:

kvm-host:~ # kpartx -a /dev/volgrpk/vm1_hd1 
kvm-host:~ # kpartx -a /dev/volgrpk/vm2_hd1 
kvm-host:~ # cat /proc/partitions
major minor  #blocks  name

   8        0 3906228224 sda
   8        1    2088450 sda1
   8        2  104856576 sda2
   8        3  125829112 sda3
   8        4  125837312 sda4
   8        5  104855552 sda5
   8        6  104856576 sda6
   8        7    2096128 sda7
   8        8  629145600 sda8
   8        9  419433472 sda9
   8       10  146809856 sda10
  11        0    1048575 sr0
 253        0  125829120 dm-0
 253        1   62914560 dm-1
 253        2   62914560 dm-2
 253        3   83886080 dm-3
 253        4   41943040 dm-4
 253        5  104857600 dm-5
 253        6  188743680 dm-6
 253        7  104857600 dm-7
  43        0   62914560 nbd0
 253        8    2095104 dm-8
 253        9   60818432 dm-9
 253       10    2095104 dm-10
 253       11   60818432 dm-11
kvm-host:~ # file -s /dev/dm-8
/dev/dm-8: Linux/i386 swap file (new style), version 1 (4K pages), size 523775 pages, no label, UUID=23b50bc8-dbeb-4b2d-be10-75b9f390377e
kvm-host:~ # file -s /dev/dm-9
/dev/dm-9: DOS/MBR boot sector
kvm-host:~ # file -s /dev/dm-10
/dev/dm-10: Linux/i386 swap file (new style), version 1 (4K pages), size 523775 pages, no label, UUID=83220483-e419-42d8-88b6-e6dfe202aebc
kvm-host:~ # file -s /dev/dm-11
/dev/dm-11: DOS/MBR boot sector; GRand Unified Bootloader, stage1 version 0x3, LBA flag 0x1, 1st sector stage2 0x1448200, GRUB version 0.97
kvm-host:~ #

Clearly four additional partitions dm-8 until dm-11 are recognized. Two of them are SWAP partitions. “/dev/dm-9” and “/dev/dm-11” are data partitions each of which is mounted on “/” on my virtual KVM guest servers.

The dm-9 partition is the relevant one for the planned fsck check, which we now can perform in the usual way:

kvm-host:~ # <strong>fsck -f  /dev/dm-9</strong>
fsck from util-linux 2.23.2
e2fsck 1.42.8 (20-Jun-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/volgrpk-vm1_hd1p2: 444945/3808512 files (0.2% non-contiguous), 14281802/15204352 blocks

By the way: A look into “/dev/mapper” shows the available devices and partitions and their device names automatically provided by kpartx:

kvm-host:~ # ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Oct 27 07:46 control
....
....
lrwxrwxrwx 1 root root       7 Oct 27 12:09 volgrpk-vm1_hd1 -> ../dm-1
lrwxrwxrwx 1 root root       7 Oct 27 12:10 volgrpk-vm1_hd1_part1 -> ../dm-8
lrwxrwxrwx 1 root root       7 Oct 27 12:21 volgrpk-vm1_hd1_part2 -> ../dm-9
lrwxrwxrwx 1 root root       7 Oct 27 12:10 volgrpk-vm1_hd1p1 -> ../dm-8
lrwxrwxrwx 1 root root       7 Oct 27 12:21 volgrpk-vm1_hd1p2 -> ../dm-9
lrwxrwxrwx 1 root root       7 Oct 27 08:43 volgrpk-vm2_hd1 -> ../dm-2
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1_part1 -> ../dm-10
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1_part2 -> ../dm-11
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1p1 -> ../dm-10
lrwxrwxrwx 1 root root       8 Oct 27 12:11 volgrpk-vm2_hd1p2 -> ../dm-11
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd0 -> ../dm-3
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd1 -> ../dm-4

I omitted some lines and devices in the list
above. The last 2 entries stem from a running virtual IMAP server. Note that one and the same device may be presented with different names. again
Now, we can test our virtual server again by booting it – which in my case worked perfectly …

Before testing the virtual LAMP sever with the repaired file system we should not forget to detach the devices still controlled by kpartx:

kvm-host:~ # kpartx -d /dev/volgrp1/vm1_hd1 
kvm-host:~ # kpartx -d /dev/volgrp1/vm2_hd1

A look into “/dev/mapper” again shows that the raw image disk devices are still shown. However, the partitions have disappeared.

kvm-host:~ # ls -l /dev/mapper 
total 0
crw------- 1 root root 10, 236 Oct 27 07:46 control
lrwxrwxrwx 1 root root       7 Oct 27 13:18 volgrpk-vm1_hd1 -> ../dm-1
lrwxrwxrwx 1 root root       7 Oct 27 08:43 volgrpk-vm2_hd1 -> ../dm-2
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd0 -> ../dm-3
lrwxrwxrwx 1 root root       7 Oct 27 07:46 volgrpk-vm_imap_hd1 -> ../dm-4 

Now, we can test our virtual server (here vm1) by booting it – which worked perfectly in my case …

Hosted Strato server – backups and chained firewall tunneling via SSH and shell

I administer several Linux web servers for partner and customer companies. Most of these servers are V-servers hosted at the German Telekom daughter Strato.

[I have described some reasonable security measures for a Strato server with Opensuse in some articles in this blog. See e.g.:
Strato-V-Server mit Opensuse 12.3 – IV – SSH Key Authentication
Strato-V-Server mit Opensuse 12.3 – III – Stop SSH Root Login
and previous articles referenced there.]

When you work with a hosted server, of course, you want to perform periodic backups. If you – for whatever reasons – do not want to pay for the backup facilities Strato offers, a backup requires that you transfer data from the remote server into your own company network through (encrypted) SSH tunnels. Depending on your specific situation you may wish to create a complete backup or a backup of selected directories on the server. During the backup you want to preserve user-rights and mode settings.

In addition to the topic of creating tunnels across firewalls, the question arises what Linux tool might be convenient to perform the backup task if you work with a shell based connection only. Therefore, I shall briefly discuss what I normally do to perform a manually controlled backup from a hosted Strato server. You can combine the methods described of course with some scripting. The described firewall tunneling by SSH is independent of a specific provider.

Tunneling firewalls with SSH

We presume that our company network has a DMZ with two firewalls – a perimeter firewall “PFW” protecting the DMZ and its hosts against the Internet and one or several inner firewalls “IFW(s)” separating the DMZ from inner network segments. Creating firewall tunnels on the outer and/or inner side of the DMZ is always a critical issue.

Our own backup server system for customer data – let us call it “bup” – is e.g. located in an inner segment of our network. The perimeter firewall “PFW” to the Internet resides on a router and can be configured to forward external WAN ports to ports of special hosts inside the DMZ. Servers inside an inner network segment are allowed to build up SSH connections to hosts in the DMZ across their “IFW(s)”; however the servers in the DMZ are not allowed to initiate SSH connections to the servers in the inner segments. For security reasons we poll data from the servers in the DMZ; we do not forward or push from the DMZ to inner segments. For our scenaio we assume that there is only one inner segment and that we only have to deal with a single “IFW”.

I personally prefer a 2-step process to get the data from the hosted server to the backup destination server “bup”:

  • Step 1: I open a reverse SSH tunnel from a special host “intmid” in our DMZ to the Strato server on the Internet and transfer the data from the Strato server to “intmid”. There, I first of all scan the backup files for viruses and malware.
  • Step 2: Afterwards I stop the reverse tunnel and disallow any incoming connections to “intmid”. I then open a ssh reverse tunnel from my destination server “bup” to “intmid” across the IFW, transfer the files to “bup” and encapsulate them there in an encrypted container.

Some readers may think a 2 step procedure may be exaggerated. If you prefer direct connections from the hosted Strato server to “bup” inside your network you may

  • either use a chain of two reverse SSH tunnels (one tunnel per firewall)
  • or a combination of port forwarding on the “PFW” with
    a reverse tunnel across the “IFW”

Both solutions provide a complete connection from the Strato server to “bup” passing both firewalls of your DMZ. Below I describe both methods briefly.

In both of the following methods we need a reverse SSH tunnel from the backup destination server “bup” (inner segment) to the DMZ host “intmid”. I.e., we export a SSH port of server “bup” temporarily to the host “intmid” in the DMZ. This you may e.g. achieve by entering the command

ssh -R 19999:localhost:22 ssh_user_on_intmid@intmid

at your prompt on host “bup”. The “-R” option triggers the creation of a “reverse” tunnel. Port 19999 is assumed to be unused on “intmid”; port 22 is your local port on “bup” for the SSH daemon sshd. “ssh_user_on_intmid” represents a user for SSH connections on “intmid” with sufficient privileges.

Note: If you just want to build the tunnel in the background without using a terminal or executing remote commands you may use the following combination of options for SSH (instead of a plain “-R”) : ssh -nNTfR …...
Please see the man pages of ssh. These options may be especially helpful when using scripts. So, a more general form of the command could be

ssh -p<sshd port server> -nNTfR <forwarded port server>:localhost:<sshd port local> <username>@<ip server>

Note 1: If and when you really work with background SSH tunnels: Take care not to forget to stop the processes for the background tunnels at the end of your backup operation! In scripts this requires some “pid” determination operations for the running ssh processes.

Note 2: Depending on your sshd configuration you may need to have root rights to build the tunnel.

You can test your reverse tunnel through the inner firewall “IFW” by entering

ssh -p 19999 userid_on_bup@localhost

at the prompt of the “intmid” host. You will of course be asked for the password of a valid user “userid_on_bup” on server “bup” and should then get a prompt on “bup”.

Method 1: Chaining two reverse tunnels

The hosted Strato server allows for a ssh connection on a specific port as this is your only way to work on it. You can use this SSH-connection to build up a reverse tunnel from your DMZ host “intmid” to the Strato server. I.e., the SSH port of your DMZ host shall be exported e.g. as port 19999 on the hosted Strato server. By issuing the following command on “intmid” we tunnel the perimeter firewall “PFW”:

ssh -R 19999:localhost:19999 strato_user@ip_of_your_Strato-server -p 61111

Note: Here we have assumed that the Strato-Server listens on port 61111 as its configured SSH port. (Normally, I change the standard port 22 to a high port number. See:
Strato-V-Server mit Opensuse 12.3 – II – Installation / SSH-Port ändern)

You will of course have to give the password for your “strato_user” on the hosted Strato server before entering the shell there.

Why did we use port 19999 on the DMZ host “intmid”? Because we want to chain the two tunnels – the one from “bup” to “intmid” in the DMZ across firewall “IFW” and the one from “intmid” to our Strato server on the Internet across firewall “PFW” to get a complete encrypted connection from the Strato server to “bup”. Both tunnels “meet” at port 19999 on host “intmid”.

Now, at the prompt of the Strato server you can test the combined (reverse) SSH tunnels by entering

ssh -p 19999 userid_on_bup@localhost

Note: Do not get confused by “localhost” in the command above. You must provide the password for user “userid_on_bup” on the server “bup” (!) inside our inner network segment. Our connected tunnels end up there!

You then should get a prompt for user “user_id_on_bup” on our server “bup”. Ok, we now can successfully tunnel two firewalls at the same time by making use of 2 chained reverse SSH tunnels!

Method 2: Port forwarding on the perimeter firewall

Port forwarding on the perimeter firewall could be an alternative to chaining two tunnels. Having build our obligatory reverse tunnel from “bup” to “intmid” we could in addition change the behavior of our PFW – provided of course that it is configurable. In my case I can establish port forwarding for incoming connections on an some external (WAN) port from a defined IP address on the Internet to a definable port of any host at the other (DMZ) side of the perimeter firewall. E.g I could forward a connection from the Strato server trying to reach the WAN side of the perimeter firewall on port 19999 to port 19999 on “intmid” in the DMZ.

Note: If you really do something like this on your firewall you have to be careful to restrict the forwarding to a defined server on the Internet and to close this firewall hole again after the backup procedure. [It makes no sense here to give commands or details as the configuration procedure varies with the type of firewall used.]

However, for this approach to work with openSSH you in addition have to set a parameter for your sshd configuration in the file “/etc/ssh/sshd_config” on “intmid”:

GatewayPorts yes

Otherwise external hosts will not be allowed to use the reverse tunnel. This setting is somewhat dangerous and you should not forget to change it back.

In this approach we couple (local) port forwarding on the PFW with a reverse tunnel from “bup” to “intmid”. The result is basically the same as with two chained reverse tunnels. However, you have to modify your commands on the Strato server somewhat to use port 19999 at the WAN side of your “PFW”. To test the forwarding plus reverse tunnel on the Strato server you would enter:

ssh userid_on_bup@ip_address_of_your_routers_WAN_interface -p 19999

You have to provide the IP address of the WAN interface of your router. Depending on your DSL provider this may change on a daily basis. So you have to find out what it presently is – e.g by the admin interface of your PFW (router) or by asking an external service (e.g. curl icanhazip.com).

All in all I regard the second method as more complicated and potentially also more dangerous than method 1 – because you must not forget to change and change back your sshd configuration on the DMZ host “intmid” and also to reset your PFW configuration. Therefore, I shall only follow the first approach below.

Use rsync to perform the backup

As we have successfully build up tunnels across our firewalls one may think of using “scp” to perform a backup of the server directories. Actually this is not a good idea if you want to make backups of all server directories including system directories. The reason is:

“scp” follows all symbolic links! At least on Opensuse systems this may lead to infinite loops for certain directories whilst copying!

Therefore it is better to use “tar” or “rsync”. See: http://stackoverflow.com/questions/11030394/is-it-possible-to-make-scp-ignore-symbolic-links-during-copy

Personally, I find “rsync” most convenient. And you can combine it
quite easily with SSH! To use the ports defined for our method 1 (with the two chained reverse SSH tunnels) the right syntax for making a backup of e.g. the directory “/var” would be

rsync -avz -e ‘ssh -p 19999’ /var userid_on_bup@localhost:/backups/bup_strato/server_name/

This command has to be performed on the Strato server. Again: Do not get confused by “localhost” – that’s how SSH tunneling works! The “-a” option of the rsync command means “archive mode”. This preserves the mode and user settings of all files and directories throughout the data copying and performs recursions. “z” leads to a transfer of compressed data which will save you time. “/backups/bup_strato/server_name/” represents an example directory on server “bup” where you want to place the backup files of your special Strato server. Of course you should replace it by your own path on your backup server.

Note 1: You have to issue this commands as root on your Strato server to get access to all of the directories there.
Note 2: Please, note that slash “/” at the end of the directory path on the destination system! And note the missing “/” after the local directory “/var”.
Note 3: Please, note the masking of ssh -p 19999 by single quotation marks! [To specify the port by “-p 19999” at the end of the ssh command would not work as you would mix rsync and ssh parameters].
Note 4: The big advantage of rsync naturally is that you only would exchange changed files at the next backup.

Have much fun now with your Strato server and making backups through SSH tunnels!