I administer several Linux web servers for partner and customer companies. Most of these servers are V-servers hosted at the German Telekom daughter Strato.
[I have described some reasonable security measures for a Strato server with Opensuse in some articles in this blog. See e.g.:
Strato-V-Server mit Opensuse 12.3 – IV – SSH Key Authentication
Strato-V-Server mit Opensuse 12.3 – III – Stop SSH Root Login
and previous articles referenced there.]
When you work with a hosted server, of course, you want to perform periodic backups. If you – for whatever reasons – do not want to pay for the backup facilities Strato offers, a backup requires that you transfer data from the remote server into your own company network through (encrypted) SSH tunnels. Depending on your specific situation you may wish to create a complete backup or a backup of selected directories on the server. During the backup you want to preserve user-rights and mode settings.
In addition to the topic of creating tunnels across firewalls, the question arises what Linux tool might be convenient to perform the backup task if you work with a shell based connection only. Therefore, I shall briefly discuss what I normally do to perform a manually controlled backup from a hosted Strato server. You can combine the methods described of course with some scripting. The described firewall tunneling by SSH is independent of a specific provider.
Tunneling firewalls with SSH
We presume that our company network has a DMZ with two firewalls – a perimeter firewall “PFW” protecting the DMZ and its hosts against the Internet and one or several inner firewalls “IFW(s)” separating the DMZ from inner network segments. Creating firewall tunnels on the outer and/or inner side of the DMZ is always a critical issue.
Our own backup server system for customer data – let us call it “bup” – is e.g. located in an inner segment of our network. The perimeter firewall “PFW” to the Internet resides on a router and can be configured to forward external WAN ports to ports of special hosts inside the DMZ. Servers inside an inner network segment are allowed to build up SSH connections to hosts in the DMZ across their “IFW(s)”; however the servers in the DMZ are not allowed to initiate SSH connections to the servers in the inner segments. For security reasons we poll data from the servers in the DMZ; we do not forward or push from the DMZ to inner segments. For our scenaio we assume that there is only one inner segment and that we only have to deal with a single “IFW”.
I personally prefer a 2-step process to get the data from the hosted server to the backup destination server “bup”:
- Step 1: I open a reverse SSH tunnel from a special host “intmid” in our DMZ to the Strato server on the Internet and transfer the data from the Strato server to “intmid”. There, I first of all scan the backup files for viruses and malware.
- Step 2: Afterwards I stop the reverse tunnel and disallow any incoming connections to “intmid”. I then open a ssh reverse tunnel from my destination server “bup” to “intmid” across the IFW, transfer the files to “bup” and encapsulate them there in an encrypted container.
Some readers may think a 2 step procedure may be exaggerated. If you prefer direct connections from the hosted Strato server to “bup” inside your network you may
- either use a chain of two reverse SSH tunnels (one tunnel per firewall)
- or a combination of port forwarding on the “PFW” with
a reverse tunnel across the “IFW”
Both solutions provide a complete connection from the Strato server to “bup” passing both firewalls of your DMZ. Below I describe both methods briefly.
In both of the following methods we need a reverse SSH tunnel from the backup destination server “bup” (inner segment) to the DMZ host “intmid”. I.e., we export a SSH port of server “bup” temporarily to the host “intmid” in the DMZ. This you may e.g. achieve by entering the command
ssh -R 19999:localhost:22 ssh_user_on_intmid@intmid
at your prompt on host “bup”. The “-R” option triggers the creation of a “reverse” tunnel. Port 19999 is assumed to be unused on “intmid”; port 22 is your local port on “bup” for the SSH daemon sshd. “ssh_user_on_intmid” represents a user for SSH connections on “intmid” with sufficient privileges.
Note: If you just want to build the tunnel in the background without using a terminal or executing remote commands you may use the following combination of options for SSH (instead of a plain “-R”) : ssh -nNTfR …...
Please see the man pages of ssh. These options may be especially helpful when using scripts. So, a more general form of the command could be
ssh -p<sshd port server> -nNTfR <forwarded port server>:localhost:<sshd port local> <username>@<ip server>
Note 1: If and when you really work with background SSH tunnels: Take care not to forget to stop the processes for the background tunnels at the end of your backup operation! In scripts this requires some “pid” determination operations for the running ssh processes.
Note 2: Depending on your sshd configuration you may need to have root rights to build the tunnel.
You can test your reverse tunnel through the inner firewall “IFW” by entering
ssh -p 19999 userid_on_bup@localhost
at the prompt of the “intmid” host. You will of course be asked for the password of a valid user “userid_on_bup” on server “bup” and should then get a prompt on “bup”.
Method 1: Chaining two reverse tunnels
The hosted Strato server allows for a ssh connection on a specific port as this is your only way to work on it. You can use this SSH-connection to build up a reverse tunnel from your DMZ host “intmid” to the Strato server. I.e., the SSH port of your DMZ host shall be exported e.g. as port 19999 on the hosted Strato server. By issuing the following command on “intmid” we tunnel the perimeter firewall “PFW”:
ssh -R 19999:localhost:19999 strato_user@ip_of_your_Strato-server -p 61111
Note: Here we have assumed that the Strato-Server listens on port 61111 as its configured SSH port. (Normally, I change the standard port 22 to a high port number. See:
Strato-V-Server mit Opensuse 12.3 – II – Installation / SSH-Port ändern)
You will of course have to give the password for your “strato_user” on the hosted Strato server before entering the shell there.
Why did we use port 19999 on the DMZ host “intmid”? Because we want to chain the two tunnels – the one from “bup” to “intmid” in the DMZ across firewall “IFW” and the one from “intmid” to our Strato server on the Internet across firewall “PFW” to get a complete encrypted connection from the Strato server to “bup”. Both tunnels “meet” at port 19999 on host “intmid”.
Now, at the prompt of the Strato server you can test the combined (reverse) SSH tunnels by entering
ssh -p 19999 userid_on_bup@localhost
Note: Do not get confused by “localhost” in the command above. You must provide the password for user “userid_on_bup” on the server “bup” (!) inside our inner network segment. Our connected tunnels end up there!
You then should get a prompt for user “user_id_on_bup” on our server “bup”. Ok, we now can successfully tunnel two firewalls at the same time by making use of 2 chained reverse SSH tunnels!
Method 2: Port forwarding on the perimeter firewall
Port forwarding on the perimeter firewall could be an alternative to chaining two tunnels. Having build our obligatory reverse tunnel from “bup” to “intmid” we could in addition change the behavior of our PFW – provided of course that it is configurable. In my case I can establish port forwarding for incoming connections on an some external (WAN) port from a defined IP address on the Internet to a definable port of any host at the other (DMZ) side of the perimeter firewall. E.g I could forward a connection from the Strato server trying to reach the WAN side of the perimeter firewall on port 19999 to port 19999 on “intmid” in the DMZ.
Note: If you really do something like this on your firewall you have to be careful to restrict the forwarding to a defined server on the Internet and to close this firewall hole again after the backup procedure. [It makes no sense here to give commands or details as the configuration procedure varies with the type of firewall used.]
However, for this approach to work with openSSH you in addition have to set a parameter for your sshd configuration in the file “/etc/ssh/sshd_config” on “intmid”:
GatewayPorts yes
Otherwise external hosts will not be allowed to use the reverse tunnel. This setting is somewhat dangerous and you should not forget to change it back.
In this approach we couple (local) port forwarding on the PFW with a reverse tunnel from “bup” to “intmid”. The result is basically the same as with two chained reverse tunnels. However, you have to modify your commands on the Strato server somewhat to use port 19999 at the WAN side of your “PFW”. To test the forwarding plus reverse tunnel on the Strato server you would enter:
ssh userid_on_bup@ip_address_of_your_routers_WAN_interface -p 19999
You have to provide the IP address of the WAN interface of your router. Depending on your DSL provider this may change on a daily basis. So you have to find out what it presently is – e.g by the admin interface of your PFW (router) or by asking an external service (e.g. curl icanhazip.com).
All in all I regard the second method as more complicated and potentially also more dangerous than method 1 – because you must not forget to change and change back your sshd configuration on the DMZ host “intmid” and also to reset your PFW configuration. Therefore, I shall only follow the first approach below.
Use rsync to perform the backup
As we have successfully build up tunnels across our firewalls one may think of using “scp” to perform a backup of the server directories. Actually this is not a good idea if you want to make backups of all server directories including system directories. The reason is:
“scp” follows all symbolic links! At least on Opensuse systems this may lead to infinite loops for certain directories whilst copying!
Therefore it is better to use “tar” or “rsync”. See: http://stackoverflow.com/questions/11030394/is-it-possible-to-make-scp-ignore-symbolic-links-during-copy
Personally, I find “rsync” most convenient. And you can combine it
quite easily with SSH! To use the ports defined for our method 1 (with the two chained reverse SSH tunnels) the right syntax for making a backup of e.g. the directory “/var” would be
rsync -avz -e ‘ssh -p 19999’ /var userid_on_bup@localhost:/backups/bup_strato/server_name/
This command has to be performed on the Strato server. Again: Do not get confused by “localhost” – that’s how SSH tunneling works! The “-a” option of the rsync command means “archive mode”. This preserves the mode and user settings of all files and directories throughout the data copying and performs recursions. “z” leads to a transfer of compressed data which will save you time. “/backups/bup_strato/server_name/” represents an example directory on server “bup” where you want to place the backup files of your special Strato server. Of course you should replace it by your own path on your backup server.
Note 1: You have to issue this commands as root on your Strato server to get access to all of the directories there.
Note 2: Please, note that slash “/” at the end of the directory path on the destination system! And note the missing “/” after the local directory “/var”.
Note 3: Please, note the masking of ‘ssh -p 19999‘ by single quotation marks! [To specify the port by “-p 19999” at the end of the ssh command would not work as you would mix rsync and ssh parameters].
Note 4: The big advantage of rsync naturally is that you only would exchange changed files at the next backup.
Have much fun now with your Strato server and making backups through SSH tunnels!