Simple automated backup solution

There are many tools today that can be used to backup your data. Most of them come with shiny eye-candy GUIs and with a few clicks you can synchronize your data to Dropbox, Google Drive or wherever you want. So, why not use some of them and end this blog post right here? First of all, these solutions are boring, then there is the problem of giving your data to third parties (call me crazy, but I’m never going to upload private SSH keys to Google) and finally I wanted to have daily snapshots. So, I wrote a small shell script that does the job.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/bin/bash

## Automated backup script
##
## Uploads backed up archived to the server, runs daily
##
## Author Milos Milutinovic
## 2017-02-07

#put all backup dirs here
bkp_start_dir='/home/milos/tmp/bkp'
bkp_sec_dir='/home/milos/secure/tmp/bkp'

#daily dir
today=`date +"%Y-%m-%d"`

cd $bkp_start_dir

#create archives into this dir
tar -cjf ssh.tar.bz2 /home/milos/.ssh

#encrypt SSH archive
rm ssh.tar.bz2.gpg #remove old
gpg --passphrase-file /home/milos/secure/keys/gpg.key --simple-sk-checksum --batch -c ssh.tar.bz2
rm ssh.tar.bz2

# secure
cd $bkp_sec_dir
rm sec.tar.bz2.gpg #remove old
tar -cjf sec.tar.bz2 /home/milos/secure
gpg --passphrase-file /home/milos/secure/keys/gpg2.key --simple-sk-checksum --batch -c sec.tar.bz2
rm sec.tar.bz2

cd $bkp_start_dir

#create one daily archive
tar -cjf $today.tar.bz2 ssh.tar.bz2.gpg /home/milos/scripts /home/milos/Documents/AC /home/milos/secure/tmp/bkp/sec.tar.bz2.gpg /home/milos/Documents/db1/code

#scp to the server
scp -p -i /home/milos/secure/keys/bkpuser $today.tar.bz2 bkpuser@miloske.tk:/path/to/folder/

Let me explain it. First interesting bit is line 15. This is how archive name is generated, it will be in format YYYY-MM-DD. Then I archive my ~/.ssh folder and encrypt it with gpg, using symmetric encryption with a passphrase file stored in an secure location. I have to remove the encrypted archive from the previous day and after encrypting it, I remove the plaintext one.

I then do similar thing with another location I want to backup securely and finally, on line 37, I create an archive that contains all of the data. You might say that for creating those encrypted archives, I didn’t have to use bzip2 option (create .tar archive instead), as they would be packed into the final archive, but think again. Those archive are encrypted, if I was creating tar archive (which are compressible) and then encrypting them, I wouldn’t be able to compress them. Random data is not compressible.

Another approach would be to create a folder each day and put several archives in it, then upload the folder to the server. This would be a bit more efficient, as it would avoid running bzip2 compression on archives that are already compressed (and encrypted), but the difference in negligible and having all files instead of folders means that it’s a lot easier to get rid of the old files on my server. On the server, I just have this kind of thing in a file in /etc/cron.daily:

1
find /var/www/miloske.tk/bkp/ -mtime +15 | xargs rm

This deletes any files older than 15 days in this location.

In the end, I scp data to my server. I’m uploading only one file, so rsync is not necessary. I do use rsync on my home backup server to pull the data from the online server, but here I’m synchronizing several folders, so I need rsync. This script is set to run as cron job on my work machine, so I always have backups of important files.

Read More

When mv command fails

I’m a lazy blogger and it’s been a long pause, but here I go again…

I’ve been with Linux for quite some time. In fact, I’ve banished Windows from all my machines and I’m running only Linux now. So, I considered myself to be fairly familiar with the command line., however, surprises are still possible.

Today I was uploading large amount of files to a web server. Because of the way server is set, the application directories are writable only by sudo users and I didn’t find a way of making Filezilla use the sudo command. SCP wasn’t an option either. So I had to upload everything to my home directory on that machine and then login via SSH and move the files. Easy, right, it’s just:

1
sudo mv ~/files/* /var/www/app/files/

Well, not quite. This command returned an error:

1
bash: /bin/mv: Argument list too long

I checked what I’ve typed, mv expects two arguments, input file/folder and output file/folder. How could this error be reported then, I haven’t made an error.

Short  online search gave me a hint, mv command can fail when it has to move too many files. I checked the folder where I’ve uploaded them, and there were several thousands. By the way, you can count the files in a directory with this command: “ls -l | wc -l”, subtract two from the number you get and that’s how many files you have in the current directory (wc -l counts lines, it will also count “.” and “..”).

So, what to do now, how to move files when mv command has failed and there is nothing else available. The dump way of solving this would be to feed mv one small subset of files, like this:

1
2
3
4
sudo mv ~/files/0* /var/www/app/files/
sudo mv ~/files/1* /var/www/app/files/
sudo mv ~/files/2* /var/www/app/files/
...

In my case, I had files with hex character set (0-9, a-f), so there were 16 possible combinations of the first character of the file name. There weren’t that many files in my case and this approach would have been possible, but it’s still tedious. In my opinion, the best way is to use rsync. Another good way is to use oneliner shell script, this approach is also useful for other actions, such as changing permissions.

Here’s an example of oneliner shell script:

 

1
for file in source/*; do cp $file dest/ ; done

Read More