You may have important files in your computer such as a password manager database, gpg key, bitcoin private keys. There is a risk of loss or modification of these files for multiple possible reason such as hardware failure. If that happen it may have a big impact for you. In this article some simple backup method for Linux will be presented as well as how to automatize them.
Importance of backup
Losing data can happen for multiple reasons such as:
- accidental delete
- mechanical or electronic defect on device
- random errors on file
For this reason, important data and especially private key, password manager, etc., should be backed up:
Some methods of backup
In this part different method for backing up files with Linux will be presented. First a custom bash script is created. In a second part, an attempt of a one-liner will be presented.
Let's make a custom bash script to backing up files into another folder
Attention: This script creates and delete files permanently. You can use it or modify it to your needs,
but I cannot be held responsible for any bad manipulation you could do.
I also recommend using
rm -i when testing script using
rm to have interactive deletion and avoid accidental deletion.
# Change following strings to adapt to your case file='/path/to/my/important/data/' number_backup=6 # the maximum number of copies we want # end now="$(date +%Y_%m_%d)" # retrieve the current day in the format YYYY_MM_DD new_name="$file""_$now"".bak" cp "$file" "$new_name" # To delete the old # Manage the number of copies and delete when there are too many counter=$(find "$(dirname $file)" -name "$(basename $file)_*.bak" | wc -l) if [ "$counter" -le "$number_backup" ] then echo "only $counter (<=$number_backup) copy of the file" echo "Nothing to do. Exiting" exit 0 else diff=$(( "$counter" - "$number_backup")) to_delete=$(find "$(dirname $file)" -name "$(basename $file)_*" | sort | head -n "$diff") rm -i "$to_delete" # When you finish testing it # you can replace it by rm "$to_delete" fi
Of course this script can be adapted for example to add logs or any other actions
This script has some problems which are:
- the script is long
- it cannot be used remotely easily
- it would be even better if it can be done in one line (linux powa (^^)/)
One liner with rsync
In this part I will present the rsync command and how to make the backup command in one line.
Quick presentation of rsync
From the man page of rsync:
rsync - a fast, versatile, remote (and local) file-copying tool
Indeed, rsync is a command that can be easily used to make copies and synchronize folders locally or remotely. It is easy to use:
rsync [OPTION] SRC DST
Let's see some options that are useful (by reading
#[useful command to debug] --verbose, -v increase verbosity --dry-run, -n perform a trial run with no changes made # For Backups --backup, -b make backups (see --suffix & --backup-dir) --rsh=COMMAND, -e specify the remote shell to use # Commands that are useful when doing system backup and being able to restore the permission --perms, -p preserve permission --acls, -A preserve ACLs (implies --perms) --xattrs, -X preserve extended attributes --owner, -o preserve owner (super-user only) --group, -g preserve group
So for example if I want to back up a file in a remote server, I can use the following command:
rsync --verbose --backup -e "ssh -i <private_key>" <file> <username>@<server address>:/path/to/bakup/file>
You should, of course, give correct access permission to the private key file, especially if you do not password protect it for automating the backup. In this case, I advise using a key that will be only be used to for the backing up data.
Let's back it up
From the previous part, it is easy to adapt the first script by using rsync instead of cp:
rsync --backup <file> <user>@<address>:</path/to/backup/file>`date +'.%Y_%m_%d.bak'`
or if we want to back it up remotely
rsync --backup -e "ssh -i </path/to/private/key>" <file> <user>@<address>:</path/to/backup/file>`date +'.%Y_%m_%d.bak'`
Now let’s try to improve the deletions of the old files.
# The files older than n days # /!\ deleted files are deleted forever. You should test it carefully find . -name '*.bak' -mtime +<n> -delete # Or only keep the <n> more recent files ls -1t *.bak | tail -n +$((<n>+1)) | xargs -d '\n' rm -i # replace by rm when needed
The first command is not to difficult to understand. Let's explain the second command:
ls -1t *.bak: Find all files ending by .bak and sort them in one column
tail -n +$((<n>+1)): keep only the lines after line n
xargs -d '\n' rm: consider each entry separated by a newline '\n' as a file name and delete it
From it, the backup can become a one-liner:
rsync --backup <file> </path/to/backup/file>`date +'.%Y_%m_%d.bak'` && find . -name '*.bak' -mtime +<n> -delete # or rsync --backup <file> </path/to/backup/file>`date +'.%Y_%m_%d.bak'` && ls -1t *.bak | tail -n +$((<n>+1)) | xargs -d '\n' rm -i
And in case it is done remotely, we should use ssh command to execute it remotely or launch the deletion script directly on the machine:
key=</path/to/private/key> && rsync --backup -e "ssh -i $key" <file> <user>@<address>:</path/to/backup/file>`date +'.%Y_%m_%d.bak'` && ssh -i $key <user>@<address> "find . -name '*.bak' -mtime +<n> -delete" # or key=</path/to/private/key> && rsync --backup -e "ssh -i $key" <file> <user>@<address>:</path/to/backup/file>`date +'.%Y_%m_%d.bak'` && ssh -i $key <user>@<address> "ls -1t *.bak | tail -n +$((<n>+1)) | xargs -d '\n' rm -i"
Of course it is better to put all this one-liner in a file to execute it.
Let's automatize it
With the previous scripts, backups become easy to make. However, it would be better if there is no need to remember to do backup and that they are done automatically. Cronjob is a very good option to automatize script execution
Once you have installed cron in your machine, let’s see how to use it.
To list all the cronjob (all automatic task) simply use:
New automatic tasks can be added by:
Each task as the following format
# Minute(0-59) Hour(0-24) monthday(1-31) Month(1-12) weekday(0-6) Command
Which corresponds to the following possibilities
# Execute every minute * * * * * /path/to/script/to/execute # Execute two hours * */2 * * * /path/to/script/to/execute # Execute every 3rd day of the month * * 3 * * /path/to/script/to/execute # Every week on Monday at 1a.m. * 1 * * 1 /path/to/script/to/execute # or @weekly /path/to/script/to/execute
You can execute your script at the time you want. To verify that the execution is correctly done, it is advised to test it by executing the script every minute before adjusting to your needs.
Backing up important data is important to avoid any future problem. By making your scripts or using command such as rsync, you can back up easily your data. By using crontab it can be automated easily. Therefore you should put a backup policy for your personal data.
Thank you for reading this article. Hope to see you soon.