CRONBACKUP README (2024-02-06)

cronBackUp takes on the necessary task of backing up files and directories that you can't afford to loose. It uses daily, weekly and monthly backup scheduling to cover what you did today, a couple of weeks ago or four months ago. You have full control over incrementing, compressing, storing, encrypting and transferring to remote locations.

All from one small configuration file. A maximum of eleven settings. Really, there's only one mandatory setting, just list some path(s) for the script to backup. The other settings all have workable defaults or provide supplemental features.

Use cron to schedule your backup. cronBackUp will write a log showing what was done Cron can email you the log or you can read the log at it's configured location. cronBackUp can encrypt your backups and send them to alternate locations using transferCommands.

REQUIRES THE BASH SHELL
Unpack the archive to a directory of your choice. The directory must allow the execution of programs.

Unpacking this archive will create a directory called "backup". The backup directory will contain all the files and dirs needed to use cronBackUp.

The directory files.

daily-bkup -------- ## executable

daily-bkup.conf --- ## config file

weekly-bkup ------- ## executable

weekly-bkup.conf -- ## config file

monthly-bkup------- ## executable

monthly-bkup.conf - ## config file

daily-scripts ----- ## user scripts

monthly-scripts --- ## user scripts

weekly-scripts ---- ## user scripts

There are three cronBackUp scripts: daily-bkup, monthly-bkup and weekly-bkup. Each script has a corresponding configuration file (ending in .conf). There are three directories for storing personal scripts. Use of the sudirectories is not required, they are for convenience only.

The bkup scripts will look for a matching .conf file in a few places, in this order.

The current working directory.
The users $HOME dir.
The users $HOME/.config dir
The system /etc dir

The .conf files contain eleven sections.

Only section 1 requires settings, all other sections have fallback defaults. Enable/disable sections are optional, disabled by default. The .conf files are commented throughout, read the comments for more details. The .conf files use the well tested format of name=value. Settings are not required to have a value set, they can be empty. Example: 'bkupPath=' is valid, but is required even though the value is empty. Do not change the name. Do not quote any of the values.

1) REQUIRED path=
List of paths to backup, at least one path. List as many 'path=xxx' as you like. Each one must be a full path to an existing file/directory. Paths that are not valid will be skipped and noted in the log. The file/directory must be readable by the user running the script. Paths can have spaces and apostrophes in them.

path=/home/username/bin
path=/var/www
path=/etc/apache2
path=/etc/fstab
path=/etc/hosts

2) bkupPath=
The full path to the directory that will store the backup files (bkupPath). This directory can not be the root directory (/). This directory must be writable by the user running the cronBackUp script. Files in the bkupPath are susceptible to removal by the cleanCommand configuration (see cleanCommand below). Files in the bkupPath are susceptible to being overwritten (see fullBackUp below). If no bkupPath is given, cronBackUp will default to PWD.

3) logFile=
Write a log file (logFile). Enter a file name for a log file, if you want one. It will be written to the the configured bkupPath if it's just a file name. If it's a file path (starts with a /) then it will be written to that path.

4) preTarCommand=
Add a command in front of the tar command. The tar command can generate a large amount of cache. A command can be used to reduce that if placed in front of the tar command. The default utility to accomplish this is a program called 'nocache'. Other programs could be used to do this or any other appropriate task.

5) fullBackUp=
The day/month/week to force a full backup (defaults to a full backup). The backups are incremented daily, monthly or weekly depending on the script executed. Full backups are done periodically in accordance with the fullBackup value. When fullBackUp is not set, cronBackUp will do a Full Backup each time it is executed. When fullBackUp is set to 1, cronBackUp will do a Full Backup each time it is executed. This behavior can be changed so that additional full backups are done at specific intervals.

If running daily-bkup, fullBackup will be a day number (1-7 Sun=1). Daily backup files are prefaced with the day name for that day. When that day number is evenly divisible by fullBackup the increment will be reset. On that day the script will reset the increment and run a full backup. Previous backups for that day will be overwritten when the day name repeats. Read the daily-bkup.conf file for more instructions on how to use backup intervals.

If running weekly-bkup, the fullBackUp will be a week number (1-52). By default weekly backup files are prefaced with the week number for that year. When that week number is evenly divisible by fullBackup the increment will be reset. Each time the increment is reset a new full backup will be written. Previous backups for a week number will be overwritten when the week number repeats. Read the weekly-bkup.conf file for more instructions on how to use backup intervals.

If running monthly-bkup, fullBackup will be a month number (1-12). By default monthly backup files are prefaced with the month name for that month. When that month number is evenly divisible by fullBackup the increment will be reset. On the scheduled month the script will reset the increment and run a full backup. Previous backups for each month will be overwritten when that month repeats. Read the monthly-bkup.conf file for more instructions on how to use backup intervals.

6) dumpCommand=
Make dump files from databases and repositories. Create a 'dumpCommand=' line to enter the dump command for each source. The full command must be provided as well as a desired fileName to dump the data to. Credentials must be provided such that execution will not force the shell to prompt for them. To avoid giving credentials on the command line, use files like '.my.cf', '.pgpass', '.subversion'. Dump files are not incremental, they are full backups and will overwrite the previous backup.

7) fileExt=
Compress the tar archive (fileExt). Tar can automatically compress the archive given a file extension that represents a type of compression. Supported extensions (.gz, .tgz, .taz, .Z, .taZ, .bz2, .tz2, .tbz2, .tbz). Read the tar docs for others. The program that can create the selected compression needs to be installed into the system path.

8) encryptCommand=
The archive files can be encrypted for remote or local storage. Provide the full command needed to encrypt the files for a given user. Credentials must be provided such that execution will not prompt the user for them.

9) cleanCommand=
If files are stored in secondary location(s) and local copies are not wanted. The cleanCommand text is appended to the command 'rm'. Bash filename globbing is supported. The cleanCommand(s) will remove files from bkupPath after the transferCommand(s) have ended. Multiple cleanCommand values can be used, one per line.

10) symlinks=
Allow tar to follow symlinks (symlinks). Tar can be configured to follow all symlinks that it finds in the path it is given. Caution is recommended here, as this can cause unforeseen increases in the size of the archives.

11) transferCommands=
Commands used to transfer files to remote sites (transferCommand). An unlimited number of transfer commands are allowed. Each command given will be executed, in the order that they are presented. Credentials must be provided such that execution will not force the shell to prompt for them. Some examples are provided in the conf file, but any valid shell command will work. regardless of where personal scripts are located, the full path must be provided. Variable values assigned in cronBackUp files can be used in these transferCommands ($dayName, $weekName, $monthName, $dayNum, $weekNum, $monthNum, $bkupPath).

You will need a cron job configured if you intend to have the backups automated.

Running a cron job as a regular user usually makes the setup easier. Any authentication or credential files used must be in the user's home directory. Execute "crontab -e" to edit your personal crontab.

Refer to the example crontab file in the Doc directory. All directories are example directories, the directories you use can be different. Note that the cd command changes the PWD to the backup script dir. This insures that the script can find the files it needs easily. In this example the daily-bkup script is run every day at 8pm (hour 20 minutes 0). The weekly-bkup script is run at 6pm on the scheduled 'day of week' (dow 0, Sun). The monthly-bkup script is run at 7pm on the scheduled 'day of month' (dom 1).

Running a cron job as root can be useful for backing up files not readable by the regular user. There may be additional security concerns when using root.

You can use the root crontab for this or the builtin /etc/cron* directories. When a cron job is run as root the commands will look in /root for external sources. In this case authentication and credential files used must be in the /root directory.

Example ftp and sftp shell scripts are included in each of the xxx-scripts directories. Refer to backup/daily-scripts to follow this example.

Let's propose you would like to use sftp to transfer a copy of your daily backup files to a remote host. There maybe some problems to overcome.

You need a batch file that will have time specific values in it. You will need to provide a source directory for the files you want to transfer. You want to limit the files transferred by filename, perhaps by file extension and/or day name. The sftp.sh shell script can do this for you, after editing the four required values.

Executing the script will write a batch file for sftp, specifically for the current day. It will provide the target directory and the source directory. The files transferred must start with $dayName and end with .gpg. It will be named sftp.daily and be written to daily-scripts.

After executing sftp.sh the daily-scripts directory will contain the sftp.daily batch file. The sftp.daily file will be updated every time the sftp.sh script is run. sftp.sh will be run from the daily-bkup.conf file as a transferCommand. Followed by a transferCommand to run the sftp program with -b (batch) like this.

transferCommand=/path/to/sftp.sh
transferCommand=sftp -b /path/to/sftp.daily user@host

Any of these backup scripts can be executed from the command line. You will need to have a matching conf file in one of the known paths. When the fullBackup value is 1, cronBackUp will do a full backup every time it is executed. Open a terminal program and, using the example path, cd to /home/username/backup.

Execute: dave@debian:~/backup$ ./daily-bkup

Common programs used when backing up local sources to remote targets often require credentials. Examples would be rsync, ftp, sftp, ssh, lftp and perhaps many others. In addition sources such as databases and version control systems require credentials for access.

The problems:
Credentials will have to be provided to cronBackUp so that cronBackUp is not prompted for passwords. In some cases this can be done on the command line or using an external file. However, be aware, it is not considered best practice to provide credentials on the command line. Command line credentials can end up in system log files or command history files. These files could be viewed by interested parties, without your approval or knowledge.

Some solutions:
There are mechanisms that can provide the needed separation between credentials and execution. Files should be protected with strict permissions, allowing only the owner to view the contents. For any ssh based transfer command, sharing public ssh keys is a way to accomplish this. For ftp the .netrc file can be used to provide credentials. For mariadb (mysql type) databases the .my.cnf file can be used to provide credentials. For postresql databases the .pgass file can be used to provide credentials. For subversion repositories the .subversion/auth/svn.simple file can be used to provide credentials.

All of these solutions should be researched before use. File permissions should be kept minimal. In addition these may not be the only solutions or the best solutions, see what else you can find.

These scripts are very specifically written to function under a system that is configured with a single user in mind, the author. The dependencies required are all available from Debian repositories. Although an honest effort was made to accomplish a useful task, the author does not guaranty these scripts to be useful in any way. Users of these scripts are responsible for the results of their use. The code herein is free as in 'beer and chicken', there are no restrictions inferred. Anyone may alter/distribute any part of the code at will, without any need to notify or request permissions from the original author or any other person.