At other sites a secondary tape backup will be required. This tape
backup can be done perhaps weekly from the BackupPC pool file system.
-One comment: in the US in particular, permanent backups of things like
-email are becoming strongly discouraged by lawyers because of discovery
-prior to possible litigation. Using BackupPC without tape backup allows
-recent file changes or losses to be restored, but without keeping a
-history more than a month or two old (although this doesn't avoid the
-problem of old emails languishing in user's email folders forever).
-
=back
=head2 Resources
=item Mail lists
-Two BackupPC mailing lists exist for announcements (backuppc-announce)
-and reporting information, asking questions, discussing development or
-any other topic relevant to BackupPC (backuppc-users).
+Three BackupPC mailing lists exist for announcements (backuppc-announce),
+developers (backuppc-devel), and a general user list for support, asking
+questions or any other topic relevant to BackupPC (backuppc-users).
-You are encouraged to subscribe to either the backuppc-announce
-or backuppc-users mail list on sourceforge.net at either:
+You can subscribe to these lists by visiting:
http://lists.sourceforge.net/lists/listinfo/backuppc-announce
http://lists.sourceforge.net/lists/listinfo/backuppc-users
+ http://lists.sourceforge.net/lists/listinfo/backuppc-devel
The backuppc-announce list is moderated and is used only for
important announcements (eg: new versions). It is low traffic.
-You only need to subscribe to one list: backuppc-users also
-receives any messages on backuppc-announce.
+You only need to subscribe to one of backuppc-announce and
+backuppc-users: backuppc-users also receives any messages on
+backuppc-announce.
+
+The backuppc-devel list is only for developers who are working on BackupPC.
+Do not post questions or support requests there. But detailed technical
+discussions should happen on this list.
To post a message to the backuppc-users list, send an email to
=item Other Programs of Interest
If you want to mirror linux or unix files or directories to a remote server
-you should consider rsync, L<http://rsync.samba.org>. BackupPC uses
+you should consider rsync, L<http://rsync.samba.org>. BackupPC now uses
rsync as a transport mechanism; if you are already an rsync user you
can think of BackupPC as adding efficient storage (compression and
pooling) and a convenient user interface to rsync.
Unison is a utility that can do two-way, interactive, synchronization.
See L<http://www.cis.upenn.edu/~bcpierce/unison>.
-Two popular open source packages that do tape backup are
-Amanda (L<http://www.amanda.org>) and
-afbackup (L<http://sourceforge.net/projects/afbackup>).
+Three popular open source packages that do tape backup are
+Amanda (L<http://www.amanda.org>),
+afbackup (L<http://sourceforge.net/projects/afbackup>), and
+Bacula (L<http://www.bacula.org>).
Amanda can also backup WinXX machines to tape using samba.
These packages can be used as back ends to BackupPC to backup the
BackupPC server data to tape.
+Various programs and scripts use rsync to provide hardlinked backups.
+See, for example, Mike Rubel's site (L<http://www.mikerubel.org>),
+J. W. Schultz's dirvish (L<http://www.pegasys.ws/dirvish>),
+and John Bowman's rlbackup (L<http://www.math.ualberta.ca/imaging/rlbackup>).
+BackupPC provides many additional features, such as compressed storage,
+hardlinking any matching files (rather than just files with the same name),
+and storing special files without root priviliges. But these other scripts
+provide simple and effective solutions and are worthy of consideration.
+
=back
=head2 Road map
=item *
-Adding support for rsync as a transport method, in addition to
-smb and tar. This will give big savings in network traffic for
-linux/unix clients. I haven't decided whether to save the pool file
-rsync checksums (that would double the number of files in the pool, but
-eliminate most server disk reads), or recompute them every time. I expect
-to use native rsync on the client side. On the server, rsync would
-need to understand the compressed file format, the file name mangling
-and the attribute files, so I will either have to add features to rsync
-or emulate rsync on the server side in perl.
+Adding hardlink support to rsync.
+
+=item *
+
+Adding block and file checksum caching to rsync. This will significantly
+increase performance since the server doesn't have to read each file
+twice to compute the block and file checksums.
=item *
=item *
-Resuming incomplete completed full backups. Useful if a machine
+Allow editing of config parameters via the CGI interface. Users should
+have permission to edit a subset of the parameters for their clients.
+Additionally, allow an optional self-service capability so that users
+can sign up and setup their own clients with no need for IT support.
+
+=item *
+
+Add backend SQL support for various BackupPC metadata, including
+configuration parameters, client lists, and backup and restore
+information. At installation time the backend data engine will
+be specified (eg: MySQL, ascii text etc).
+
+=item *
+
+Disconnect the notion of a physical host and a backup client.
+Currently there is a one-to-one match between physical hosts
+and backup clients. Instead, the current notion of a host
+should be replaced by a backup client. Each backup client
+corresponds to a physical host. A physical host could have
+several backup clients. This is useful for backing up
+different types of data, or backing up different portions
+of a machine with different frequencies or settings.
+
+=item *
+
+Resuming incomplete full backups. Useful if a machine
(eg: laptop) is disconnected from the network during a backup,
-or if the user manually stops a backup. This would work by
-excluding directories that were already complete.
+or if the user manually stops a backup. This would be supported
+initially for rsync. The partial dump would be kept, and be
+browsable. When the next dump starts, an incremental against
+the partial dump would be done to make sure it was up to date,
+and then the rest of the full dump would be done.
+
+=item *
+
+Replacing smbclient with the perl module FileSys::SmbClient. This
+gives much more direct control of the smb transfer, allowing
+incrementals to depend on any attribute change (eg: exist, mtime,
+file size, uid, gid), and better support for include and exclude.
+Currently smbclient incrementals only depend upon mtime, so
+deleted files or renamed files are not detected. FileSys::SmbClient
+would also allow resuming of incomplete full backups in the
+same manner as rsync will.
+
+=item *
+
+Support --listed-incremental or --incremental for tar,
+so that incrementals will depend upon any attribute change (eg: exist,
+mtime, file size, uid, gid), rather than just mtime. This will allow
+tar to be to as capable as FileSys::SmbClient and rsync.
+
+=item *
+
+For rysnc (and smb when FileSys::SmbClient is supported, and tar when
+--listed-incremental is supported) support multi-level incrementals.
+In fact, since incrementals will now be more "accurate", you could
+choose to never to full dumps (except the first time), or at a
+minimum do them infrequently: each incremental would depend upon
+the last, giving a continuous chain of differential dumps.
+
+=item *
+
+Add a backup browsing feature that shows backup history by file.
+So rather than a single directory view, it would be a table showing
+the files (down) and the backups (across). The internal hardlinks
+encode which files are identical across backups. You could immediately
+see which files changed on which backups.
=item *
around 15-20%, which isn't spectacular, and likely not worth the
implementation effort. The program xdelta (v1) on SourceForge (see
L<http://sourceforge.net/projects/xdelta>) uses an rsync algorithm for
-doing efficient binary file deltas.
+doing efficient binary file deltas. Rather than using an external
+program, File::RsyncP will eventually get the necessary delta
+generataion code from rsync.
=back
version 2.5.5 on each client machine. See L<http://rsync.samba.org>.
Use "rsync --version" to check your version.
-For BackupPC to use Rsync you will also need to install the perl Rsync
-module, which is available from L<http://backuppc.sourceforge.net>.
+For BackupPC to use Rsync you will also need to install the perl
+File::RsyncP module, which is available from
+L<http://perlrsync.sourceforge.net>.
+Version 0.20 is required.
=item *
=head2 Step 2: Installing the distribution
-First off, to enable compression, you will need to install Compress::Zlib
-from L<http://www.cpan.org>. It is optional, but strongly recommended.
+First off, there are three perl modules you should install.
+These are all optional, but highly recommended:
+
+=over 4
+
+=item Compress::Zlib
+
+To enable compression, you will need to install Compress::Zlib
+from L<http://www.cpan.org>.
+You can run "perldoc Compress::Zlib" to see if this module is installed.
+
+=item Archive::Zip
+
To support restore via Zip archives you will need to install
-Archive::Zip, also from L<http://www.cpan.org>. You can run
-"perldoc Compress::Zlib" to see if this module is installed.
-Finally, you will need the Rsync module. To build and install these
-packages you should run these commands:
+Archive::Zip, also from L<http://www.cpan.org>.
+You can run "perldoc Archive::Zip" to see if this module is installed.
+
+=item File::RsyncP
+
+To use rsync and rsyncd with BackupPC you will need to install File::RsyncP.
+You can run "perldoc File::RsyncP" to see if this module is installed.
+File::RsyncP is available from L<http://perlrsync.sourceforge.net>.
+Version 0.20 is required.
+
+=back
+
+To build and install these packages, fetch the tar.gz file and
+then run these commands:
tar zxvf Archive-Zip-1.01.tar.gz
cd Archive-Zip-1.01
make test
make install
+The same sequence of commands can be used for each module.
+
Now let's move onto BackupPC itself. After fetching
BackupPC-__VERSION__.tar.gz, run these commands as root:
files. Ssh is setup so that BackupPC on the server (an otherwise low
privileged user) can ssh as root on the client, without being prompted
for a password. There are two common versions of ssh: v1 and v2. Here
-are some instructions for one way to setup ssh v2:
+are some instructions for one way to setup ssh. (Check which version
+of SSH you have by typing "ssh" or "man ssh".)
+
+=over 4
+
+=item OpenSSH Instructions
=over 4
=item Key generation
-As root on the client machine, use ssh2-keygen to generate a
+As root on the client machine, use ssh-keygen to generate a
+public/private key pair, without a pass-phrase:
+
+ ssh-keygen -t rsa -N ''
+
+This will save the public key in ~/.ssh/id_rsa.pub and the private
+key in ~/.ssh/id_rsa.
+
+=item BackupPC setup
+
+Repeat the above steps for the BackupPC user (__BACKUPPCUSER__) on the server.
+Make a copy of the public key to make it recognizable, eg:
+
+ ssh-keygen -t rsa -N ''
+ cp ~/.ssh/id_rsa.pub ~/.ssh/BackupPC_id_rsa.pub
+
+See the ssh and sshd manual pages for extra configuration information.
+
+=item Key exchange
+
+To allow BackupPC to ssh to the client as root, you need to place
+BackupPC's public key into root's authorized list on the client.
+Append BackupPC's public key (BackupPC_id_rsa.pub) to root's
+~/.ssh/authorized_keys2 file on the client:
+
+ touch ~/.ssh/authorized_keys2
+ cat BackupPC_id_rsa.pub >> ~/.ssh/authorized_keys2
+
+You should edit ~/.ssh/authorized_keys2 and add further specifiers,
+eg: from, to limit which hosts can login using this key. For example,
+if your BackupPC host is called backuppc.my.com, there should be
+one line in ~/.ssh/authorized_keys2 that looks like:
+
+ from="backuppc.my.com" ssh-rsa [base64 key, eg: ABwBCEAIIALyoqa8....]
+
+=item Fix permissions
+
+You will probably need to make sure that all the files
+in ~/.ssh have no group or other read/write permission:
+
+ chmod -R go-rwx ~/.ssh
+
+You should do the same thing for the BackupPC user on the server.
+
+=item Testing
+
+As the BackupPC user on the server, verify that this command:
+
+ ssh -l root clientHostName whoami
+
+prints
+
+ root
+
+You might be prompted the first time to accept the client's host key and
+you might be prompted for root's password on the client. Make sure that
+this command runs cleanly with no prompts after the first time. You
+might need to check /etc/hosts.equiv on the client. Look at the
+man pages for more information. The "-v" option to ssh is a good way
+to get detailed information about what fails.
+
+=back
+
+=item SSH2 Instructions
+
+=over 4
+
+=item Key generation
+
+As root on the client machine, use ssh-keygen2 to generate a
public/private key pair, without a pass-phrase:
ssh-keygen2 -t rsa -P
+or:
+
+ ssh-keygen -t rsa -N ''
+
+(This command might just be called ssh-keygen on your machine.)
+
This will save the public key in /.ssh2/id_rsa_1024_a.pub and the private
key in /.ssh2/id_rsa_1024_a.
man pages for more information. The "-v" option to ssh2 is a good way
to get detailed information about what fails.
-=item ssh version 1 instructions
+=back
+
+=item SSH version 1 Instructions
The concept is identical and the steps are similar, but the specific
commands and file names are slightly different.
projects / BackupPC.git / blobdiff