No client-side software is needed. On WinXX the standard smb
protocol is used to extract backup data. On linux or unix clients,
-tar over ssh/rsh/nfs is used to extract backup data (or alternatively
-Samba can be installed on the linux or unix client to provide smb shares).
+rsync or tar (over ssh/rsh/nfs) is used to extract backup data.
+Alternatively, rsync can also be used on WinXX (using cygwin),
+and Samba could be installed on the linux or unix client to
+provide smb shares).
=item *
any backup directly from the CGI interface. Zip or Tar archives
for selected files or directories from any backup can also be
downloaded from the CGI interface. Finally, direct restore to
-the client machine (using SMB or tar) for selected files or
+the client machine (using smb or tar) for selected files or
directories is also supported from the CGI interface.
=item *
=item Incremental Backup
An incremental backup is a backup of files that have changed (based on their
-modification time) since the last successful full backup. To be safe,
-BackupPC backups all files that have changed since one hour prior to the
-start of the last successful full backup. BackupPC can also be configured
-to keep a certain number of incremental backups, and to keep a smaller
-number of very old incremental backups. (BackupPC does not support
-multi-level incremental backups, although it would be easy to do so.)
+modification time) since the last successful full backup. For SMB and
+tar, BackupPC backups all files that have changed since one hour prior to the
+start of the last successful full backup. Rsync is more clever: any files
+who attributes have changed (ie: uid, gid, mtime, modes, size) since the
+last full are backed up. Deleted and new files are also detected by
+Rsync incrementals (SMB and tar are not able to detect deleted files or
+new files whose modification time is prior to the last full dump.
+
+BackupPC can also be configured to keep a certain number of incremental
+backups, and to keep a smaller number of very old incremental backups.
+(BackupPC does not support multi-level incremental backups, although it
+would be easy to do so.)
BackupPC's CGI interface "fills-in" incremental backups based on the
last full backup, giving every backup a "full" appearance. This makes
At other sites a secondary tape backup will be required. This tape
backup can be done perhaps weekly from the BackupPC pool file system.
-One comment: in the US in particular, permanent backups of things like
-email are becoming strongly discouraged by lawyers because of discovery
-prior to possible litigation. Using BackupPC without tape backup allows
-recent file changes or losses to be restored, but without keeping a
-history more than a month or two old (although this doesn't avoid the
-problem of old emails languishing in user's email folders forever).
-
=back
=head2 Resources
=item Mail lists
-Two BackupPC mailing lists exist for announcements (backuppc-announce)
-and reporting information, asking questions, discussing development or
-any other topic relevant to BackupPC (backuppc-users).
+Three BackupPC mailing lists exist for announcements (backuppc-announce),
+developers (backuppc-devel), and a general user list for support, asking
+questions or any other topic relevant to BackupPC (backuppc-users).
-You are encouraged to subscribe to either the backuppc-announce
-or backuppc-users mail list on sourceforge.net at either:
+You can subscribe to these lists by visiting:
http://lists.sourceforge.net/lists/listinfo/backuppc-announce
http://lists.sourceforge.net/lists/listinfo/backuppc-users
+ http://lists.sourceforge.net/lists/listinfo/backuppc-devel
The backuppc-announce list is moderated and is used only for
important announcements (eg: new versions). It is low traffic.
-You only need to subscribe to one list: backuppc-users also
-receives any messages on backuppc-announce.
+You only need to subscribe to one of backuppc-announce and
+backuppc-users: backuppc-users also receives any messages on
+backuppc-announce.
+
+The backuppc-devel list is only for developers who are working on BackupPC.
+Do not post questions or support requests there. But detailed technical
+discussions should happen on this list.
To post a message to the backuppc-users list, send an email to
=item Other Programs of Interest
If you want to mirror linux or unix files or directories to a remote server
-you should look at rsync, L<http://rsync.samba.org>.
-
-Two popular open source packages that do tape backup are
-Amanda (L<http://www.amanda.org>) and
-afbackup (L<http://sourceforge.net/projects/afbackup>).
+you should consider rsync, L<http://rsync.samba.org>. BackupPC now uses
+rsync as a transport mechanism; if you are already an rsync user you
+can think of BackupPC as adding efficient storage (compression and
+pooling) and a convenient user interface to rsync.
+
+Unison is a utility that can do two-way, interactive, synchronization.
+See L<http://www.cis.upenn.edu/~bcpierce/unison>.
+
+Three popular open source packages that do tape backup are
+Amanda (L<http://www.amanda.org>),
+afbackup (L<http://sourceforge.net/projects/afbackup>), and
+Bacula (L<http://www.bacula.org>).
Amanda can also backup WinXX machines to tape using samba.
These packages can be used as back ends to BackupPC to backup the
BackupPC server data to tape.
+Various programs and scripts use rsync to provide hardlinked backups.
+See, for example, Mike Rubel's site (L<http://www.mikerubel.org>),
+J. W. Schultz's dirvish (L<http://www.pegasys.ws/dirvish>),
+and John Bowman's rlbackup (L<http://www.math.ualberta.ca/imaging/rlbackup>).
+BackupPC provides many additional features, such as compressed storage,
+hardlinking any matching files (rather than just files with the same name),
+and storing special files without root priviliges. But these other scripts
+provide simple and effective solutions and are worthy of consideration.
+
=back
=head2 Road map
=item *
-Adding support for rsync as a transport method, in addition to
-smb and tar. This will give big savings in network traffic for
-linux/unix clients. I haven't decided whether to save the pool file
-rsync checksums (that would double the number of files in the pool, but
-eliminate most server disk reads), or recompute them every time. I expect
-to use native rsync on the client side. On the server, rsync would
-need to understand the compressed file format, the file name mangling
-and the attribute files, so I will either have to add features to rsync
-or emulate rsync on the server side in perl.
+Adding hardlink support to rsync.
+
+=item *
+
+Adding block and file checksum caching to rsync. This will significantly
+increase performance since the server doesn't have to read each file
+twice to compute the block and file checksums.
=item *
=item *
-Resuming incomplete completed full backups. Useful if a machine
+Allow editing of config parameters via the CGI interface. Users should
+have permission to edit a subset of the parameters for their clients.
+Additionally, allow an optional self-service capability so that users
+can sign up and setup their own clients with no need for IT support.
+
+=item *
+
+Add backend SQL support for various BackupPC metadata, including
+configuration parameters, client lists, and backup and restore
+information. At installation time the backend data engine will
+be specified (eg: MySQL, ascii text etc).
+
+=item *
+
+Disconnect the notion of a physical host and a backup client.
+Currently there is a one-to-one match between physical hosts
+and backup clients. Instead, the current notion of a host
+should be replaced by a backup client. Each backup client
+corresponds to a physical host. A physical host could have
+several backup clients. This is useful for backing up
+different types of data, or backing up different portions
+of a machine with different frequencies or settings.
+
+=item *
+
+Resuming incomplete full backups. Useful if a machine
(eg: laptop) is disconnected from the network during a backup,
-or if the user manually stops a backup. This would work by
-excluding directories that were already complete.
+or if the user manually stops a backup. This would be supported
+initially for rsync. The partial dump would be kept, and be
+browsable. When the next dump starts, an incremental against
+the partial dump would be done to make sure it was up to date,
+and then the rest of the full dump would be done.
+
+=item *
+
+Replacing smbclient with the perl module FileSys::SmbClient. This
+gives much more direct control of the smb transfer, allowing
+incrementals to depend on any attribute change (eg: exist, mtime,
+file size, uid, gid), and better support for include and exclude.
+Currently smbclient incrementals only depend upon mtime, so
+deleted files or renamed files are not detected. FileSys::SmbClient
+would also allow resuming of incomplete full backups in the
+same manner as rsync will.
+
+=item *
+
+Support --listed-incremental or --incremental for tar,
+so that incrementals will depend upon any attribute change (eg: exist,
+mtime, file size, uid, gid), rather than just mtime. This will allow
+tar to be to as capable as FileSys::SmbClient and rsync.
+
+=item *
+
+For rysnc (and smb when FileSys::SmbClient is supported, and tar when
+--listed-incremental is supported) support multi-level incrementals.
+In fact, since incrementals will now be more "accurate", you could
+choose to never to full dumps (except the first time), or at a
+minimum do them infrequently: each incremental would depend upon
+the last, giving a continuous chain of differential dumps.
+
+=item *
+
+Add a backup browsing feature that shows backup history by file.
+So rather than a single directory view, it would be a table showing
+the files (down) and the backups (across). The internal hardlinks
+encode which files are identical across backups. You could immediately
+see which files changed on which backups.
=item *
around 15-20%, which isn't spectacular, and likely not worth the
implementation effort. The program xdelta (v1) on SourceForge (see
L<http://sourceforge.net/projects/xdelta>) uses an rsync algorithm for
-doing efficient binary file deltas.
+doing efficient binary file deltas. Rather than using an external
+program, File::RsyncP will eventually get the necessary delta
+generataion code from rsync.
=back
evaluated BackupPC but didn't use it because it doesn't ...".
Beyond being a satisfied user and telling other people about it, everyone
-is encouraged to add links to backuppc.sourceforge.net (I'll see then
-via Google) or otherwise publicize BackupPC. Unlike the commercial
-products in this space, I have a zero budget (in both time and money)
-for marketing, PR and advertising, so it's up to all of you!
+is encouraged to add links to L<http://backuppc.sourceforge.net> (I'll
+see then via Google) or otherwise publicize BackupPC. Unlike the
+commercial products in this space, I have a zero budget (in both
+time and money) for marketing, PR and advertising, so it's up to
+all of you!
Also, everyone is encouraged to contribute patches, bug reports, feature
and design suggestions, code, and documentation corrections or
=item *
-Perl modules Compress::Zlib and Archive::Zip. Try "perldoc Compress::Zlib"
-and "perldoc Archive::Zip" to see if you have these modules. If not,
-fetch them from L<http://www.cpan.org> and see the instructions below
-for how to build and install them.
+Perl modules Compress::Zlib, Archive::Zip and Rsync. Try "perldoc
+Compress::Zlib" and "perldoc Archive::Zip" to see if you have these
+modules. If not, fetch them from L<http://www.cpan.org> and see the
+instructions below for how to build and install them.
+
+The Rsync module is available from L<http://backuppc.sourceforge.net>.
+You'll need to install the Rsync module if you want to use Rsync as
+a transport method.
=item *
=item *
+If you are using rsync to backup linux/unix machines you should have
+version 2.5.5 on each client machine. See L<http://rsync.samba.org>.
+Use "rsync --version" to check your version.
+
+For BackupPC to use Rsync you will also need to install the perl
+File::RsyncP module, which is available from
+L<http://perlrsync.sourceforge.net>.
+Version 0.20 is required.
+
+=item *
+
The Apache web server, see L<http://www.apache.org>, preferably built
with mod_perl support.
=head2 Step 2: Installing the distribution
-First off, to enable compression, you will need to install Compress::Zlib
-from L<http://www.cpan.org>. It is optional, but strongly recommended.
-Also, to support restore via Zip archives you will need to install
-Archive::Zip, also from L<http://www.cpan.org>. You can run
-"perldoc Compress::Zlib" to see if this module is installed.
-To build and install these packages you should run these commands:
+First off, there are three perl modules you should install.
+These are all optional, but highly recommended:
+
+=over 4
+
+=item Compress::Zlib
+
+To enable compression, you will need to install Compress::Zlib
+from L<http://www.cpan.org>.
+You can run "perldoc Compress::Zlib" to see if this module is installed.
+
+=item Archive::Zip
+
+To support restore via Zip archives you will need to install
+Archive::Zip, also from L<http://www.cpan.org>.
+You can run "perldoc Archive::Zip" to see if this module is installed.
+
+=item File::RsyncP
+
+To use rsync and rsyncd with BackupPC you will need to install File::RsyncP.
+You can run "perldoc File::RsyncP" to see if this module is installed.
+File::RsyncP is available from L<http://perlrsync.sourceforge.net>.
+Version 0.20 is required.
+
+=back
+
+To build and install these packages, fetch the tar.gz file and
+then run these commands:
tar zxvf Archive-Zip-1.01.tar.gz
cd Archive-Zip-1.01
make test
make install
+The same sequence of commands can be used for each module.
+
Now let's move onto BackupPC itself. After fetching
BackupPC-__VERSION__.tar.gz, run these commands as root:
=item *
-As an environment variable PASSWD set before BackupPC starts. If you
-start BackupPC manually this PASSWD must be set manually first.
+As an environment variable BPC_SMB_PASSWD set before BackupPC starts.
+If you start BackupPC manually the BPC_SMB_PASSWD variable must be set
+manually first. For backward compatability for v1.5.0 and prior, the
+environment variable PASSWD can be used if BPC_SMB_PASSWD is not set.
+Warning: on some systems it is possible to see environment variables of
+running processes.
=item *
-Alternatively the PASSWD setting can be included in /etc/init.d/backuppc,
-in which case you must make sure this file is not world (other) readable.
+Alternatively the BPC_SMB_PASSWD setting can be included in
+/etc/init.d/backuppc, in which case you must make sure this file
+is not world (other) readable.
=item *
files. Ssh is setup so that BackupPC on the server (an otherwise low
privileged user) can ssh as root on the client, without being prompted
for a password. There are two common versions of ssh: v1 and v2. Here
-are some instructions for one way to setup ssh v2:
+are some instructions for one way to setup ssh. (Check which version
+of SSH you have by typing "ssh" or "man ssh".)
+
+=over 4
+
+=item OpenSSH Instructions
=over 4
=item Key generation
-As root on the client machine, use ssh2-keygen to generate a
+As root on the client machine, use ssh-keygen to generate a
+public/private key pair, without a pass-phrase:
+
+ ssh-keygen -t rsa -N ''
+
+This will save the public key in ~/.ssh/id_rsa.pub and the private
+key in ~/.ssh/id_rsa.
+
+=item BackupPC setup
+
+Repeat the above steps for the BackupPC user (__BACKUPPCUSER__) on the server.
+Make a copy of the public key to make it recognizable, eg:
+
+ ssh-keygen -t rsa -N ''
+ cp ~/.ssh/id_rsa.pub ~/.ssh/BackupPC_id_rsa.pub
+
+See the ssh and sshd manual pages for extra configuration information.
+
+=item Key exchange
+
+To allow BackupPC to ssh to the client as root, you need to place
+BackupPC's public key into root's authorized list on the client.
+Append BackupPC's public key (BackupPC_id_rsa.pub) to root's
+~/.ssh/authorized_keys2 file on the client:
+
+ touch ~/.ssh/authorized_keys2
+ cat BackupPC_id_rsa.pub >> ~/.ssh/authorized_keys2
+
+You should edit ~/.ssh/authorized_keys2 and add further specifiers,
+eg: from, to limit which hosts can login using this key. For example,
+if your BackupPC host is called backuppc.my.com, there should be
+one line in ~/.ssh/authorized_keys2 that looks like:
+
+ from="backuppc.my.com" ssh-rsa [base64 key, eg: ABwBCEAIIALyoqa8....]
+
+=item Fix permissions
+
+You will probably need to make sure that all the files
+in ~/.ssh have no group or other read/write permission:
+
+ chmod -R go-rwx ~/.ssh
+
+You should do the same thing for the BackupPC user on the server.
+
+=item Testing
+
+As the BackupPC user on the server, verify that this command:
+
+ ssh -l root clientHostName whoami
+
+prints
+
+ root
+
+You might be prompted the first time to accept the client's host key and
+you might be prompted for root's password on the client. Make sure that
+this command runs cleanly with no prompts after the first time. You
+might need to check /etc/hosts.equiv on the client. Look at the
+man pages for more information. The "-v" option to ssh is a good way
+to get detailed information about what fails.
+
+=back
+
+=item SSH2 Instructions
+
+=over 4
+
+=item Key generation
+
+As root on the client machine, use ssh-keygen2 to generate a
public/private key pair, without a pass-phrase:
ssh-keygen2 -t rsa -P
+or:
+
+ ssh-keygen -t rsa -N ''
+
+(This command might just be called ssh-keygen on your machine.)
+
This will save the public key in /.ssh2/id_rsa_1024_a.pub and the private
key in /.ssh2/id_rsa_1024_a.
man pages for more information. The "-v" option to ssh2 is a good way
to get detailed information about what fails.
-=item ssh version 1 instructions
+=back
+
+=item SSH version 1 Instructions
The concept is identical and the steps are similar, but the specific
commands and file names are slightly different.
in a single large file and keeps it locked whenever it is running.
Since many users keep Outlook running all the time their machine
is up their Outlook file will not be backed up. Sadly, this file
-is the most important file to backup. How do commercial backup
-systems solve this problem?
+is the most important file to backup. As one workaround, Microsoft has
+a user-level application that periodically asks the user if they want to
+make a copy of their outlook.pst file. This copy can then be backed up
+by BackupPC. See L<http://office.microsoft.com/downloads/2002/pfbackup.aspx>.
Similarly, all of the data for WinXX services like SQL databases,
-Exchange etc won't be backed up.
+Exchange etc won't be backed up. If these applications support
+some kind of export or utility to save their data to disk then this
+can =used to create files that BackupPC can backup.
So far, the best that BackupPC can do is send warning emails to
the user saying that their outlook files haven't been backed up in
projects / BackupPC.git / blobdiff