=item Full Backup
-A full backup is a complete backup of a share. BackupPC can be configured to
-do a full backup at a regular interval (often weekly). BackupPC can also
-be configured to keep a certain number of full backups, and to keep
-a smaller number of very old full backups.
+A full backup is a complete backup of a share. BackupPC can be
+configured to do a full backup at a regular interval (typically
+weekly). BackupPC can be configured to keep a certain number
+of full backups. Exponential expiry is also supported, allowing
+full backups with various vintages to be kept (for example, a
+settable number of most recent weekly fulls, plus a settable
+number of older fulls that are 2, 4, 8, or 16 weeks apart).
=item Incremental Backup
last full backup, giving every backup a "full" appearance. This makes
browsing and restoring backups easier.
+=item Partial Backup
+
+When a full backup fails or is canceled, and some files have already
+been backed up, BackupPC keeps a partial backup containing just the
+files that were backed up successfully. The partial backup is removed
+when the next successful backup completes, or if another full backup
+fails resulting in a newer partial backup. A failed full backup
+that has not backed up any files, or any failed incremental backup,
+is removed; no partial backup is saved in these cases.
+
+The partial backup may be browsed or used to restore files just like
+a successful full or incremental backup.
+
+With the rsync transfer method the partial backup is used to resume
+the next full backup, avoiding the need to retransfer the file data
+already in the partial backup.
+
=item Identical Files
BackupPC pools identical files using hardlinks. By "identical
developers (backuppc-devel), and a general user list for support, asking
questions or any other topic relevant to BackupPC (backuppc-users).
+The lists are archived on SourceForge and Gmane. The SourceForge lists
+are not always up to date and the searching is limited, so Gmane is
+a good alternative. See:
+
+ http://news.gmane.org/index.php?prefix=gmane.comp.sysutils.backup.backuppc
+ http://sourceforge.net/mailarchive/forum.php?forum_id=503
+
You can subscribe to these lists by visiting:
http://lists.sourceforge.net/lists/listinfo/backuppc-announce
=item *
-Adding hardlink support to rsync.
+Adding hardlink support to rsync. Currently the tar XferMethod
+correctly saves/restores hardlinks, but rsync does not. Rsync
+2.6.2 has greatly improved the efficiency of saving hardlink
+information, so File::RsyncP and BackupPC::Xfer::RsyncFileIO need
+the corresponding changes. Hardlink support is necessary for
+doing a bare-metal restore of a *nix system.
+
+=item *
+
+Adding more complete utf8 support. BackupPC should use utf8 as the
+native charset for storing file names, and the CGI script should emit
+utf8 so that the file names can be rendered correctly. Additional
+configuration parameters should allow you to specify the client Xfer
+charset (ie: the filcharset delivered by the XferMethod). BackupPC
+should encode/decode between this charset and utf8 when doing a
+backup/restore. That way BackupPC can store all files in utf8 no
+matter what charset is used by the XferMethod to deliver the file
+names. Secondly, the CGI charset should be configurable (default
+utf8) and the CGI script BackupPC_Admin should encode the utf8 file
+names in the desired output charset. Finally, the charset used to
+deliver file names when restoring individual file names should also
+be configurable, and again BackupPC_Admin should encode the file
+names in this charset (again, utf8 default). That should allow the
+"Save As" IE dialog to default to the correct file name.
=item *
=item *
+Because of file name mangling (adding "f" to start of each file
+name) and with pending utf8 changes, BackupPC is not able to
+store files whose file name length is 255 bytes or greater. The
+format of the attrib file should be extended so that very long
+file names (ie: >= 255) are abbreviated on the file system, but
+the full name is stored in the attrib file. This could also be
+used to eliminate the leading "f", since that is there to avoid
+collisions with the attrib file.
+
+=item *
+
+Adding support for FTP (via Net::FTP) and/or wget (for HTTP and FTP)
+as additional XferMethods. One question with ftp is whether you can
+support incrementals. It should be possible. For example, you could
+use Net::FTP->ls to list each directory, and get the file sizes and
+mtimes. That would allow incrementals to be supported: only backup
+the files that have different sizes or mtimes for an incremental.
+You also need the ls() function to recurse directories. The code
+would need to be robust to the different formats returned by ls() on
+different clients.
+
+For wget there would be a new module called BackupPC::Xfer::Wget that
+uses wget. Wget can do both http and ftp. Certainly backing up a web
+site via ftp is better than http, especially when there is active
+content and not just static pages. But the benefit of supporting http
+is that you could use it to backup config pages of network hardware
+(eg: routers etc). So if a router fails you have a copy of the config
+screens and settings. (And the future tripwire feature on the todo
+list could tell you if someone messed with the router settings.)
+Probably the only choice with wget is to fetch all the files
+(either via ftp or http) into a temporary directory, then run
+tar on that directory and pipe it into BackupPC_tarExtract.
+
+The advantage of using wget is you get both http and ftp.
+The disadvantage is that you can't support incrementals
+with wget, but you can with Net::FTP. Also people will
+find wget harder to configure and run.
+
+=item *
+
Replacing smbclient with the perl module FileSys::SmbClient. This
gives much more direct control of the smb transfer, allowing
incrementals to depend on any attribute change (eg: exist, mtime,
Support --listed-incremental or --incremental for tar,
so that incrementals will depend upon any attribute change (eg: exist,
mtime, file size, uid, gid), rather than just mtime. This will allow
-tar to be to as capable as FileSys::SmbClient and rsync.
+tar to be to as capable as FileSys::SmbClient and rsync. This is
+low priority since rsync is really the preferred method.
+
+=item *
+
+In addition to allowing a specific backup (incremental or full) to
+be started from the CGI interface, also allow a "regular" backup
+to be started. This would behave just like a regular background
+backup and determine whether a full, incremental or nothing
+should be done.
=item *
For rysnc (and smb when FileSys::SmbClient is supported, and tar when
--listed-incremental is supported) support multi-level incrementals.
In fact, since incrementals will now be more "accurate", you could
-choose to never to full dumps (except the first time), or at a
+choose to never do full backups (except the first time), or at a
minimum do them infrequently: each incremental would depend upon
-the last, giving a continuous chain of differential dumps.
+the last, giving a continuous chain of differential backups.
+
+=item *
+
+Allow diffs between two different backed up files to be displayed.
+The history feature in 2.1.0 does show files that are the same
+between backups. Most often we would like to just take the diff of
+the same host, path and file between different backups, but it would
+be nice to generalize it to any pair of files (ie: different hosts,
+backups, paths etc). But I'm not sure how the user interface would
+look.
=item *
implementation effort. The program xdelta (v1) on SourceForge (see
L<http://sourceforge.net/projects/xdelta>) uses an rsync algorithm for
doing efficient binary file deltas. Rather than using an external
-program, File::RsyncP will eventually get the necessary delta
+program, File::RsyncP would eventually get the necessary delta
generation code from rsync.
=back
all of you!
Also, everyone is encouraged to contribute patches, bug reports, feature
-and design suggestions, code, and documentation corrections or
+and design suggestions, new code, and documentation corrections or
improvements.
=head1 Installing BackupPC
=item *
Perl version 5.6.0 or later. BackupPC has been tested with
-version 5.6.0, 5.6.1 and 5.8.0. If you don't have perl, please
-see L<http://www.cpan.org>.
+version 5.6.0, 5.6.1, 5.8.0, 5.8.1 and 5.8.2. If you don't
+have perl, please see L<http://www.cpan.org>.
=item *
Version 2.2.0 or later of Samba is required (smbclient's tar feature in
2.0.8 and prior has bugs for file path lengths around 100 characters
and generates bad output when file lengths change during the backup).
+Samba versions 3.x are stable and recommended instead of 2.x.
See L<http://www.samba.org> for source and binaries. It's pretty easy to
fetch and compile samba, and just grab smbclient and nmblookup, without
If you are using rsync to backup linux/unix machines you should have
version 2.5.5 or higher on each client machine. See
-L<http://rsync.samba.org>. Use "rsync --version" to check your
-version.
+L<http://rsync.samba.org>. Use "rsync --version" to check your version.
For BackupPC to use Rsync you will also need to install the perl
File::RsyncP module, which is available from
-L<http://perlrsync.sourceforge.net>. Version 0.44 or later is required.
+L<http://perlrsync.sourceforge.net>.
+Version 0.51 or later is required.
=item *
To use rsync and rsyncd with BackupPC you will need to install File::RsyncP.
You can run "perldoc File::RsyncP" to see if this module is installed.
File::RsyncP is available from L<http://perlrsync.sourceforge.net>.
-Version 0.44 or later is required.
+Version 0.51 or later is required.
=back
cd BackupPC-__VERSION__
perl configure.pl
-You will be prompted for the full paths of various executables, and
-you will be prompted for the following information:
+The configure.pl also accepts command-line options if you wish
+to run it in a non-interactive manner. It has self-contained
+documentation for all the command-line options, which you can
+read with perldoc:
+
+ perldoc configure.pl
+
+When you run configure.pl you will be prompted for the full paths
+of various executables, and you will be prompted for the following
+information:
=over 4
Note also that the $Conf{ClientNameAlias} feature does not work for
clients with DHCP set to 1.
-
+
=item User name
This should be the unix login/email name of the user who "owns" or uses
contains instructions for running rsync as a service, so it starts
automatically everytime you boot your machine.
+If you build your own rsync, for rsync 2.6.2 it is strongly
+recommended you apply the patch in the cygwin-rsync package on
+L<http://backuppc.sourceforge.net>. This patch adds the --checksum-seed
+option for checksum caching, and also sends all errors to the client,
+which is important so BackupPC can log all file access errors.
+
Otherwise, to use SMB, you need to create shares for the data you want
to backup. Open "My Computer", right click on the drive (eg: C), and
select "Sharing..." (or select "Properties" and select the "Sharing"
The relevant configuration settings are $Conf{RsyncClientPath},
$Conf{RsyncClientCmd}, $Conf{RsyncClientRestoreCmd}, $Conf{RsyncShareName},
-$Conf{RsyncArgs}, $Conf{RsyncRestoreArgs} and $Conf{RsyncLogLevel}.
+$Conf{RsyncArgs}, and $Conf{RsyncRestoreArgs}.
=item rsyncd
The relevant configuration settings are $Conf{RsyncdClientPort},
$Conf{RsyncdUserName}, $Conf{RsyncdPasswd}, $Conf{RsyncdAuthRequired},
-$Conf{RsyncShareName}, $Conf{RsyncArgs}, $Conf{RsyncRestoreArgs}
-and $Conf{RsyncLogLevel}. In the case of rsyncd, $Conf{RsyncShareName}
-is the name of an rsync module (ie: the thing in square brackets in
-rsyncd's conf file -- see rsyncd.conf), not a file system path.
+$Conf{RsyncShareName}, $Conf{RsyncArgs}, and $Conf{RsyncRestoreArgs}.
+$Conf{RsyncShareName} is the name of an rsync module (ie: the thing
+in square brackets in rsyncd's conf file -- see rsyncd.conf), not a
+file system path.
Be aware that rsyncd will remove the leading '/' from path names in
symbolic links if you specify "use chroot = no" in the rsynd.conf file.
set to 'archive'. This allows for multiple configurations at sites where
there might be a combination of tape and cd/dvd backups being made.
+BackupPC provides a menu that allows one or more hosts to be archived.
+The most recent backup of each host is archived using BackupPC_tarCreate,
+and the output is optionally compressed and split into fixed-sized
+files (eg: 650MB).
+
+The archive for each host is done by default using
+__INSTALLDIR__/BackupPC_archiveHost. This script can be copied
+and customized as needed.
+
=head2 Configuring an Archive Host
To create an Archive Host, add it to the hosts file just as any other host
$Conf{XferMethod} = 'archive';
To further customise the archive's parameters you can adding the changed
-parameters in the host's config.pl file. The parameters are explained in the config.pl
-file.
+parameters in the host's config.pl file. The parameters are explained in
+the config.pl file. Parameters may be fixed or the user can be allowed
+to change them (eg: output device).
-The example archive programs included with BackupPC are for a CD and
-Tape archive. The programs are called BackupPC_archivecd and
-BackupPC_archivetape. These are specified by the ArchiveClientCmd configuration
-parameter.
+The per-host archive command is $Conf{ArchiveClientCmd}. By default
+this invokes __INSTALLDIR__/BackupPC_archiveHost, which you can
+copy and customize as necessary.
=head2 Starting an Archive
BackupPC_zcat can be found in __INSTALLDIR__/bin. For each
file name argument it inflates the file and writes it to stdout.
+=head2 Rsync checksum caching
+
+An incremental backup with rsync compares attributes on the client
+with the last full backup. Any files with identical attributes
+are skipped. A full backup with rsync sets the --ignore-times
+option, which causes every file to be examined independent of
+attributes.
+
+Each file is examined by generating block checksums (default 2K
+blocks) on the receiving side (that's the BackupPC side), sending
+those checksums to the client, where the remote rsync matches those
+checksums with the corresponding file. The matching blocks and new
+data is sent back, allowing the client file to be reassembled.
+A checksum for the entire file is sent to as an extra check the
+the reconstructed file is correct.
+
+This results in significant disk IO and computation for BackupPC:
+every file in a full backup, or any file with non-matching attributes
+in an incremental backup, needs to be uncompressed, block checksums
+computed and sent. Then the receiving side reassembles the file and
+has to verify the whole-file checksum. Even if the file is identical,
+prior to 2.1.0, BackupPC had to read and uncompress the file twice,
+once to compute the block checksums and later to verify the whole-file
+checksum.
+
+Starting in 2.1.0, BackupPC supports optional checksum caching,
+which means the block and file checksums only need to be computed
+once for each file. This results in a significant performance
+improvement. This only works for compressed pool files.
+It is enabled by adding
+
+ '--checksum-seed=32761',
+
+to $Conf{RsyncArgs} and $Conf{RsyncRestoreArgs}.
+
+Rsync versions prior to and including rsync-2.6.2 need a small patch to
+add support for the --checksum-seed option. This patch is available in
+the cygwin-rsyncd package at L<http://backuppc.sourceforge.net>.
+This patch is already included in rsync CVS, so it will be standard
+in future versions of rsync.
+
+When this option is present, BackupPC will add block and file checksums
+to the compressed pool file the next time a pool file is used and it
+doesn't already have cached checksums. The first time a new file is
+written to the pool, the checksums are not appended. The next time
+checksums are needed for a file, they are computed and added. So the
+full performance benefit of checksum caching won't be noticed until the
+third time a pool file is used (eg: the third full backup).
+
+With checksum caching enabled, there is a risk that should a file's contents
+in the pool be corrupted due to a disk problem, but the cached checksums
+are still correct, the corruption will not be detected by a full backup,
+since the file contents are no longer read and compared. To reduce the
+chance that this remains undetected, BackupPC can recheck cached checksums
+for a fraction of the files. This fraction is set with the
+$Conf{RsyncCsumCacheVerifyProb} setting. The default value of 0.01 means
+that 1% of the time a file's checksums are read, the checksums are verified.
+This reduces performance slightly, but, over time, ensures that files
+contents are in sync with the cached checksums.
+
+The format of the cached checksum data can be discovered by looking at
+the code. Basically, the first byte of the compressed file is changed
+to denote that checksums are appended. The block and file checksum
+data, plus some other information and magic word, are appended to the
+compressed file. This allows the cache update to be done in-place.
+
=head2 File name mangling
Backup file names are stored in "mangled" form. Each node of
=head1 Copyright
-Copyright (C) 2001-2003 Craig Barratt
+Copyright (C) 2001-2004 Craig Barratt
=head1 Credits
projects / BackupPC.git / blobdiff