This document tries to describe ASA extensions for BackupPC 3.2.0 Written by Dobrica Pavlinusic 2011-01-27 This is second iteration of adding search over arbitrary filename substrings and archive to CD/DVD media with tracking of copies and additional md5sum creation on them for easy burned media verification. ASA maintains it's data in PostgreSQL and KinoSearch (for faster part-of-filename matching). Since full-text search is single-writer, we need to serialize somehow requests for it's update. Implementation is based on archive host feature in BackupPC using _search_archive.pl configuration file located at /etc/BackupPC/pc/_search_archive.pl This provides us with serialization and hooks around it, but lacked incremental tar creation which is essential because we want to burn always increasing archive on CD/DVD media. This is implemented using new global configuration directive TarCreateIncremental Using BackupPC hooks to integrate and archive host also provided following advantages: - web interface for archive host contains our log messages - all updates are invoked automatically on end of each run (system is always up to date) BackupPC can dump multiple machines in parallel, this invoking our _search_archive host and index update while update from different machine is still in process. Archive host will reject request, but next invocation of same host will fix problem automatically. To be sure that all pending archives are indexed, you can also run cron job which invokes _search_archive on all pending increments: /BackupPC_ASA_ArchiveStart _search_archive backuppc You can also force archival of particual pending backups from single host by adding hostname(s) or hostname:num to backup individual increment. Alternativly, you can use _search_archive web interface to invoke increment creation and indexing. There are two global options which had to be set for all hosts: # # /etc/BackupPC/config.pl # # invoke archive of dump - ASA extension DumpPostCmd was too early $Conf{DumpPostFinishCmd} = '/srv/BackupPC/bin/BackupPC_ASA_ArchiveStart _search_archive backuppc $host'; # dump only incremental changes in tars not whole content - ASA extension $Conf{TarCreateIncremental} = 1; You can manually trigger all pending backups using: BackupPC_ASA_ArchiveStart _search_archive backuppc This will start archive host _search_archive which will run it's configuration: # # /etc/BackupPC/pc/_search_archive.pl # # Set this client's XferMethod to archive to make it an archive host: $Conf{XferMethod} = 'archive'; # The path on the local file system where archives will be written: $Conf{ArchiveDest} = '/data/BackupPC/_search_archive'; # the type and level of compression used on the archive: $Conf{ArchiveComp} = 'gzip'; $Conf{CompressLevel} = 9; # archive media size (in bytes) 4.2Gb for DVD #$Conf{ArchiveMediaSize} = 4200 * 1024 * 1024; # DVD $Conf{ArchiveMediaSize} = 630 * 1024 * 1024; # CD #$Conf{ArchiveMediaSize} = 1440 * 1024; # floppy #$Conf{ArchiveMediaSize} = 42 * 1024 * 1024; # FIXME # A size in megabytes to split the archive in to parts at. # This is useful where the file size of the archive might exceed the # capacity of the removable media. For example specify 700 if you are using CDs. #$Conf{ArchiveSplit} = 650; $Conf{ArchiveSplit} = 100; # FIXME small testing chunks # The amount of parity data to create for the archive using the par2 utility. # In some cases, corrupted archives can be recovered from parity data. $Conf{ArchivePar} = 30; $Conf{ParPath} = '/srv/par2cmdline-0.4-tbb-20100203-lin64/par2'; # http://chuchusoft.com/par2_tbb/download.html # par2cmdline 0.4 with Intel Threading Building Blocks 2.2 # use parallel gzip (speedup on multi-code machines) $Conf{GzipPath} = '/usr/bin/pigz'; # The full command to run to create archives: $Conf{ArchiveClientCmd} = '$Installdir/bin/BackupPC_archiveHost' . ' $tarCreatePath $splitpath $parpath $host $backupnumber' . ' $compression $compext $splitsize $archiveloc $parfile *'; # host provides serialization, so we can safely update fulltext index $Conf{ArchivePreUserCmd} = '/srv/BackupPC/bin/BackupPC_ASA_SearchUpdate -h$HostList'; $Conf{Md5sumPath} = '/usr/bin/md5sum'; # after archives are created, pull data back in database - ASA extension $Conf{ArchivePostUserCmd} = '/srv/BackupPC/bin/BackupPC_ASA_PostArchive_Update -h$HostList -n$BackupList'; # Logging verbosity: $Conf{XferLogLevel} = 1;