[Duplicity-talk] Backup succeeding but not writing manifest / sigtar

Discussion:

Mark Grandi

2016-02-25 08:40:46 UTC

Running the command below seems to work, the backup completes fine, and the errors seem to just correspond to permission errors (i am excluding those directories but i guess duplicity still tries to read inside them....), but when the backup finishes, it doesn't actually write the sigtar or manifest files to the backup location, so essentially the backup fails for all duplicity cares. Running 'collection-status' will delete the sigtar and manifest files from the archive_dir directory, because they are not present in the backup location, and 'cleanup' will just list all the files (the gpg and par2 files) as 'extra files', for the same reason. Why is duplicity not writing / copying these files to the backup directory?

It seems that it creates the manifest.gpg and sigtar.gpg files and writes them to the archive_dir, but fails to copy them over to the backup location. If I copy these files to the backup location manually , then it suddenly recognizes it as a backup.

command:

duplicity --encrypt-sign-key 0x2CB2C0948891C4830B11AFB6E7B785F6FA5CBE16 --log-file ~/Temp/test_duplicity.log --name "Gypaetus-mgrandi" --use-agent --exclude-device-files --volsize 1024 --file-prefix "Gypaetus-mgrandi-" --exclude-filelist /Users/markgrandi/duplicity_exclude.txt / par2+file:///Volumes/mgrandi_256SSD/backup <par2+file:///Volumes/mgrandi_256SSD/backup>

log entries:

Reading globbing filelist /Users/markgrandi/duplicity_exclude.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /.DocumentRevisions-V100
Error accessing possibly locked file /.Spotlight-V100
Error accessing possibly locked file /.Trashes
Error accessing possibly locked file /.file
Error accessing possibly locked file /.fseventsd
Error accessing possibly locked file /Users/Guest/.CFUserTextEncoding
Error accessing possibly locked file /Users/Guest/Desktop
Error accessing possibly locked file /Users/Guest/Documents
Error accessing possibly locked file /Users/Guest/Downloads
Error accessing possibly locked file /Users/Guest/Library
Error accessing possibly locked file /Users/Guest/Movies
Error accessing possibly locked file /Users/Guest/Music
Error accessing possibly locked file /Users/Guest/Pictures
Error accessing possibly locked file /Users/Guest/Public/Drop Box
Error accessing possibly locked file /Users/markgrandi/.viminfo
Error accessing possibly locked file /Users/markgrandi/Library/Saved Application State/com.adobe.flashplayer.installmanager.savedState
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1456382712.45 (Wed Feb 24 23:45:12 2016)
EndTime 1456386174.70 (Thu Feb 25 00:42:54 2016)
ElapsedTime 3462.25 (57 minutes 42.25 seconds)
SourceFiles 429045
SourceFileSize 22295414551 (20.8 GB)
NewFiles 429045
NewFileSize 22295414551 (20.8 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 429045
RawDeltaSize 22276747699 (20.7 GB)
TotalDestinationSizeChange 13905691762 (13.0 GB)
Errors 16
-------------------------------------------------

contents of ~/.cache/<archive_dir>/:

Gypaetus:Gypaetus-mgrandi markgrandi$ ls -la
total 1250080
drwxr-xr-x 6 markgrandi staff 204 Feb 25 00:45 .
drwxr-xr-x 5 markgrandi staff 170 Feb 24 23:45 ..
-rw------- 1 markgrandi staff 295425508 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gpg
-rw------- 1 markgrandi staff 294998550 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gz
-rw------- 1 markgrandi staff 46875395 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest
-rw------- 1 markgrandi staff 2731077 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest.gpg

contents of the backup dir:

Gypaetus:backup markgrandi$ ls -la
total 29885864
drwxr-xr-x 41 markgrandi staff 1394 Feb 25 00:44 .
drwxrwxr-x 19 markgrandi staff 714 Feb 24 23:42 ..
-rw------- 1 markgrandi staff 1073732207 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.par2
-rw------- 1 markgrandi staff 107736576 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073782430 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.par2
-rw------- 1 markgrandi staff 107741376 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073806871 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073771432 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.par2
-rw------- 1 markgrandi staff 107740576 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1020263590 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.par2
-rw------- 1 markgrandi staff 102387776 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073752418 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.par2
-rw------- 1 markgrandi staff 107738976 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807227 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807149 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073742782 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.par2
-rw------- 1 markgrandi staff 107737376 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073808088 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073817767 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.par2
-rw------- 1 markgrandi staff 107745376 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073811095 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073788706 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.par2
-rw------- 1 markgrandi staff 107742176 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.vol000+200.par2

~Mark

Kenneth Loafman

2016-02-25 16:37:40 UTC

Permalink

If you are backing up the system directories use "sudo".

The errors should not have caused a failure. Try again without the par+
schema, just file://.

Post by Mark Grandi
Running the command below seems to work, the backup completes fine, and
the errors seem to just correspond to permission errors (i am excluding
those directories but i guess duplicity still tries to read inside
them....), but when the backup finishes, it doesn't actually write the
sigtar or manifest files to the backup location, so essentially the backup
fails for all duplicity cares. Running 'collection-status' will delete the
sigtar and manifest files from the archive_dir directory, because they are
not present in the backup location, and 'cleanup' will just list all the
files (the gpg and par2 files) as 'extra files', for the same reason. Why
is duplicity not writing / copying these files to the backup directory?
It seems that it creates the manifest.gpg and sigtar.gpg files and writes
them to the archive_dir, but fails to copy them over to the backup
location. If I copy these files to the backup location manually , then it
suddenly recognizes it as a backup.
duplicity --encrypt-sign-key 0x2CB2C0948891C4830B11AFB6E7B785F6FA5CBE16
--log-file ~/Temp/test_duplicity.log --name "Gypaetus-mgrandi" --use-agent
--exclude-device-files --volsize 1024 --file-prefix "Gypaetus-mgrandi-"
--exclude-filelist /Users/markgrandi/duplicity_exclude.txt /
par2+file:///Volumes/mgrandi_256SSD/backup
Reading globbing filelist /Users/markgrandi/duplicity_exclude.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /.DocumentRevisions-V100
Error accessing possibly locked file /.Spotlight-V100
Error accessing possibly locked file /.Trashes
Error accessing possibly locked file /.file
Error accessing possibly locked file /.fseventsd
Error accessing possibly locked file /Users/Guest/.CFUserTextEncoding
Error accessing possibly locked file /Users/Guest/Desktop
Error accessing possibly locked file /Users/Guest/Documents
Error accessing possibly locked file /Users/Guest/Downloads
Error accessing possibly locked file /Users/Guest/Library
Error accessing possibly locked file /Users/Guest/Movies
Error accessing possibly locked file /Users/Guest/Music
Error accessing possibly locked file /Users/Guest/Pictures
Error accessing possibly locked file /Users/Guest/Public/Drop Box
Error accessing possibly locked file /Users/markgrandi/.viminfo
Error accessing possibly locked file /Users/markgrandi/Library/Saved
Application State/com.adobe.flashplayer.installmanager.savedState
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1456382712.45 (Wed Feb 24 23:45:12 2016)
EndTime 1456386174.70 (Thu Feb 25 00:42:54 2016)
ElapsedTime 3462.25 (57 minutes 42.25 seconds)
SourceFiles 429045
SourceFileSize 22295414551 (20.8 GB)
NewFiles 429045
NewFileSize 22295414551 (20.8 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 429045
RawDeltaSize 22276747699 (20.7 GB)
TotalDestinationSizeChange 13905691762 (13.0 GB)
Errors 16
-------------------------------------------------
Gypaetus:Gypaetus-mgrandi markgrandi$ ls -la
total 1250080
drwxr-xr-x 6 markgrandi staff 204 Feb 25 00:45 .
drwxr-xr-x 5 markgrandi staff 170 Feb 24 23:45 ..
-rw------- 1 markgrandi staff 295425508 Feb 25 00:44
Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gpg
-rw------- 1 markgrandi staff 294998550 Feb 25 00:45
Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gz
-rw------- 1 markgrandi staff 46875395 Feb 25 00:45
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest
-rw------- 1 markgrandi staff 2731077 Feb 25 00:45
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest.gpg
Gypaetus:backup markgrandi$ ls -la
total 29885864
drwxr-xr-x 41 markgrandi staff 1394 Feb 25 00:44 .
drwxrwxr-x 19 markgrandi staff 714 Feb 24 23:42 ..
-rw------- 1 markgrandi staff 1073732207 Feb 24 23:50
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:50
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.par2
-rw------- 1 markgrandi staff 107736576 Feb 24 23:50
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073782430 Feb 25 00:32
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:32
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.par2
-rw------- 1 markgrandi staff 107741376 Feb 25 00:32
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073806871 Feb 25 00:36
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:36
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:36
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073771432 Feb 25 00:40
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:40
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.par2
-rw------- 1 markgrandi staff 107740576 Feb 25 00:40
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1020263590 Feb 25 00:44
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:44
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.par2
-rw------- 1 markgrandi staff 102387776 Feb 25 00:44
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073752418 Feb 24 23:57
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:57
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.par2
-rw------- 1 markgrandi staff 107738976 Feb 24 23:57
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807227 Feb 25 00:03
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:03
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:03
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807149 Feb 25 00:09
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:09
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:09
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073742782 Feb 25 00:12
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:12
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.par2
-rw------- 1 markgrandi staff 107737376 Feb 25 00:12
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073808088 Feb 25 00:20
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:20
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:20
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073817767 Feb 25 00:23
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:23
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.par2
-rw------- 1 markgrandi staff 107745376 Feb 25 00:23
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073811095 Feb 25 00:25
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:25
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:25
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073788706 Feb 25 00:28
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:28
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.par2
-rw------- 1 markgrandi staff 107742176 Feb 25 00:28
Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.vol000+200.par2
~Mark
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Mark Grandi

2016-03-01 04:50:28 UTC

Permalink

I am backing up root, because on Mac OS X, almost all of the applications are stored in /Applications , (shared between all users), so I am backing up root, and excluding all of the other directories besides /Users , but that doesn't seem to stop Duplicity from trying to access them anyway, and failing due to not having the read permissions.

Anyway, I looked into my problem myself since I had a free afternoon, and found myself honestly quite concerned with what some of the code does, or rather doesn't do. For instance, with Par2Backend, if for some reason the par2 command doesn't return 0, then it just silently doesn't create any par2 files without any notice to the user. In LocalBackend, if any of the methods in there that have try/except blocks, in the except blocks, they are either 'pass' or just 'return False', and often the caller doesn't care about the return result anyway, so any errors that occur during moving/copying files are silently discarded and not presented to the user, nor does it invoke any of the retry logic in Duplicity.

I mentioned the two above things, because I had trouble compiling the par2 binary on my Mac, and upon googling, I found a Mac OS X app that is a GUI frontend to par and par2, and it appears that the author created their own fork of par2, that works on mac (and has improvements such as using grand central dispatch, the website is here: https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html <https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html>). Anyway, it sees this version of par2 does not like symlinks, and the Par2Backend class uses symlinks when creating the par2 files. Discovering why the par2 files were not being created required me stepping into the debugger, because no errors are logged to the user.

Once I changed the symlink to a hardlink, and modified the command line being passed to par2 somewhat, I was stuck on the issue that I originally posted: the difftar files + par2 files were being moved to the correct spot, but not the manifest or sigtar files and their par2 counterparts. Stepping into the debugger, i found that the difftar files are being written with Par2Backend wrapping LocalBackend.put(), but the sigtar and manifest files are wrapping LocalBackend._move(). However, LocalBackend._move() is using Path.rename() instead of Path.move() ! It even says in the docstring for Path.move(): "Like rename but destination may be on different file system".

So there lies my problem, my backup destination was on a different file system, but due to LocalBackend._move() calling the wrong underlying Path method, and swallowing exceptions and only returning false (that Par2Backend doesn't even check), it was not moving the sigtar and manifest files + par2 files because of "[Errno 18] Cross-device link".

The patch to fix this is just simply replacing LocalBackend._move() with:

def _move(self, source_path, remote_filename):
target_path = self.remote_pathdir.append(remote_filename)
try:
source_path.setdata() # maybe only needed in Par2Backend?
source_path.move(target_path)
return True
except OSError as e:
return False

This does not solve the concerns I mentioned with swallowing exceptions however.

~Mark

Post by Kenneth Loafman
If you are backing up the system directories use "sudo".
The errors should not have caused a failure. Try again without the par+ schema, just file://.
Running the command below seems to work, the backup completes fine, and the errors seem to just correspond to permission errors (i am excluding those directories but i guess duplicity still tries to read inside them....), but when the backup finishes, it doesn't actually write the sigtar or manifest files to the backup location, so essentially the backup fails for all duplicity cares. Running 'collection-status' will delete the sigtar and manifest files from the archive_dir directory, because they are not present in the backup location, and 'cleanup' will just list all the files (the gpg and par2 files) as 'extra files', for the same reason. Why is duplicity not writing / copying these files to the backup directory?
It seems that it creates the manifest.gpg and sigtar.gpg files and writes them to the archive_dir, but fails to copy them over to the backup location. If I copy these files to the backup location manually , then it suddenly recognizes it as a backup.
duplicity --encrypt-sign-key 0x2CB2C0948891C4830B11AFB6E7B785F6FA5CBE16 --log-file ~/Temp/test_duplicity.log --name "Gypaetus-mgrandi" --use-agent --exclude-device-files --volsize 1024 --file-prefix "Gypaetus-mgrandi-" --exclude-filelist /Users/markgrandi/duplicity_exclude.txt / par2+file:///Volumes/mgrandi_256SSD/backup <>
Reading globbing filelist /Users/markgrandi/duplicity_exclude.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Error accessing possibly locked file /.DocumentRevisions-V100
Error accessing possibly locked file /.Spotlight-V100
Error accessing possibly locked file /.Trashes
Error accessing possibly locked file /.file
Error accessing possibly locked file /.fseventsd
Error accessing possibly locked file /Users/Guest/.CFUserTextEncoding
Error accessing possibly locked file /Users/Guest/Desktop
Error accessing possibly locked file /Users/Guest/Documents
Error accessing possibly locked file /Users/Guest/Downloads
Error accessing possibly locked file /Users/Guest/Library
Error accessing possibly locked file /Users/Guest/Movies
Error accessing possibly locked file /Users/Guest/Music
Error accessing possibly locked file /Users/Guest/Pictures
Error accessing possibly locked file /Users/Guest/Public/Drop Box
Error accessing possibly locked file /Users/markgrandi/.viminfo
Error accessing possibly locked file /Users/markgrandi/Library/Saved Application State/com.adobe.flashplayer.installmanager.savedState
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1456382712.45 (Wed Feb 24 23:45:12 2016)
EndTime 1456386174.70 (Thu Feb 25 00:42:54 2016)
ElapsedTime 3462.25 (57 minutes 42.25 seconds)
SourceFiles 429045
SourceFileSize 22295414551 (20.8 GB)
NewFiles 429045
NewFileSize 22295414551 (20.8 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 429045
RawDeltaSize 22276747699 (20.7 GB)
TotalDestinationSizeChange 13905691762 (13.0 GB)
Errors 16
-------------------------------------------------
Gypaetus:Gypaetus-mgrandi markgrandi$ ls -la
total 1250080
drwxr-xr-x 6 markgrandi staff 204 Feb 25 00:45 .
drwxr-xr-x 5 markgrandi staff 170 Feb 24 23:45 ..
-rw------- 1 markgrandi staff 295425508 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gpg
-rw------- 1 markgrandi staff 294998550 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full-signatures.20160225T064512Z.sigtar.gz
-rw------- 1 markgrandi staff 46875395 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest
-rw------- 1 markgrandi staff 2731077 Feb 25 00:45 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.manifest.gpg
Gypaetus:backup markgrandi$ ls -la
total 29885864
drwxr-xr-x 41 markgrandi staff 1394 Feb 25 00:44 .
drwxrwxr-x 19 markgrandi staff 714 Feb 24 23:42 ..
-rw------- 1 markgrandi staff 1073732207 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.par2
-rw------- 1 markgrandi staff 107736576 Feb 24 23:50 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol1.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073782430 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.par2
-rw------- 1 markgrandi staff 107741376 Feb 25 00:32 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol10.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073806871 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:36 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol11.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073771432 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.par2
-rw------- 1 markgrandi staff 107740576 Feb 25 00:40 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol12.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1020263590 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.par2
-rw------- 1 markgrandi staff 102387776 Feb 25 00:44 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol13.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073752418 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.par2
-rw------- 1 markgrandi staff 107738976 Feb 24 23:57 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol2.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807227 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:03 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol3.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073807149 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.par2
-rw------- 1 markgrandi staff 107743776 Feb 25 00:09 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol4.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073742782 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.par2
-rw------- 1 markgrandi staff 107737376 Feb 25 00:12 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol5.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073808088 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:20 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol6.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073817767 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.par2
-rw------- 1 markgrandi staff 107745376 Feb 25 00:23 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol7.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073811095 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.par2
-rw------- 1 markgrandi staff 107744576 Feb 25 00:25 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol8.difftar.gpg.vol000+200.par2
-rw------- 1 markgrandi staff 1073788706 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg
-rw------- 1 markgrandi staff 40456 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.par2
-rw------- 1 markgrandi staff 107742176 Feb 25 00:28 Gypaetus-mgrandi-duplicity-full.20160225T064512Z.vol9.difftar.gpg.vol000+200.par2
~Mark
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

e***@web.de

2016-03-01 12:04:05 UTC

Permalink

Post by Mark Grandi
I am backing up root, because on Mac OS X, almost all of the applications are stored in /Applications , (shared between all users), so I am backing up root, and excluding all of the other directories besides /Users , but that doesn't seem to stop Duplicity from trying to access them anyway, and failing due to not having the read permissions.
Anyway, I looked into my problem myself since I had a free afternoon, and found myself honestly quite concerned with what some of the code does, or rather doesn't do. For instance, with Par2Backend, if for some reason the par2 command doesn't return 0, then it just silently doesn't create any par2 files without any notice to the user. In LocalBackend, if any of the methods in there that have try/except blocks, in the except blocks, they are either 'pass' or just 'return False', and often the caller doesn't care about the return result anyway, so any errors that occur during moving/copying files are silently discarded and not presented to the user, nor does it invoke any of the retry logic in Duplicity.
I mentioned the two above things, because I had trouble compiling the par2 binary on my Mac, and upon googling, I found a Mac OS X app that is a GUI frontend to par and par2, and it appears that the author created their own fork of par2, that works on mac (and has improvements such as using grand central dispatch, the website is here: https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html). Anyway, it sees this version of par2 does not like symlinks, and the Par2Backend class uses symlinks when creating the par2 files. Discovering why the par2 files were not being created required me stepping into the debugger, because no errors are logged to the user.
Once I changed the symlink to a hardlink, and modified the command line being passed to par2 somewhat, I was stuck on the issue that I originally posted: the difftar files + par2 files were being moved to the correct spot, but not the manifest or sigtar files and their par2 counterparts. Stepping into the debugger, i found that the difftar files are being written with Par2Backend wrapping LocalBackend.put(), but the sigtar and manifest files are wrapping LocalBackend._move(). However, LocalBackend._move() is using Path.rename() instead of Path.move() ! It even says in the docstring for Path.move(): "Like rename but destination may be on different file system".
So there lies my problem, my backup destination was on a different file system, but due to LocalBackend._move() calling the wrong underlying Path method, and swallowing exceptions and only returning false (that Par2Backend doesn't even check), it was not moving the sigtar and manifest files + par2 files because of "[Errno 18] Cross-device link".
target_path = self.remote_pathdir.append(remote_filename)
source_path.setdata() # maybe only needed in Par2Backend?
source_path.move(target_path)
return True
return False
This does not solve the concerns I mentioned with swallowing exceptions however.

hey Mark,

well done debugging the issue! i totally agree w/ you that the duplicity code is not in perfect shape. thanks for investing the time to find and fix your issue.

wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root users, would you mind to have a look if you can come up w/ a solution that works w/o links at all for the par2 backend? if you provide a patch i will create a branch that fixes your issues that can be merged into the next release by Ken.

..ede

Mark Grandi

2016-03-01 15:48:24 UTC

Permalink

Post by e***@web.de

hey Mark,
well done debugging the issue! i totally agree w/ you that the duplicity code is not in perfect shape. thanks for investing the time to find and fix your issue.
wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root users, would you mind to have a look if you can come up w/ a solution that works w/o links at all for the par2 backend? if you provide a patch i will create a branch that fixes your issues that can be merged into the next release by Ken.
..ede
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>

Well, the only reason I changed os.symlink to os.link was because i'm not using the 'official' par2 I guess you could say, but par2SL? Either way, I think it should be a separate backend, or the par2 backend should handle both versions of par2.

Either way, I'm also confused by the statement that hardlinks are only available to root users? This certainly isn't the case for Mac OS X as well as linux (at least the version of Ubuntu that I run on my server). Perhaps you are thinking of windows and NTFS junction points, which require some policy enabled for that user (which is the bane of every VCS software that supports symlinks).

HOWEVER, I do agree that it is probably easiest to just refactor it to not need a symlink or a hardlink by just doing some renaming magic, so i'll work on that. I use bazaar myself so i'll create a branch on launchpad for both this (and the other issue that i posted in a different thread on this mailing list)

~Mark

e***@web.de

2016-03-01 16:13:10 UTC

Permalink

Post by Mark Grandi

Post by e***@web.de

hey Mark,
well done debugging the issue! i totally agree w/ you that the duplicity code is not in perfect shape. thanks for investing the time to find and fix your issue.
wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root users, would you mind to have a look if you can come up w/ a solution that works w/o links at all for the par2 backend? if you provide a patch i will create a branch that fixes your issues that can be merged into the next release by Ken.
..ede
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

w/o looking into the par2 backend. why is it actually necessary to create links at all?

Post by Mark Grandi
Either way, I'm also confused by the statement that hardlinks are only available to root users? This certainly isn't the case for Mac OS X as well as linux (at least the version of Ubuntu that I run on my server). Perhaps you are thinking of windows and NTFS junction points, which require some policy enabled for that user (which is the bane of every VCS software that supports symlinks).

just using hard links to sparsely (pun intended).. with linux you have to be root to hardlink folders, but files work fine - you are correct.

Post by Mark Grandi
HOWEVER, I do agree that it is probably easiest to just refactor it to not need a symlink or a hardlink by just doing some renaming magic, so i'll work on that. I use bazaar myself so i'll create a branch on launchpad for both this (and the other issue that i posted in a different thread on this mailing list)

just looked a bit further into the move issue and path.py:Path.move(...) seems to copy/del all of the time. wouldn't it make sense performancewise to add os.rename() here and only copy/del if the rename failed?
i guess the author of the code you fixed had the performance advantage of rename in mind when he wrote that code.

wish we had more contributors like you!!.. thanks in advance.. ede

Mark Grandi

2016-03-01 18:40:49 UTC

Permalink

Post by e***@web.de

Post by Mark Grandi

Post by e***@web.de

hey Mark,
well done debugging the issue! i totally agree w/ you that the duplicity code is not in perfect shape. thanks for investing the time to find and fix your issue.
wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root users, would you mind to have a look if you can come up w/ a solution that works w/o links at all for the par2 backend? if you provide a patch i will create a branch that fixes your issues that can be merged into the next release by Ken.
..ede
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

w/o looking into the par2 backend. why is it actually necessary to create links at all?

par2 files have the filename of the file that they contain the recovery information for, so from the comments from Par2Backend, it seems that the name could possibly be different (possibly because the backend that Par2Backend is wrapping has some special naming scheme), so it creates a symlink so that it will create the par2 file for the correct filename, and then after that it can be renamed/moved to where the wrapped backend wants.

Post by e***@web.de

just using hard links to sparsely (pun intended).. with linux you have to be root to hardlink folders, but files work fine - you are correct.

I could bzr blame the code and see if there are any insights, but yeah, a lot of the path code seems to be very buggy and uses a lot of assumptions. For example, the new bug that I found is that when moving the file, it assumes it can change the ownership of the files after moving them, but that is returning a '[Errno 1] Operation not permitted:' error , possibly because I am mounting a drive through AFP, and AFP is doing funky stuff, since while its mounted, its not necessarily a real physical drive and seems to not support all the normal FS operations...*scratches head*

Post by e***@web.de
wish we had more contributors like you!!.. thanks in advance.. ede
_______________________________________________
Duplicity-talk mailing list
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Edgar Soldin

2016-03-27 12:56:27 UTC

Permalink

hey Mark,

any progress on the matter? looks like another happy user stumbled over the same issue.
https://answers.launchpad.net/duplicity/+question/288636

..ede/duply.net