[Duplicity-talk] Over Uploading and Status Bar Not Making Sense

[Andrew Claus]

Hello All,

Cygwin, Duplicity 0.7.07.1, Python 2.7, GnuPG 1.4.20, boto3

I am trying to backup 1.7 GB to S3, but Duplicity seems to be running in a
loop and keeping on uploading more than is necessary, but not of the right
volumes. Right now the progress shows the below, and has been similar for a
while. The temp directory only has files ending in 1,3,4. S3 only shows
volume 1 uploaded.

4.0GB 00:47:00 [1.4MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-1: Uploaded 13508608/26214400 bytes

4.0GB 00:47:03 [990.6KB/s] [=======================================> ] 99%
ETA < 30sec

Thread-2: Uploaded 18104320/26214400 bytes

Thread-4: Uploaded 15597568/26214400 bytes

Thread-3: Uploaded 14065664/26214400 bytes

4.1GB 00:47:06 [1.1MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-1: Uploaded 13647872/26214400 bytes

Thread-2: Uploaded 18243584/26214400 bytes

4.1GB 00:47:09 [1.2MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-1: Uploaded 13787136/26214400 bytes

Thread-2: Uploaded 18382848/26214400 bytes

Thread-3: Uploaded 14204928/26214400 bytes

4.1GB 00:47:12 [1.3MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-1: Uploaded 13926400/26214400 bytes

Thread-2: Uploaded 18522112/26214400 bytes

4.1GB 00:47:15 [1.3MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-4: Uploaded 15736832/26214400 bytes

Thread-1: Uploaded 14065664/26214400 bytes

Thread-3: Uploaded 14344192/26214400 bytes

Thread-2: Uploaded 18661376/26214400 bytes

4.1GB 00:47:18 [1.4MB/s] [=======================================> ] 99%
ETA < 30sec

Thread-1: Uploaded 14204928/26214400 bytes



My options are as follows:

--timeout=500
--asynchronous-upload
--progress
--s3-use-new-style
--s3-use-multiprocessing
--s3-use-server-side-encryption
--volsize 200
--archive-dir /home/andrew/duplicity_hashes/
-vDEBUG


I tried putting it to a flash drive instead of S3 and could see each of the
volumes being created in the temp directory and then appear in the flash
drive in real time. It showed a correct total transfer amount, but oddly
the status bar stayed at 0% and stalled while files were being transferred,
and then suddenly jumped to 99% and then done.

I appreciate any help.

Thanks,
Andrew

[Andrew Claus]

Follow-up: it finally stopped at 6.9GB! But only 2 volumes were actually
uploaded to S3...

6.9GB 01:13:58 [195.4KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-1: Uploaded 26042368/26214400 bytes
Thread-1: Uploaded 26181632/26214400 bytes
Thread-3: Uploaded 24788992/26214400 bytes
Thread-1: Uploaded 26214400/26214400 bytes
6.9GB 01:14:01 [303.2KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-1: Uploaded chunk 8 at roughly 26083.8271029 bytes/second
Thread-1: Upload of chunk 8 complete
6.9GB 01:14:04 [212.2KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-3: Uploaded 24928256/26214400 bytes
Thread-3: Uploaded 25067520/26214400 bytes
Thread-3: Uploaded 25206784/26214400 bytes
6.9GB 01:14:07 [175.7KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-3: Uploaded 25346048/26214400 bytes
Thread-3: Uploaded 25485312/26214400 bytes
6.9GB 01:14:10 [150.2KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-3: Uploaded 25624576/26214400 bytes
Thread-3: Uploaded 25763840/26214400 bytes
6.9GB 01:14:13 [132.3KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-3: Uploaded 25903104/26214400 bytes
Thread-3: Uploaded 26042368/26214400 bytes
6.9GB 01:14:16 [119.8KB/s] [=======================================> ] 99%
ETA < 45sec
6.9GB 01:14:19 [83.8KB/s] [=======================================> ] 99%
ETA < 45sec
6.9GB 01:14:22 [0.0B/s] [=======================================> ] 99% ETA
Stalled!
Thread-3: Uploaded 26181632/26214400 bytes
Thread-3: Uploaded 26214400/26214400 bytes
6.9GB 01:14:25 [75.5KB/s] [=======================================> ] 99%
ETA < 45sec
Thread-3: Uploaded chunk 6 at roughly 22478.6619143 bytes/second
Thread-3: Upload of chunk 6 complete
Done waiting for the pool to finish processing
Uploaded s3://
s3.amazonaws.com/PRIVATE/duplicity/SurfacePro4Documents/duplicity-full.20160511T182940Z.vol2.difftar.gpg
to STANDARD Storage at roughly 93857.529269 bytes/second
Deleting /tmp/duplicity-HTVBro-tempdir/mktemp-9a2Ll7-3
Forgetting temporary file /tmp/duplicity-HTVBro-tempdir/mktemp-9a2Ll7-3
AsyncScheduler: task execution done (success: True)
AsyncScheduler: active workers = 0
Releasing lockfile
/home/andrew/duplicity_hashes/SurfacePro4Documents/lockfile.lock
Removing still remembered temporary file
/tmp/duplicity-HTVBro-tempdir/mkstemp-6YdDqP-1
Removing still remembered temporary file
/tmp/duplicity-HTVBro-tempdir/mktemp-fiG4h1-4
INT intercepted...exiting.


Andrew Claus

Post by Andrew Claus
Hello All,
Cygwin, Duplicity 0.7.07.1, Python 2.7, GnuPG 1.4.20, boto3
I am trying to backup 1.7 GB to S3, but Duplicity seems to be running in a
loop and keeping on uploading more than is necessary, but not of the right
volumes. Right now the progress shows the below, and has been similar for a
while. The temp directory only has files ending in 1,3,4. S3 only shows
volume 1 uploaded.
4.0GB 00:47:00 [1.4MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-1: Uploaded 13508608/26214400 bytes
4.0GB 00:47:03 [990.6KB/s] [=======================================> ] 99%
ETA < 30sec
Thread-2: Uploaded 18104320/26214400 bytes
Thread-4: Uploaded 15597568/26214400 bytes
Thread-3: Uploaded 14065664/26214400 bytes
4.1GB 00:47:06 [1.1MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-1: Uploaded 13647872/26214400 bytes
Thread-2: Uploaded 18243584/26214400 bytes
4.1GB 00:47:09 [1.2MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-1: Uploaded 13787136/26214400 bytes
Thread-2: Uploaded 18382848/26214400 bytes
Thread-3: Uploaded 14204928/26214400 bytes
4.1GB 00:47:12 [1.3MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-1: Uploaded 13926400/26214400 bytes
Thread-2: Uploaded 18522112/26214400 bytes
4.1GB 00:47:15 [1.3MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-4: Uploaded 15736832/26214400 bytes
Thread-1: Uploaded 14065664/26214400 bytes
Thread-3: Uploaded 14344192/26214400 bytes
Thread-2: Uploaded 18661376/26214400 bytes
4.1GB 00:47:18 [1.4MB/s] [=======================================> ] 99%
ETA < 30sec
Thread-1: Uploaded 14204928/26214400 bytes
--timeout=500
--asynchronous-upload
--progress
--s3-use-new-style
--s3-use-multiprocessing
--s3-use-server-side-encryption
--volsize 200
--archive-dir /home/andrew/duplicity_hashes/
-vDEBUG
I tried putting it to a flash drive instead of S3 and could see each of
the volumes being created in the temp directory and then appear in the
flash drive in real time. It showed a correct total transfer amount, but
oddly the status bar stayed at 0% and stalled while files were being
transferred, and then suddenly jumped to 99% and then done.
I appreciate any help.
Thanks,
Andrew