Is there a way to speed this up?

sudo rsync -aogW --partial --progress --exclude-from=/etc/restic.excludes  -e 'ssh -p 22' / root@10.0.0.122:/Volumes/TD/rsync/beast_2023_11_07_1106

On the Debian side, rsync 3.2.7
On the iMac Pro side, rsync 3.2.7
Wired ethernet, 10GbE, send speed (in Mission Control) varies around 250MB/s to 500MB/s.
There are about 6.7M files to transfer. After ~42 hours, 1.1M have transferred.
This is what the output of the rsync command looks like:

tank/data/sleep_08-30/tank/data/sleep/files/md5/73/0905320c84d53c108436297fbc5f48
         50,429 100%   53.76kB/s    0:00:00 (xfr#82163, to-chk=4655176/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/090a02e70e1eff0b4bd613026fe67c
         51,262 100%   54.35kB/s    0:00:00 (xfr#82164, to-chk=4655175/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/093b2096cb530739c6f62354f1ce37

Restic, on the other hand, is wicked fast backing up to the same external drive hanging off the iMac (even while rsync is simultaneously running). I’m using rsync because I would like to have a file-by-file backup, as this drive is being backed up to Backblaze, and without a file-by-file backup, I would be faced with restoring the whole restic backup (8TB) if I lost my external drive and needed to recover files.
Thanks for any suggestions!

  • EddieJWinkler@alien.topBEnglish
    1·
    1 year ago

    --size-only or other checksum options might help.

    Back in the day, we used to make a cpio archive on the fly, pipe it into ssh and un-cpio it at the other end, because that was faster than rsync for many files. Not sure why.