Is there a way to speed this up?

sudo rsync -aogW --partial --progress --exclude-from=/etc/restic.excludes  -e 'ssh -p 22' / root@10.0.0.122:/Volumes/TD/rsync/beast_2023_11_07_1106

On the Debian side, rsync 3.2.7
On the iMac Pro side, rsync 3.2.7
Wired ethernet, 10GbE, send speed (in Mission Control) varies around 250MB/s to 500MB/s.
There are about 6.7M files to transfer. After ~42 hours, 1.1M have transferred.
This is what the output of the rsync command looks like:

tank/data/sleep_08-30/tank/data/sleep/files/md5/73/0905320c84d53c108436297fbc5f48
         50,429 100%   53.76kB/s    0:00:00 (xfr#82163, to-chk=4655176/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/090a02e70e1eff0b4bd613026fe67c
         51,262 100%   54.35kB/s    0:00:00 (xfr#82164, to-chk=4655175/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/093b2096cb530739c6f62354f1ce37

Restic, on the other hand, is wicked fast backing up to the same external drive hanging off the iMac (even while rsync is simultaneously running). I’m using rsync because I would like to have a file-by-file backup, as this drive is being backed up to Backblaze, and without a file-by-file backup, I would be faced with restoring the whole restic backup (8TB) if I lost my external drive and needed to recover files.
Thanks for any suggestions!

  • bloopernova@programming.devEnglish
    1·
    10 months ago

    Rsync over ssh needs a specific encryption parameter so it’s not wasting CPU cycles on compression and encryption.

    Better would be to use the rsync server listening on the source, and pull to the target.

    Sorry not able to write the exact commands, work is busy right now. Good luck, hope you can find a solution!

  • aqjo@alien.topOPBEnglish
    1·
    10 months ago

    zfssend might be an option too, but setting up zfs on macOS could be a challenge, and I’m not sure if backblaze will back up a drive with that file system. Also don’t know how reliable zfs on macOS is.

  • EddieJWinkler@alien.topBEnglish
    1·
    10 months ago

    --size-only or other checksum options might help.

    Back in the day, we used to make a cpio archive on the fly, pipe it into ssh and un-cpio it at the other end, because that was faster than rsync for many files. Not sure why.