rsync errors

I was having an issue with my rsync command:

time rsync --progress --verbose --acls --xattrs --exclude="/lib/mysql/ibdata1" --exclude="*.tmp" --stats --human-readable --recursive --del --force --times --links --hard-links --executability --numeric-ids --owner --group --perms --sparse --compress-level=6 diligence-test:/usr/ /data/temp/2017-11-15-163925/usr/

getting stuck during “receiving incremental file list” then giving up with:

Timeout, server diligence-test not responding.
rsync: connection unexpectedly closed (16384 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2]
rsync: connection unexpectedly closed (32569 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(235) [generator=3.1.2]
Command exited with non-zero status 255

I could see the server-side process hang in select() with:

root@diligence-test:/home/jj5# ps aux | grep rsync
root      5421  0.0  0.1  15636  2660 ?        Ss   18:55   0:00 rsync --server --sender -vlHogtpAXrSe.iLsfxC --numeric-ids . /usr/

root@diligence-test:/home/jj5# strace -p 5421
strace: Process 5421 attached
select(1, [0], [], [0], {42, 979828})   = 0 (Timeout)

Anyway I figured out how to dodge the problem by nominating –delete-before instead of –del, e.g.:

time rsync --progress --verbose --acls --xattrs --exclude="/lib/mysql/ibdata1" --exclude="*.tmp" --stats --human-readable --recursive --delete-before --force --times --links --hard-links --executability --numeric-ids --owner --group --perms --sparse --compress-level=6 diligence-test:/usr/ /data/temp/2017-11-15-163925/usr/

rsync: unexpected remote arg

So I was getting a baffling “unexpected remote arg” error from rsync today. Eventually I figured out the problem was that my argument “–executability” had become “– executablility”, I think due to a copy and paste problem where I copied some shell script code from Vim in a Konsole terminal into another Vim in another Konsole terminal. Traps for young players! If you get baffled by this error try putting an ‘echo’ in front of the command and then resize your terminal window to see if that affects things…

Error in rsync protocol data stream

So I was running a backup with rsync and I saw this:

lib/mysql/ibdata1
        437.40M  33%    4.53MB/s    0:03:10  
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.2]
rsync: connection unexpectedly closed (155602 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]

The issue seems to be that if you’re using rsync compression and the remote file gets changed while the rsync copy is in progress then shit gets corrupted. My solution was to handle error level ’12’ and retry without compression. If the file changes while the rsync is in progress the file will be corrupt, so you shouldn’t rely on the integrity of such files.