The Hitchhiker’s Guide to Compression popped up on Hacker News today. I wish I had time to read it closely.
So I was running a backup with rsync and I saw this:
lib/mysql/ibdata1 437.40M 33% 4.53MB/s 0:03:10 inflate returned -3 (0 bytes) rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.2] rsync: connection unexpectedly closed (155602 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]
The issue seems to be that if you’re using rsync compression and the remote file gets changed while the rsync copy is in progress then shit gets corrupted. My solution was to handle error level ’12’ and retry without compression. If the file changes while the rsync is in progress the file will be corrupt, so you shouldn’t rely on the integrity of such files.
Found a list of MySQL Encryption and Compression Functions today. Handy.