Rsync - Revision history

Admin: /* Errors */

2011-07-20T11:09:06Z

Errors

← Older revision		Revision as of 11:09, 20 July 2011
Line 201:		Line 201:

	I think this is actually a different problem with a different solution. Adding <code>-e ssh</code> fixed it.		I think this is actually a different problem with a different solution. Adding <code>-e ssh</code> fixed it.

			==More Errors==

			<pre>
			sent 448940044 bytes received 818 bytes 305090.63 bytes/sec
			total size is 11987303971 speedup is 26.70
			wiki.charlesmartinreid.com: Operation timed out
			rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
			rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]
			</pre>

			==More "Corrupted MAC on Input" Errors==

			<pre>
			Received disconnect from 155.97.192.16: 2: Corrupted MAC on input.

			rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
			rsync: connection unexpectedly closed (351 bytes received so far) [sender]
			rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
			</pre>

	=Resources=		=Resources=

Admin at 09:53, 20 July 2011

2011-07-20T09:53:17Z

@@ Line 82: / Line 82: @@
 As far as I know, this version works as expected.  Your rsync should be above version 3.
@@ Line 137: / Line 199: @@
 rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]
 </pre>
 =Resources=
 * Latest version of rsync is avilable for download here: http://rsync.samba.org/ftp/rsync/

Admin: /* Corrupted MAC on input */

2011-07-16T22:33:48Z

Corrupted MAC on input

@@ Line 122: / Line 122: @@
 This led me to the conclusion that the underlying issue was that I was using rsync version 2, NOT version 3.  I installed version 3 by hand.  I'm not entirely sure this has solved all my problems, though.
 =Resources=
 * Latest version of rsync is avilable for download here: http://rsync.samba.org/ftp/rsync/

Admin: /* Errors */

2011-07-12T03:11:05Z

Errors

← Older revision		Revision as of 03:11, 12 July 2011
Line 86:		Line 86:

	=Errors=		=Errors=

	==

	==Bus Error/Segmentation Fault==		==Bus Error/Segmentation Fault==
Line 101:		Line 99:
	==Corrupted MAC on input==		==Corrupted MAC on input==

			Once I had tried the above solution, I got a new error, with error text that looks like this:

	~~This problem is likely due to lack of memory [http://blog.e-shell.org/270]~~:

	<pre>		<pre>
Line 112:		Line 108:
	</pre>		</pre>

	Here [http://fixunix.com/tools/520822-memory-usage-rsync-3-0-3-how-much-ram-should-i-have-totransfer-13-million-files.html] someone says you should see it say that it is incrementally updating files		This problem is likely due to lack of memory, based on information here: [http://blog.e-shell.org/270]

			Here [http://fixunix.com/tools/520822-memory-usage-rsync-3-0-3-how-much-ram-should-i-have-totransfer-13-million-files.html] someone says you should see it say that it is incrementally updating files.

	~~but~~ here [http://linux.die.net/man/1/rsync] it says that that only happens with rsync version 3+		But here [http://linux.die.net/man/1/rsync] it says that that only happens with rsync version 3+:

	<pre>		<pre>
Line 123:		Line 121:
	</pre>		</pre>

			This led me to the conclusion that the underlying issue was that I was using rsync version 2, NOT version 3. I installed version 3 by hand. I'm not entirely sure this has solved all my problems, though.

	=Resources=		=Resources=

	* Latest version of rsync is avilable for download here: http://rsync.samba.org/ftp/rsync/		* Latest version of rsync is avilable for download here: http://rsync.samba.org/ftp/rsync/

Admin at 03:09, 12 July 2011

2011-07-12T03:09:00Z

← Older revision		Revision as of 03:09, 12 July 2011
Line 1:		Line 1:
			=Notes=

			I initially tried to use rsync on Mac OS X (Leopard and Snow Leopard both), but ran into some problems trying to rsync a large directory structure with many files (~10 GB) with a Linux server running Ubuntu.

			I ran into some of the errors listed in the [[#Errors]] section below, so hopefully the solutions there will help you if you run into any of those issues. If you are using rsync for the first time and you follow the [[#Installation]] instructions, you should be able to avoid major issues.

			=Installation=

			In order to install a working/healthy rsync, you shouldn't use the rsync that comes with Mac (Leopard or Snow Leopard, both are broken). You should also avoid using the [[Fink]] version of rsync, for the same reason.

			In order to install Rsrync, follow the instructions below.

			==Mac OS X==

			The problem with both Mac OS X rsync executables is that they are all earlier than version 3. There were some major improvements made with rsync version 3, so the version of rsync you use should be ''AT LEAST''' version 3.0.

			The latest version of rsync is available here: http://rsync.samba.org/ftp/rsync/

			===Mac OS X 10.5 (Leopard)===

			First: download the latest version of rsync from http://rsync.samba.org/ftp/rsync/ (called <code>rsync.X.Y.Z.tar.gz</code>)

			Second: download the patches available for rsync (same location, called <code>rsync-patches.X.Y.Z.tar.gz</code>)

			To apply the patches to the source code and install, run these commands:

			<source lang="bash">
			#!/bin/sh

			mkdir /tmp/rsync
			cd /tmp/rsync

			# Download the tarballs
			wget http://rsync.samba.org/ftp/rsync/rsync-3.0.8.tar.gz
			wget http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.8.tar.gz

			# Untar the tarballs
			tar xvzf rsync-3.0.8.tar.gz
			tar xvzf rsync-patches-3.0.8.tar.gz

			# Move to the source code directory location
			mv rsync-3.0.8.tar.gz ~/pkg/_sources/.
			cd ~/pkg/_sources/.

			# Apply patches
			patch -p1 < patches/atimes.diff
			patch -p1 < patches/crtimes-64bit.diff
			patch -p1 < patches/crtimes-hfs.diff
			patch -p1 < patches/crtimes.diff
			patch -p1 < patches/fileflags.diff
			patch -p1 < patches/osx-xattr-nodev.diff

			# Run configure
			./configure \
			--prefix="${HOME}/pkg/rsync/3.0.7" \
			--with-rsh=rsh

			</source>

			===Mac OS X 10.6 (Snow Leopard)===

			Follow instructions given above.

			==Ubuntu Linux==

			If you use [[Aptitude]], you can run <code>apt-get install rsync</code>. This will install the following version of rsync:

			<pre>
			$ rsync --version
			rsync version 3.0.4 protocol version 30
			Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
			Web site: http://rsync.samba.org/
			Capabilities:
			64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
			socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
			append, ACLs, xattrs, iconv, symtimes

			rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
			are welcome to redistribute it under certain conditions. See the GNU
			General Public Licence for details.
			</pre>

			As far as I know, this version works as expected. Your rsync should be above version 3.



			=Errors=

			==

			==Bus Error/Segmentation Fault==

			Initially, when I tried to rsync large directory/tree structures, I ran into a "Bus Error" (equivalent to a "Bus Error/Segmentation Fault" error) before it transferred files, or after it had transferred only a small number of files (between 10 and 30 files).

			This page [http://www.lartmaker.nl/rsync/] says this is caused by a "buffer overrun", meaning it tries to prep the list of files it's going to transfer, but there are too many (or they are too big), and it overfills the buffer.

			The solution that is specified by the same page (http://www.lartmaker.nl/rsync/), is to install a patched version of rsync 2.6.3.

			'''HOWEVER!''' This patched version has several problems. First, it is a very old version of rsync. Second, it fixes one problem, but causes another.

			==Corrupted MAC on input==



	This problem is likely due to lack of memory [http://blog.e-shell.org/270]:		This problem is likely due to lack of memory [http://blog.e-shell.org/270]:

Line 20:		Line 124:


	~~Download latest~~ version of rsync ~~(3.0.8)~~ here: http://rsync.samba.org/ftp/rsync/		=Resources=

			* Latest version of rsync is avilable for download here: http://rsync.samba.org/ftp/rsync/

Admin at 18:48, 11 July 2011

2011-07-11T18:48:41Z

← Older revision		Revision as of 18:48, 11 July 2011
Line 20:		Line 20:


	http://rsync.samba.org/ftp/rsync/		Download latest version of rsync (3.0.8) here: http://rsync.samba.org/ftp/rsync/

Admin at 18:48, 11 July 2011

2011-07-11T18:48:25Z

← Older revision		Revision as of 18:48, 11 July 2011
Line 18:		Line 18:
	incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.		incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.
	</pre>		</pre>


			http://rsync.samba.org/ftp/rsync/

Admin at 18:48, 11 July 2011

2011-07-11T18:48:07Z

← Older revision		Revision as of 18:48, 11 July 2011
Line 1:		Line 1:
	~~<!--~~		This problem is likely due to lack of memory [http://blog.e-shell.org/270]:

	~~Likely~~ due to lack of memory [http://blog.e-shell.org/270]:

	<pre>		<pre>
Line 20:		Line 18:
	incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.		incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.
	</pre>		</pre>

	~~-->~~

Admin at 18:47, 11 July 2011

2011-07-11T18:47:36Z

← Older revision		Revision as of 18:47, 11 July 2011
Line 8:		Line 8:
	rsync: connection unexpectedly closed (8874 bytes received so far) [sender]		rsync: connection unexpectedly closed (8874 bytes received so far) [sender]
	rsync error: unexplained error (code 255) at io.c(453) [sender=2.6.9]		rsync error: unexplained error (code 255) at io.c(453) [sender=2.6.9]
			</pre>

			Here [http://fixunix.com/tools/520822-memory-usage-rsync-3-0-3-how-much-ram-should-i-have-totransfer-13-million-files.html] someone says you should see it say that it is incrementally updating files

			but here [http://linux.die.net/man/1/rsync] it says that that only happens with rsync version 3+

			<pre>
			-r, --recursive
			This tells rsync to copy directories recursively. See also --dirs (-d).
			Beginning with rsync 3.0.0, the recursive algorithm used is now an
			incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.
	</pre>		</pre>

	-->		-->

Admin: Created page with "