Admin: Replacing charlesreid1.com:3000 with git.charlesreid1.com

2019-10-09T03:56:21Z

Replacing charlesreid1.com:3000 with git.charlesreid1.com

← Older revision		Revision as of 03:56, 9 October 2019
Line 54:		Line 54:
	where A.B.C.D is the server's IP address.)		where A.B.C.D is the server's IP address.)

	See https://charlesreid1.com~~:3000~~/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf for details		See https://git.charlesreid1.com/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf for details

	==Stunnel Server for Rsync==		==Stunnel Server for Rsync==
Line 81:		Line 81:
	</pre>		</pre>

	See https://charlesreid1.com~~:3000~~/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf		See https://git.charlesreid1.com/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf

	===Running Stunnel Server for Rsync===		===Running Stunnel Server for Rsync===
Line 157:		Line 157:
	</pre>		</pre>

	See stunnel for rsync config file for Raspberry Pi clients: https://charlesreid1.com~~:3000~~/rpi/pi-stunnel/src/master/stunnel.rpiclient.rsync_over_273.conf		See stunnel for rsync config file for Raspberry Pi clients: https://git.charlesreid1.com/rpi/pi-stunnel/src/master/stunnel.rpiclient.rsync_over_273.conf

	==Stunnel Client==		==Stunnel Client==

Admin: /* Running Rsync Server */

2017-04-05T06:18:38Z

Running Rsync Server

← Older revision		Revision as of 06:18, 5 April 2017
Line 30:		Line 30:
	===Running Rsync Server===		===Running Rsync Server===

	~~Run the~~ rsync ~~daemon as~~ a ~~standard system service:~~		Running rsync in a docker container.

	~~<pre>~~		Get the docker file, edit the config file, run the build script, then run the run script: https://git.charlesreid1.com/docker/d-rsync
	~~$ sudo service~~ rsync ~~start~~
	~~</pre>~~

	===Adding Tcp Wrappers for Secure Rsync Protocol===		===Adding Tcp Wrappers for Secure Rsync Protocol===

Admin: /* Configure Rsync Server */

2017-04-05T06:13:31Z

Configure Rsync Server

← Older revision		Revision as of 06:13, 5 April 2017
Line 16:		Line 16:
	===Configure Rsync Server===		===Configure Rsync Server===

	Run an rsync daemon to listen (locally) for rsync requests (these will come from stunnel):		Run an rsync daemon to listen (locally) for rsync requests (these will come from stunnel).

	~~Edit~~ rsync ~~config file at <code>~~/~~etc~~/~~rsyncd~~.~~conf<~~/~~code>.~~		Use the docker rsync image here: https://git.charlesreid1.com/docker/d-rsync

	Set the name of the module. We use "pi". See rsyncd.conf in ~~https://charlesreid1~~.~~com:3000/rpi/pi-transmission~~		Set the name of the module. We use "pi". See rsyncd.conf in above repo.

	Also set the bind address to be the local host only (since stunnel takes care of the external connection):		Also set the bind address to be the local host only (since stunnel takes care of the external connection):

	<pre>		<pre>
	~~address~~ = 127.0.0.1		allow hosts = 127.0.0.1
	</pre>		</pre>

Admin: /* Configure Rsync Server */

2017-04-05T05:47:30Z

Configure Rsync Server

← Older revision		Revision as of 05:47, 5 April 2017
Line 21:		Line 21:

	Set the name of the module. We use "pi". See rsyncd.conf in https://charlesreid1.com:3000/rpi/pi-transmission		Set the name of the module. We use "pi". See rsyncd.conf in https://charlesreid1.com:3000/rpi/pi-transmission

			Also set the bind address to be the local host only (since stunnel takes care of the external connection):

			<pre>
			address = 127.0.0.1
			</pre>

	===Running Rsync Server===		===Running Rsync Server===

Admin: Created page with "This page was immensely helpful: http://www.netbits.us/docs/stunnel_rsync.html This page covers modifications to the instructions on Stunnel/Rsync to run rsync over port..."

2017-04-05T05:31:21Z

Created page with "This page was immensely helpful: http://www.netbits.us/docs/stunnel_rsync.html This page covers modifications to the instructions on Stunnel/Rsync to run rsync over port..."

New page

This page was immensely helpful: http://www.netbits.us/docs/stunnel_rsync.html

This page covers modifications to the instructions on [[Stunnel/Rsync]] to run rsync over port 443. This is useful if you need to run rsync over stunnel, and the firewall only allows HTTP or HTTPS traffic.

=Server=

* Server:
** Adding secure rsync protocol
** Setting up rsync server config (which dir to rsync to)
** Running rsync service
** Setting up stunnel config
** Running stunnel service

==Rsync==

===Configure Rsync Server===

Run an rsync daemon to listen (locally) for rsync requests (these will come from stunnel):

Edit rsync config file at <code>/etc/rsyncd.conf</code>.

Set the name of the module. We use "pi". See rsyncd.conf in https://charlesreid1.com:3000/rpi/pi-transmission

===Running Rsync Server===

Run the rsync daemon as a standard system service:

<pre>
$ sudo service rsync start
</pre>

===Adding Tcp Wrappers for Secure Rsync Protocol===

Rsync is tcp traffic, so define new rsync protocol over port 443.

We do this using <code>/etc/services</code>.

Add the following line to the server's <code>/etc/services</code> (match it exactly):

<pre>
ssyncd 443/tcp # secure rsync over stunnel 443
</pre>

(Optional: may need to add this to <code>/etc/hosts.allow</code> but this messed things up for me:

<pre>
ssyncd : A.B.C.D
</pre>

where A.B.C.D is the server's IP address.)

See https://charlesreid1.com:3000/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf for details

==Stunnel Server for Rsync==

===Configure Stunnel Server for Rsync===

The rsync daemon will run on port 873. The stunnel connection we will set up will be over port 443. The stunnel configuration file will accept connections on 443 and connect them to port 873.

Note these require the rsync service to be defined following the above steps.

Server stunnel.conf file for rsync over port 443:

<pre>
output = /var/log/stunnel4/stunnel.log
cert = /etc/stunnel/stunnel.fullchain.pem
key = /etc/stunnel/stunnel.key.pem
pid = /var/run/stunnel4/stunnel.pid
client = no

debug = 7
foreground = yes

[rsync]
accept = 443
connect = 873
</pre>

See https://charlesreid1.com:3000/docker/d-stunnel/src/master/stunnel.server.rsync_over_443.conf

===Running Stunnel Server for Rsync===

Fire up stunnel with the <code>stunnel</code> command:

<pre>
[remote] $ stunnel
</pre>

Ideally, nothing will happen. Errors may be printed to the screen, or stunnel may fail silently. Check the log to make sure stunnel is running okay. If you set <code>foreground = yes</code> in the config file, the console will display the contents of the log. Otherwise, check the log via:

<pre>
[remote] $ tail -f /var/log/stunnel4/stunnel.log
</pre>

Note that you still won't be able to reach your stunnel server from the outside until you expose the port with iptables.

==Opening Server Firewall to Stunnel==

Need to open firewall to incoming connections on the stunnel rsync port (273):

<pre>
[remote] $ iptables -A INPUT -p tcp --dport 273 -j ACCEPT
[remote] $ iptables -A FORWARD -p tcp -j ACCEPT --dport 273 -m state --state NEW
</pre>

=Client=

This describes the client configuration required to get rsync working over stunnel.

==Rsync Client==

The client's rsync will operate, as usual, on local port 873. Behind the scenes, stunnel will transparently map that port to a remote rsync daemon/server, so it will appear to the rsync client that it is communicating with a local rsync daemon/server.

===Configuring Rsync Client===

No special rsync setup is needed for the client. All we have to do is point the rsync client to the rsync server, and specify one of the modules in our configuration file. The rsync daemon running on the remote server will take care of the rest.

===Running Rsync Client===

The client does not need to run the rsync daemon, only the server does. When you are on the client, you can just run rsync from the command line, and pass it the location of the rsync server (which is localhost port 873). Stunnel takes care of the rest, so the rsync daemon on the remote server appears to be just another local service.

The rsync server should appear to be a local service on port 873. You can check the local rsync port is being mapped correctly to the remote rsync daemon/server by running:

<pre>
[client] $ rsync -vv localhost::
</pre>

This runs rsync and connects to the server at localhost (since no host is specified, it defaults to 873). This is where stunnel is listening and forwarding traffic between the client and server. If the server is listening you should get a simple success message back:

<pre>
[client] $ rsync -vv localhost::
opening tcp connection to localhost port 873
sending daemon args: --server --sender -vvde.Lsfx . / (5 args)
pi Raspberry Pi Wifi Data
</pre>

Here is the rsync script that syncs wifi data between Raspberry Pis and the remote server: https://git.charlesreid1.com/rpi/pi-transmission/src/master/rsync_with_server.sh

The <code>rsync_with_server.sh</code> script wraps the basic functionality of running rsync a single time. This script is ultimately called by <code>check_stunnel.sh</code>, which is a "forever loop" that opens the stunnel connection and runs rsync over stunnel to update the server with the latest wifi data from the Raspberry Pi's wifi sensor. We'll cover that script below.

===Adding Tcp Wrappers for Secure Rsync Protocol===

Add the following to <code>/etc/services</code> on the client:

<pre>
ssync 273/tcp
</pre>

(Optional: you may need to add the following to the client <code>/etc/hosts.allow</code>. This actually caused problems for me.)

<pre>
ssync : LOCAL
</pre>

See stunnel for rsync config file for Raspberry Pi clients: https://charlesreid1.com:3000/rpi/pi-stunnel/src/master/stunnel.rpiclient.rsync_over_273.conf

==Stunnel Client==

===Configuring Stunnel Client for Rsync===

Client stunnel over rsync configuration file goes here.

===Running Stunnel Client for Rsync===

The usual - run stunnel using the stunnel command.

To monitor what is happening use debug = 7 and foreground = yes.

==Forever Loop: Listening for Client Changes==

To listen for changes to the folder containing wifi data on the Raspberry Pi clients, we want to have a loop that runs forever: it should keep the stunnel connection open if it fails, and it should run rsync over stunnel to keep the latest wifi data (CSV files) up-to-date on the remote server.

It should also be resilient to failure - if the stunnel connection fails (if the Raspberry Pi can't find a wifi network to connect to), the program should not crash but should continue running in the background and attempt to connect after a periodic wait.

This script is contained in <code>check_stunnel.sh</code> in the Raspberry Pi pi-transmission repository: https://git.charlesreid1.com/rpi/pi-transmission/src/master/check_stunnel.sh

Here's that relatively modest script:

<pre>
#!/bin/bash
#
# Check if stunnel is running.
# If not, try to start it.
# In any case, try to rsync.
# Then take a nap.

while true; do

echo "$(ps aux | grep [s]tunnel | grep -v check_stunnel)"
if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
echo "Stunnel not running, trying to start it..."
stunnel
sleep 10
if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
echo "No die!"
fi
fi

# If stunnel is not running, this will raise an error and keep going.
# Do rsync here
./rsync_with_server.sh

# wait X seconds, then do it again.
sleep 300;

done
</pre>

=Debugging=

Debugging problems with the two interacting stunnel-rsync layers can get tricky. Here's a good workflow.

==Debugging stunnel==

First, if you want to see what stunnel is actually doing, add the following to the stunnel configuration file:

<pre>
debug = 7
foreground = yes
</pre>

These will run stunnel in the foreground and print out lots of information. You can run this in a terminal window, then open another window and run rsync commands. You should see activity in the stunnel window, indicating it is initiating a connection with the server and passing traffic.

You can do the same thing on the server to monitor the server instance of stunnel, so if you need to troubleshoot a problem on the server side, edit the server stunnel configuration file and add the debug and foreground options.

==Debugging rsync==

If you are confident stunnel is working properly and that the problem is with rsync, you can monitor rsync using the system log. rsync does not log to its own log file.

By running <code>tail -f /var/log/syslog</code> on the server in a window, then running rsync over stunnel commands in another, you should see messages about rsync activity showing up in the syslog. This should also give you more helpful and descriptive information when things go wrong, and help you diagnose the error.

=Flags=

{{StunnelFlag}}

[[Category:Rsync]]

[[Category:Stunnel.conf]]

Stunnel/Rsync 443 - Revision history

Admin: Replacing charlesreid1.com:3000 with git.charlesreid1.com

Admin: /* Running Rsync Server */

Admin: /* Configure Rsync Server */

Admin: /* Configure Rsync Server */

Admin: Created page with "This page was immensely helpful: http://www.netbits.us/docs/stunnel_rsync.html This page covers modifications to the instructions on Stunnel/Rsync to run rsync over port..."