maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Applications (https://talk.maemo.org/forumdisplay.php?f=41)
-   -   cloudsync.sh, replacement to DropBox and whatever you have (https://talk.maemo.org/showthread.php?t=89875)

juiceme 2013-04-23 08:49

cloudsync.sh, replacement to DropBox and whatever you have
 
Well, lately there were some threads about problems with Dropbox clients etc, and this makes me wonder...

I myself question the need to have an application and some external service just to sync my stuff. It seems a overkill, why do you need to do things the hard way when same (or better) functionality can be achieved with a simple shell script?

I have been syncing my data up&down with this kind of setup. This is easy to implement, portable, robust and beautiful.

1. You need to install few niceties first. This script uses rsync with a crontab trigger, so get those from rzr's Harmattan repository.
(I don't know about Frematle repositories but I belive cron and rsync are there too...)

2. You need to create a pair of rsa keys and install the public key to the cloudservers users authorized_keys, just the standard stuff.
I recommend creating a dummy user just for the sync purposes...

3. Here is the cloudsync.sh script, put it to your "/root" for example:
Code:

#!/bin/sh

G_LOGTAG="cloudsync"
G_LOCKFILE="/var/lock/cloudsync.pid"
G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"
G_UPLOAD_DIRS="/home/user/MyDocs/DCIM /home/user/MyDocs/.backups /home/user/MyDocs/AptBackup /home/user/MyDocs/meeTrainer_workouts /home/user/MyDocs/.CallRecorder /home/user/MyDocs/Pictures"

if [ -e "$G_LOCKFILE" ]; then
  logger -t "$G_LOGTAG" "cannot get lock $G_LOCKFILE, exiting"
  exit 1
fi

echo "$$" > "$G_LOCKFILE"

## get the wlan state to see if we're in home network
aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")
if [ "$?" -eq "0" ]; then
  G_CLOUDHOST=$G_LOCALNAME
  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"
else
  G_CLOUDHOST=$G_DNSNAME
  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"
fi

G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIRS $G_USERNAME@$G_CLOUDHOST:)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then
  logger -t "$G_LOGTAG" "Upload tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"
fi

G_SYNC=$(/usr/bin/rsync -rptv $G_USERNAME@$G_CLOUDHOST:upload/ /home/user/MyDocs/Downloads/)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then               
  logger -t "$G_LOGTAG" "Download tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Download transfer failed (error code $G_RET)"
fi

rm -f "$G_LOCKFILE"

return 0

4. Edit the variables in the script to your liking:
Code:

G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"

5. put something like this in your crontab:
Code:

~ #
~ # crontab -l

*/30  * * * * /root/cloudsync.sh > /dev/null 2>&1

~ #



DESCRIPTION:

So, for the people who bothered to read to the end of this posting :D

This script can be modified to sync any desired directories. This example syncs my:
  • DCIM
  • .backups
  • AptBackup
  • meeTrainer_workouts
  • .CallRecorder
  • Pictures
folders, but that's just my preferences. Anything goes.

The folder "upload" on the cloudserver is synced to "Downloads" on the users MyDocs, and same applies, anything goes.

The script detects if you are on home network and uses direct connection to the cloudserver if so. Of course that works this way for me because I have my servers on the same subnet as my WLAN. You might want to modify that setup based on your network configuration...

The crontab entry launches cloudsync every 30 minutes, and additionaly it can be launched at any given moment directly. The script creates a lockfile in /var/lock to prevent double invocation.

Security related thingies;
  • You will need the private rsa key in the roots .ssh/ on the device, which means that if you lose your device the key can be considered compromised. (but of course you have the root password to set to something else than "rootme", don't you?)
  • For this reason I recommend creating a separate user for backup purposes on the cloudserver. Additionally you might want to run that user on chroot jail, or forward the ssh connection to a virtual machine or separate HW.

nokiabot 2013-04-23 17:28

Re: Arch Linux ARM on N900
 
Good job......juicy buddy

jackburton 2013-04-23 17:38

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

www.rzr.online.fr 2013-04-23 18:18

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
I had a such project to do this over webdav ...

juiceme 2013-04-23 18:34

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
Quote:

Originally Posted by jackburton (Post 1338285)
This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Yes, that would easy to do really. The reason I don't have it is that I have just launched the sync up manually if I need something transferred immediately.

On the other hand, if there's something on the cloud end that needs to be synced immeiately to the phone it's a trickier situation. If that's needed then the phone would have to poll the remonte end at some shortish interval which is a sure battery killer.

juiceme 2013-04-23 20:09

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
Quote:

Originally Posted by jackburton (Post 1338285)
Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Well now as I tried it, I am thinking if it's worthwhile to push up an event about file change or if new file event is enough?

I guess normally dropbox reacts when you copy a file to the wtched directory, but what if a file is modified in the directory, should it be uploaded immediately?

qwazix 2013-04-23 21:33

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
Last time I tried to implement my own sync solution with rsync I stumbled upon the inability of rsync to track deleted files in two way syncs. If you pass -delete it deletes missing files for both ends regardless if a file is new on one side. I resorted to using unison, which works like a charm four-way between my windows pc, ubuntu box, N900 and macbook. I suppose that compiling it for N9 should be trivial, if the debian binary doesn't work oob.

juiceme 2013-04-23 21:44

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
Yes, it might get tricky if you want to handle more than just updates. That's the reason I just upsync stuff most of the time, and I have separate directory for things to download to the device from cloud.

Here is a quick&dirty solution for monitoring a directory (/home/user/mydocs/Upload/) and immediately syncing the content to cloud if a new file is dropped there:

Code:

#!/usr/bin/python
import os
import subprocess
import pyinotify

watchmanager = pyinotify.WatchManager()
watchmask =  pyinotify.IN_CREATE

class watchprocess(pyinotify.ProcessEvent):
  def process_IN_CREATE(self, event):
      subprocess.call("/root/upload2cloud.sh", shell=True)

inotifier = pyinotify.Notifier(watchmanager, watchprocess())
ret = watchmanager.add_watch('/home/user/MyDocs/Upload', watchmask, rec=True)

while True:
  try:
      inotifier.process_events()
      if inotifier.check_events():
        inotifier.read_events()
  except KeyboardInterrupt:
      inotifier.stop()
      break

You also need another script that is launched by the python inotify monitor script. I could have reused cloudsync.sh but I decided to strip it down a bit to "/root/upload2cloud.sh", like this:

Code:

G_LOGTAG="cloudsync"
G_LOCKFILE="/var/lock/update2cloud.pid"
G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"
G_UPLOAD_DIR="/home/user/MyDocs/Upload/"

logger -t "$G_LOGTAG" "Triggered upload from filesystem"

if [ -e "$G_LOCKFILE" ]; then
  logger -t "$G_LOGTAG" "Cannot get lock $G_LOCKFILE, exiting"
  exit 1
fi

echo "$$" > "$G_LOCKFILE"

## get the wlan state to see if we're in home network
aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")
if [ "$?" -eq "0" ]; then
  G_CLOUDHOST=$G_LOCALNAME
  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"
else
  G_CLOUDHOST=$G_DNSNAME
  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"
fi

G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIR $G_USERNAME@$G_CLOUDHOST:download)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then
  logger -t "$G_LOGTAG" "Upload tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"
fi

rm -f "$G_LOCKFILE"

return 0


jackburton 2013-04-25 00:58

Re: cloudsync.sh, replacement to DropBox and whatever you have
 
While on topic, would anybody here want to try to evaluate sparkleshare for fremantle and harmattan?

knobtviker 2013-04-27 08:20

Just a question:
When you update an existing file, do you upload/download a whole file or just the part of the file that changed?


All times are GMT. The time now is 14:10.

vBulletin® Version 3.8.8