maemo.org - Talk
Page 1 of 6 1 23 Last »
Show 40 post(s) from this thread on one page

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Nokia N9 / N950 (https://talk.maemo.org/forumdisplay.php?f=51)
-   -   A command-line tool for SMS backup and restore (https://talk.maemo.org/showthread.php?t=83166)

wwwjfy 2012-03-21 16:34
A command-line tool for SMS backup and restore
 
Hi, I've just developed a small command-line utility to backup and restore SMS on N9. Also the one for call log.

It can be downloaded here:
SMSBackupRestore
CallBackupRestore

Here is a GUI version forked, JFYI: https://github.com/harmattan/MeegoSMSBackupRestore

If you're interested in building from source, a little bit instruction here.

Use at your own risk.

Donation appreciated to Paypal account: wwwjfy@gmail.com

thedead1440 2012-03-21 17:47
Re: A command-line tool for SMS backup and restore
 
hi can some instructions on its usage be posted to make things easier?

wolke 2012-03-21 19:38
Re: A command-line tool for SMS backup and restore
 
it works beautifully {7000+ texts backed up, merged with another few hundred, tweaked, and restored}.

one fatal flaw: no donation box!!
how do i give you all my money, wwwjfy?

ciaps 2012-03-21 20:10
Re: A command-line tool for SMS backup and restore
 
I have an old nokia e70; i would like to copy the sms from there to my nokia n9, since nokia e70 works with nokia suite, would it possible using your tool to import the e70 .csv exported with the suite in the n9 ?

BlinkThinks 2012-03-21 20:26
Re: A command-line tool for SMS backup and restore
 
Is there the same tool to back-up and restore Notes for the N9?

Thanks!

wolke 2012-03-21 21:24
Re: A command-line tool for SMS backup and restore
 
import fails if you do 'delete all messages' first. it works again if you send a message.
error is:

About to commit the changes
virtual bool CommHistory::EventModelPrivate::doAddEvent(CommHis tory::Event&) Group id not set
Error adding events

wwwjfy 2012-03-21 23:54
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by thedead1440 (Post 1182452)
hi can some instructions on its usage be posted to make things easier?
It's on https://github.com/wwwjfy/MeegoSMSBackupRestore, readme part

wwwjfy 2012-03-22 01:27
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by ciaps (Post 1182500)
I have an old nokia e70; i would like to copy the sms from there to my nokia n9, since nokia e70 works with nokia suite, would it possible using your tool to import the e70 .csv exported with the suite in the n9 ?
I have no idea how e70 .csv looks like. I can do a python or lua script if you send me a sample

wwwjfy 2012-03-22 02:22
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wolke (Post 1182520)
import fails if you do 'delete all messages' first. it works again if you send a message.
error is:

About to commit the changes
virtual bool CommHistory::EventModelPrivate::doAddEvent(CommHis tory::Event&) Group id not set
Error adding events
Interesting as I tried both ways, delete all messages first or not.

Though maybe not related, could you paste a sample csv file here?

if you can, please comment there, https://github.com/wwwjfy/MeegoSMSBa...store/issues/1, so I can track and receive notifications (no idea how to set it here)

wolke 2012-03-22 15:57
Re: A command-line tool for SMS backup and restore
 
thread tools => subscribe to this thread => [instant notification by email]

if youre already subscribed to daily/weekly:
thread tools => unsubscribe to this thread

REALLY THOUGH: where is the donation box?
get a flattr or paypal or something and put it here.

wwwjfy 2012-03-22 16:41
Re: A command-line tool for SMS backup and restore
 
Thanks for the info, I've already subscribed to the thread.

I never thought of accepting donation, maybe it's the beginning.
I've put it on the first post. :)

ciaps 2012-03-23 11:38
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wwwjfy (Post 1182561)
I have no idea how e70 .csv looks like. I can do a python or lua script if you send me a sample
I can't send you a private; could you please me provide me your email so i can send you a sample ? Thanks in advance!

wwwjfy 2012-03-23 11:40
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by ciaps (Post 1183157)
I can't send you a private; could you please me provide me your email so i can send you a sample ? Thanks in advance!
sure, wwwjfy@gmail.com

ciaps 2012-03-23 12:01
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wwwjfy (Post 1183158)
Thanks man! i will send you a sample this evening (1pm here) since i dont have my old phone here with me!

wwwjfy 2012-03-25 03:25
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by ciaps (Post 1183171)
Thanks man! i will send you a sample this evening (1pm here) since i dont have my old phone here with me!
just let you know in case you've sent that I haven't received anything about that, including the spam

wolke 2012-04-01 03:29
Re: A command-line tool for SMS backup and restore
 
ok, so ive been using this for a week or two and its beautiful.

every 12 hours, if my phone and computer are on the same network, my SMSes are automatically backed up, de-duplicated, sorted into files by the contact, and committed incrementally to a private git repository.

on the restore end, i run a quick script and i get messages received in the last 30 days {or the last 5 messages per contact if there are no new ones} in a neat little file for importing.

this is everything ive ever wanted, ever, for SMS backup and restore. thanks a ton.

wwwjfy 2012-04-01 04:03
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wolke (Post 1186667)
ok, so ive been using this for a week or two and its beautiful.

every 12 hours, if my phone and computer are on the same network, my SMSes are automatically backed up, de-duplicated, sorted into files by the contact, and committed incrementally to a private git repository.

on the restore end, i run a quick script and i get messages received in the last 30 days {or the last 5 messages per contact if there are no new ones} in a neat little file for importing.

this is everything ive ever wanted, ever, for SMS backup and restore. thanks a ton.
Thank you for you patches and donation. :)

jonchew 2012-04-05 05:04
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wwwjfy (Post 1182418)
Hi, I've just developed a small command-line utility to backup and restore SMS on N9.

It can be downloaded here.

If you're interested in building from source, a little bit instruction here.

Use at your own risk.

Donation appreciated to Paypal account: wwwjfy@gmail.com
where does the app saves the back up file? i have already back up my sms but failed to remember the file name i used.

wolke 2012-04-05 05:08
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by jonchew (Post 1188235)
where does the app saves the back up file? i have already back up my sms but failed to remember the file name i used.
it saves it in the directory you ran it. chances are good that that was /, /home/user, or /root.

jonchew 2012-04-05 05:53
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wolke (Post 1188238)
it saves it in the directory you ran it. chances are good that that was /, /home/user, or /root.
ok, let me check.

jonchew 2012-04-05 08:10
Re: A command-line tool for SMS backup and restore
 
help guys! i want to add another sms back up but when i type the command smsbackuprestore import [filename] it processes nothing. here is the command i used

~# smsbackuprestore import meback
~# smsbackuprestore import meback

wwwjfy 2012-04-05 08:20
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by jonchew (Post 1188282)
help guys! i want to add another sms back up but when i type the command smsbackuprestore import [filename] it processes nothing. here is the command i used

~# smsbackuprestore import meback
~# smsbackuprestore import meback
either the file is empty, or the file doesn't exist.

please try to use the absolute path for "meback"

jonchew 2012-04-05 08:41
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wwwjfy (Post 1188287)
either the file is empty, or the file doesn't exist.

please try to use the absolute path for "meback"
hmm,should the file (meback) exist before I import? Should the app create the file for me or I should create it?

wwwjfy 2012-04-05 08:43
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by jonchew (Post 1188295)
hmm,should the file (meback) exist before I import? Should the app create the file for me or I should create it?
"import" is to import the content in a file to your device's SMS, so you have to have something to import, with correct format.

or actually you want to "export"? then the file doesn't need to exist :)

Pirvokkula 2012-04-11 20:27
Re: A command-line tool for SMS backup and restore
 
Thank you very much. This worked for me.

thedead1440 2012-04-11 20:44
Re: A command-line tool for SMS backup and restore
 
hi how can we merge two or more files from the backup.

during the backup i rec'd the error "the background thread has stalled" due to which the newer backups are smaller file sizes than the first backup. the new backups contain new smses and less than 1/3 of the old smses. so i want to merge these files and import them back.

thanks for your help

wwwjfy 2012-04-12 01:58
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by thedead1440 (Post 1190929)
hi how can we merge two or more files from the backup.

during the backup i rec'd the error "the background thread has stalled" due to which the newer backups are smaller file sizes than the first backup. the new backups contain new smses and less than 1/3 of the old smses. so i want to merge these files and import them back.

thanks for your help
If no multiline sms, you can easily run in terminal

cat 1st 2nd | sort -u -k3 -t, > new

to merge two files. (-u is for unique, -k3 is for sorting by date, -t, is for separator ,)

with multiline sms, it's much easier if you copy the files to your windows/linux/mac to process. I can provide a python script if you need

skyjumper 2012-04-12 06:49
Re: A command-line tool for SMS backup and restore
 
Thanks for the great app!!

As an earlier poster asked, is there any chance of doing the same for Notes? Is it a similar process or too involved?I would do it if I could!!

Also, so I can wipe my phone and have confidence that I can restore all my sms's as new?

thedead1440 2012-04-12 08:44
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wwwjfy (Post 1190991)
If no multiline sms, you can easily run in terminal

cat 1st 2nd | sort -u -k3 -t, > new

to merge two files. (-u is for unique, -k3 is for sorting by date, -t, is for separator ,)

with multiline sms, it's much easier if you copy the files to your windows/linux/mac to process. I can provide a python script if you need
can you please provide a script to do the same on windows. my smses are multiline.

thanks for your help.

wwwjfy 2012-04-12 14:27
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by skyjumper (Post 1191062)
Thanks for the great app!!

As an earlier poster asked, is there any chance of doing the same for Notes? Is it a similar process or too involved?I would do it if I could!!

Also, so I can wipe my phone and have confidence that I can restore all my sms's as new?
I just found that. http://talk.maemo.org/showthread.php?t=81312&page=3
check the bottom, there is already an app for that

wolke 2012-04-12 14:53
Re: A command-line tool for SMS backup and restore
 
here is my util for splitting and joining the smsbackuprestore dumps.

it looks at all your exports, automatically removes dupes, splits the texts by phone number into separate files, and sorts by date sent/received.

for restoring, it joins them together taking only texts from the last 30 days from the backup.


note: for non-united-states folks, you might wanna remove this line below before using.
@messages = removeUSCountryCode @messages;


i run it on the phone, like this:

#exports to ~/MyDocs/sms-backup/#######.sms
perl n9-sms-tool.pl backup

#reads all .sms files in ~/MyDocs/sms-backup,
#removes duplicates, and splits them out by phone number,
#into separate files in ~/MyDocs/sms-backup-repo
perl n9-sms-tool.pl split

#unnecessary step: commits a git repo {local only, no pushing}
perl n9-sms-tool.pl commit

#makes a file for importing, using texts only from the last 30 days
perl n9-sms-tool.pl join


Code:
#!/usr/bin/perl
#n9-sms-tool v0.1
#Copyright 2012 Elliot Wolk
#This program is free software: you can redistribute it and/or modify
#it under the terms of the GNU General Public License as published by
#the Free Software Foundation, either version 3 of the License, or
#(at your option) any later version.
#
#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#See the GNU General Public License for more details.
#
#You should have received a copy of the GNU General Public License
#along with this program.
#If not, see <http://www.gnu.org/licenses/>.
use strict;
use warnings;

my $cutoffDate = "30 days ago";

my $smsDir = "/home/user/MyDocs/sms-backup";
my $repoDir = "/home/user/MyDocs/sms-backup-repo";

sub filterMessages(\@);
sub getMessagesFromDir($);
sub getMessages($);
sub messageToString($);
sub writeMessageFile(\@$);
sub writeContactsFiles(\@$);
sub removeUSCountryCode(\@);
sub removeDupes(\@);

sub main(@){
  my $arg = shift;
  $arg = '' if not defined $arg;
  my @okArgs = qw(split join commit backup);
  my $ok = join "|", @okArgs;
  die "Usage: $0 $ok\n" if $arg !~ /^($ok)$/ or @_ > 0;

  if($arg eq 'split'){
    system "mkdir $repoDir -p";
    my @messages = (getMessagesFromDir($smsDir), getMessagesFromDir($repoDir));
    @messages = removeDupes @messages;
    system "mkdir -p $repoDir";
    system "rm $repoDir/*.sms";
    writeContactsFiles @messages, $repoDir;
  }elsif($arg eq 'backup'){
    system "mkdir $smsDir -p";
    chdir $smsDir;
    system "smsbackuprestore", "export", time . ".sms";
  }elsif($arg eq 'join'){
    my @messages = getMessagesFromDir $repoDir;
    @messages = removeDupes @messages;
    @messages = filterMessages @messages;
    writeMessageFile @messages, "$smsDir/filtered.sms";
  }elsif($arg eq 'commit'){
    chdir $repoDir;
    if(not -d '.git'){
      system "git init";
    }
    system "git add *.sms";
    system "git --no-pager diff --cached";
    system "git commit -m 'automatic commit'";
  }
}

sub filterMessages(\@){
  my $targetDate = `date --date="$cutoffDate" '+%Y-%m-%d %H:%M:%S'`;
  chomp $targetDate;

  my @messages = @{shift()};
  my @newMessages;
  for my $msg(@messages){
    my $date = $$msg[2];
    if($date gt $targetDate){
      push @newMessages, $msg;
    }
  }
  return @newMessages;
}

sub getMessagesFromDir($){
  my $dir = shift;
  my $content = '';
  for my $file(`ls $dir/*.sms`){
    $content .= `cat $file`;
  }
  return getMessages($content);
}

sub getMessages($){
  my $content = shift;
  my @messages;
  while($content =~ /^([^,]+),([^,]+),([^,]+),("(?:[^"]|"")*")\n/gm){
    my ($phone, $dir, $datetime, $msg) = ($1,$2,$3,$4);
    push @messages, [$1,$2,$3,$4];
  }
  @messages = removeUSCountryCode @messages;
  return @messages;
}

sub messageToString($){
  my ($phone, $dir, $datetime, $msg) = @{$_[0]};
  return "$phone,$dir,$datetime,$msg\n";
}

sub writeMessageFile(\@$){
  my @messages = @{shift()};
  my $file = shift;
  open FH, "> $file" or die "Could not open $file\n";
  @messages = sort {$$a[2] cmp $$b[2]} @messages;
  for my $msg(@messages){
    print FH messageToString $msg;
  }
  close FH;
}

sub writeContactsFiles(\@$){
  my @messages = @{shift()};
  my $dir = shift;

  my %byContact;
  for my $msg(@messages){
    my $phone = $$msg[0];
    $byContact{$phone} = [] if not defined $byContact{$phone};
    push @{$byContact{$phone}}, $msg;
  }

  for my $phone(keys %byContact){
    my $file = "$dir/$phone.sms";
    $file =~ s@:/org/freedesktop/Telepathy/Account/ring/tel/ring@@;
    my @messages = @{$byContact{$phone}};
    writeMessageFile @messages, $file;
  }
}

sub removeUSCountryCode(\@){
  my @messages = @{shift()};
  for my $msg(@messages){
    my $phone = $$msg[0];
    $phone =~ s/^\+?1(\d\d\d\d\d\d\d\d\d\d)$/$1/;
    $$msg[0] = $phone;
  }
  return @messages;
}

sub removeDupes(\@){
  my @messages = @{shift()};
  my %strings;
  my %onelineStrings;
  for my $msg(@messages){
    my $str = messageToString $msg;
   
    my $oneline = $str;
    $oneline =~ s/[\n\r]+//g;
    if(defined $strings{$str}){
      next;
    }elsif(defined $onelineStrings{$oneline}){
      my $prevMsg = $onelineStrings{$oneline};
      my $prevStr = messageToString $prevMsg;

      my $strLines = @{[$str =~ /([\n\r])/g]};
      my $prevStrLines = @{[$prevStr =~ /([\n\r])/g]};
      if($strLines > $prevStrLines){
        #previous one is no good, missing newlines
        delete $strings{$prevStr};
      }elsif($prevStrLines > $strLines){
        #this one is no good, missing newlines
        next;
      }else{
        print "WEIRD: newline count is the same, not skipping\n";
      }
    }
   
    $strings{$str} = $msg;
    $onelineStrings{$oneline} = $msg;
  }
  return values %strings;
}

&main(@ARGV);

wwwjfy 2012-04-12 15:02
Re: A command-line tool for SMS backup and restore
 
check wolke's script.

I just finished mine before I found out that :)
I also paste it here: http://wwwjfy.net/tools/merge_csv.py
It's not as powerful as his. Simple enough, just merge two csv files into one.
I finished it in a hurry, so please check before you delete the original files or restore. :)

you need python installed

wolke 2012-04-12 15:05
Re: A command-line tool for SMS backup and restore
 
sorry, i should have posted this weeks ago. ;)

wolke 2012-04-12 15:15
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by thedead1440 (Post 1190929)
hi how can we merge two or more files from the backup.

during the backup i rec'd the error "the background thread has stalled" due to which the newer backups are smaller file sizes than the first backup. the new backups contain new smses and less than 1/3 of the old smses. so i want to merge these files and import them back.

thanks for your help
the way you use it to do what you want.
1) move all your backups to /home/user/MyDocs/sms-backup
2) add ".sms" to all the backups there
3) run "perl n9-sms-tool.pl split" to populate sms-backup-repo
4) change "30 days ago" in the script to whatever you like.
to do all messages, change it to "jan 1 1970".
5) run "perl n9-sms-tool.pl join" to make a file called "filtered.sms" in sms-backup
6) run smsbackuprestore import filtered.sms

thedead1440 2012-04-13 11:45
Re: A command-line tool for SMS backup and restore
 
thanks it worked a charm.

wolke 2012-04-13 18:28
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by thedead1440 (Post 1191599)
thanks it worked a charm.
glad it worked

Froberg 2012-04-13 20:04
Re: A command-line tool for SMS backup and restore
 
Hi,

I'm attempting to back up, as my N9 is imploding in on itself. I'm trying to preserve the texts I've exchanged with my lady for the past few months, since my last backup.

I used to be able to backup with PC Suite and an NBU, not so any more.

When running your script I got a timeout error, restarted tried again, now I get a huuuuuuuuge error - with something looking like DB values. It starts with:

GDBUS.Error :org.freedesktop.Tracker1.SparqlError.Internal: Operation was cancelled.

After this it just parses a few hundred lines of what looks like a db dump...

Any advice?

Edit; Forgot to mention, I was able to get some backup done previosuly.. only 16.2 kb though.. and not a lot of texts.. but some. I actually got two of that size, several of 0kb when the "timeout" thing happened.

wolke 2012-04-13 20:19
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by Froberg (Post 1191779)
Hi,

I'm attempting to back up, as my N9 is imploding in on itself. I'm trying to preserve the texts I've exchanged with my lady for the past few months, since my last backup.

I used to be able to backup with PC Suite and an NBU, not so any more.

When running your script I got a timeout error, restarted tried again, now I get a huuuuuuuuge error - with something looking like DB values. It starts with:

GDBUS.Error :org.freedesktop.Tracker1.SparqlError.Internal: Operation was cancelled.

After this it just parses a few hundred lines of what looks like a db dump...

Any advice?

Edit; Forgot to mention, I was able to get some backup done previosuly.. only 16.2 kb though.. and not a lot of texts.. but some. I actually got two of that size, several of 0kb when the "timeout" thing happened.
some advice; make a backup of your tracker db
~/.cache/tracker/meta.db and ~/.cache/tracker/meta.db-wal

Froberg 2012-04-13 20:26
Re: A command-line tool for SMS backup and restore
 
Quote:
Originally Posted by wolke (Post 1191789)
some advice; make a backup of your tracker db
~/.cache/tracker/meta.db and ~/.cache/tracker/meta.db-wal
I'm trying cp /home/user/.cache/tracker/meta.db /home/user/MyDocs but I get a permission denied error..

wolke, I see you wrote this post as well: http://talk.maemo.org/showpost.php?p...5&postcount=18

Any chance you can tell me the pain-in-the-*** way of going about it? =)

edit: (added question and fixed link)

edit2: I think I got it.. 1MB file of text messages.. seems like the important stuff is there.. so I'm re-flashing my N9. I had a battery usage at idle at 200+ mA.. so it was fairly critical =)

Thanks for the help anyway, man! :)

thedead1440 2012-04-19 16:50
Re: A command-line tool for SMS backup and restore
 
hi,
i've used the following http://forum.meego.com/showthread.php?t=5725 to backup my smses and call logs direct to my gmail account. could you create something similar which helps not only to backup but to also re-import.

i managed to import my smses from your script above but my call logs are backed up to my gmail account and can't be imported back due to my limited coding knowledge. the publisher isn't interested in creating an import utility.

sorry if it bothers you and thanks in advance.


All times are GMT. The time now is 19:47.
Page 1 of 6 1 23 Last »
Show 40 post(s) from this thread on one page

vBulletin® Version 3.8.8