Reply
Thread Tools
Posts: 986 | Thanked: 1,526 times | Joined on Jul 2010
#31
here is my util for splitting and joining the smsbackuprestore dumps.

it looks at all your exports, automatically removes dupes, splits the texts by phone number into separate files, and sorts by date sent/received.

for restoring, it joins them together taking only texts from the last 30 days from the backup.


note: for non-united-states folks, you might wanna remove this line below before using.
@messages = removeUSCountryCode @messages;


i run it on the phone, like this:

#exports to ~/MyDocs/sms-backup/#######.sms
perl n9-sms-tool.pl backup

#reads all .sms files in ~/MyDocs/sms-backup,
#removes duplicates, and splits them out by phone number,
#into separate files in ~/MyDocs/sms-backup-repo
perl n9-sms-tool.pl split

#unnecessary step: commits a git repo {local only, no pushing}
perl n9-sms-tool.pl commit

#makes a file for importing, using texts only from the last 30 days
perl n9-sms-tool.pl join


Code:
#!/usr/bin/perl
#n9-sms-tool v0.1
#Copyright 2012 Elliot Wolk
#This program is free software: you can redistribute it and/or modify
#it under the terms of the GNU General Public License as published by
#the Free Software Foundation, either version 3 of the License, or
#(at your option) any later version.
#
#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#See the GNU General Public License for more details.
#
#You should have received a copy of the GNU General Public License
#along with this program.
#If not, see <http://www.gnu.org/licenses/>.
use strict;
use warnings;

my $cutoffDate = "30 days ago";

my $smsDir = "/home/user/MyDocs/sms-backup";
my $repoDir = "/home/user/MyDocs/sms-backup-repo";

sub filterMessages(\@);
sub getMessagesFromDir($);
sub getMessages($);
sub messageToString($);
sub writeMessageFile(\@$);
sub writeContactsFiles(\@$);
sub removeUSCountryCode(\@);
sub removeDupes(\@);

sub main(@){
  my $arg = shift;
  $arg = '' if not defined $arg;
  my @okArgs = qw(split join commit backup);
  my $ok = join "|", @okArgs;
  die "Usage: $0 $ok\n" if $arg !~ /^($ok)$/ or @_ > 0;

  if($arg eq 'split'){
    system "mkdir $repoDir -p";
    my @messages = (getMessagesFromDir($smsDir), getMessagesFromDir($repoDir));
    @messages = removeDupes @messages;
    system "mkdir -p $repoDir";
    system "rm $repoDir/*.sms";
    writeContactsFiles @messages, $repoDir;
  }elsif($arg eq 'backup'){
    system "mkdir $smsDir -p";
    chdir $smsDir;
    system "smsbackuprestore", "export", time . ".sms";
  }elsif($arg eq 'join'){
    my @messages = getMessagesFromDir $repoDir;
    @messages = removeDupes @messages;
    @messages = filterMessages @messages;
    writeMessageFile @messages, "$smsDir/filtered.sms";
  }elsif($arg eq 'commit'){
    chdir $repoDir;
    if(not -d '.git'){
      system "git init";
    }
    system "git add *.sms";
    system "git --no-pager diff --cached";
    system "git commit -m 'automatic commit'";
  }
}

sub filterMessages(\@){
  my $targetDate = `date --date="$cutoffDate" '+%Y-%m-%d %H:%M:%S'`;
  chomp $targetDate;

  my @messages = @{shift()};
  my @newMessages;
  for my $msg(@messages){
    my $date = $$msg[2];
    if($date gt $targetDate){
      push @newMessages, $msg;
    }
  }
  return @newMessages;
}

sub getMessagesFromDir($){
  my $dir = shift;
  my $content = '';
  for my $file(`ls $dir/*.sms`){
    $content .= `cat $file`;
  }
  return getMessages($content);
}

sub getMessages($){
  my $content = shift;
  my @messages;
  while($content =~ /^([^,]+),([^,]+),([^,]+),("(?:[^"]|"")*")\n/gm){
    my ($phone, $dir, $datetime, $msg) = ($1,$2,$3,$4);
    push @messages, [$1,$2,$3,$4];
  }
  @messages = removeUSCountryCode @messages;
  return @messages;
}

sub messageToString($){
  my ($phone, $dir, $datetime, $msg) = @{$_[0]};
  return "$phone,$dir,$datetime,$msg\n";
}

sub writeMessageFile(\@$){
  my @messages = @{shift()};
  my $file = shift;
  open FH, "> $file" or die "Could not open $file\n";
  @messages = sort {$$a[2] cmp $$b[2]} @messages;
  for my $msg(@messages){
    print FH messageToString $msg;
  }
  close FH;
}

sub writeContactsFiles(\@$){
  my @messages = @{shift()};
  my $dir = shift;

  my %byContact;
  for my $msg(@messages){
    my $phone = $$msg[0];
    $byContact{$phone} = [] if not defined $byContact{$phone};
    push @{$byContact{$phone}}, $msg;
  }

  for my $phone(keys %byContact){
    my $file = "$dir/$phone.sms";
    $file =~ s@:/org/freedesktop/Telepathy/Account/ring/tel/ring@@;
    my @messages = @{$byContact{$phone}};
    writeMessageFile @messages, $file;
  }
}

sub removeUSCountryCode(\@){
  my @messages = @{shift()};
  for my $msg(@messages){
    my $phone = $$msg[0];
    $phone =~ s/^\+?1(\d\d\d\d\d\d\d\d\d\d)$/$1/;
    $$msg[0] = $phone;
  }
  return @messages;
}

sub removeDupes(\@){
  my @messages = @{shift()};
  my %strings;
  my %onelineStrings;
  for my $msg(@messages){
    my $str = messageToString $msg;
    
    my $oneline = $str;
    $oneline =~ s/[\n\r]+//g;
    if(defined $strings{$str}){
      next;
    }elsif(defined $onelineStrings{$oneline}){
      my $prevMsg = $onelineStrings{$oneline};
      my $prevStr = messageToString $prevMsg;

      my $strLines = @{[$str =~ /([\n\r])/g]};
      my $prevStrLines = @{[$prevStr =~ /([\n\r])/g]};
      if($strLines > $prevStrLines){
        #previous one is no good, missing newlines
        delete $strings{$prevStr};
      }elsif($prevStrLines > $strLines){
        #this one is no good, missing newlines
        next;
      }else{
        print "WEIRD: newline count is the same, not skipping\n";
      }
    }
    
    $strings{$str} = $msg;
    $onelineStrings{$oneline} = $msg;
  }
  return values %strings;
}

&main(@ARGV);
__________________
~ teleshoes ~
 

The Following User Says Thank You to wolke For This Useful Post:
Posts: 39 | Thanked: 37 times | Joined on Mar 2012 @ Beijing
#32
check wolke's script.

I just finished mine before I found out that
I also paste it here: http://wwwjfy.net/tools/merge_csv.py
It's not as powerful as his. Simple enough, just merge two csv files into one.
I finished it in a hurry, so please check before you delete the original files or restore.

you need python installed
 

The Following User Says Thank You to wwwjfy For This Useful Post:
Posts: 986 | Thanked: 1,526 times | Joined on Jul 2010
#33
sorry, i should have posted this weeks ago.
__________________
~ teleshoes ~
 
Posts: 986 | Thanked: 1,526 times | Joined on Jul 2010
#34
Originally Posted by thedead1440 View Post
hi how can we merge two or more files from the backup.

during the backup i rec'd the error "the background thread has stalled" due to which the newer backups are smaller file sizes than the first backup. the new backups contain new smses and less than 1/3 of the old smses. so i want to merge these files and import them back.

thanks for your help
the way you use it to do what you want.
1) move all your backups to /home/user/MyDocs/sms-backup
2) add ".sms" to all the backups there
3) run "perl n9-sms-tool.pl split" to populate sms-backup-repo
4) change "30 days ago" in the script to whatever you like.
to do all messages, change it to "jan 1 1970".
5) run "perl n9-sms-tool.pl join" to make a file called "filtered.sms" in sms-backup
6) run smsbackuprestore import filtered.sms
__________________
~ teleshoes ~
 

The Following 2 Users Say Thank You to wolke For This Useful Post:
Moderator | Posts: 6,215 | Thanked: 6,400 times | Joined on Nov 2011
#35
thanks it worked a charm.
 
Posts: 986 | Thanked: 1,526 times | Joined on Jul 2010
#36
Originally Posted by thedead1440 View Post
thanks it worked a charm.
glad it worked
__________________
~ teleshoes ~
 

The Following User Says Thank You to wolke For This Useful Post:
Posts: 8 | Thanked: 10 times | Joined on Apr 2012
#37
Hi,

I'm attempting to back up, as my N9 is imploding in on itself. I'm trying to preserve the texts I've exchanged with my lady for the past few months, since my last backup.

I used to be able to backup with PC Suite and an NBU, not so any more.

When running your script I got a timeout error, restarted tried again, now I get a huuuuuuuuge error - with something looking like DB values. It starts with:

GDBUS.Error rg.freedesktop.Tracker1.SparqlError.Internal: Operation was cancelled.

After this it just parses a few hundred lines of what looks like a db dump...

Any advice?

Edit; Forgot to mention, I was able to get some backup done previosuly.. only 16.2 kb though.. and not a lot of texts.. but some. I actually got two of that size, several of 0kb when the "timeout" thing happened.

Last edited by Froberg; 2012-04-13 at 20:18.
 
Posts: 986 | Thanked: 1,526 times | Joined on Jul 2010
#38
Originally Posted by Froberg View Post
Hi,

I'm attempting to back up, as my N9 is imploding in on itself. I'm trying to preserve the texts I've exchanged with my lady for the past few months, since my last backup.

I used to be able to backup with PC Suite and an NBU, not so any more.

When running your script I got a timeout error, restarted tried again, now I get a huuuuuuuuge error - with something looking like DB values. It starts with:

GDBUS.Error rg.freedesktop.Tracker1.SparqlError.Internal: Operation was cancelled.

After this it just parses a few hundred lines of what looks like a db dump...

Any advice?

Edit; Forgot to mention, I was able to get some backup done previosuly.. only 16.2 kb though.. and not a lot of texts.. but some. I actually got two of that size, several of 0kb when the "timeout" thing happened.
some advice; make a backup of your tracker db
~/.cache/tracker/meta.db and ~/.cache/tracker/meta.db-wal
__________________
~ teleshoes ~
 

The Following User Says Thank You to wolke For This Useful Post:
Posts: 8 | Thanked: 10 times | Joined on Apr 2012
#39
Originally Posted by wolke View Post
some advice; make a backup of your tracker db
~/.cache/tracker/meta.db and ~/.cache/tracker/meta.db-wal
I'm trying cp /home/user/.cache/tracker/meta.db /home/user/MyDocs but I get a permission denied error..

wolke, I see you wrote this post as well: http://talk.maemo.org/showpost.php?p...5&postcount=18

Any chance you can tell me the pain-in-the-*** way of going about it? =)

edit: (added question and fixed link)

edit2: I think I got it.. 1MB file of text messages.. seems like the important stuff is there.. so I'm re-flashing my N9. I had a battery usage at idle at 200+ mA.. so it was fairly critical =)

Thanks for the help anyway, man!

Last edited by Froberg; 2012-04-13 at 21:47.
 
Moderator | Posts: 6,215 | Thanked: 6,400 times | Joined on Nov 2011
#40
hi,
i've used the following http://forum.meego.com/showthread.php?t=5725 to backup my smses and call logs direct to my gmail account. could you create something similar which helps not only to backup but to also re-import.

i managed to import my smses from your script above but my call logs are backed up to my gmail account and can't be imported back due to my limited coding knowledge. the publisher isn't interested in creating an import utility.

sorry if it bothers you and thanks in advance.
 
Reply

Thread Tools

 
Forum Jump


All times are GMT. The time now is 20:49.