The great offline issue [Archive] - maemo.org

View Full Version : The great offline issue

RogerS

2007-02-17, 15:48

Why I don't want to synch

For me, the 38-minute train ride from Montclair into New York City is my prime personal time for computing and writing. And fast or slow, the WiFi-ization of America won't reach that zone for a long while I'm sure.* That means the commute is offline for now and hereafter.

So I diligently worked out a plan to pluck information off the web every morning, put it automatically into my preferred reading format and transfer the info to my internet tablet to read in FBReader.

Since FBReader gobbles up the Plucker pdb format rather handily and Plucker desktop efficiently automates the webpage plucking, I thought this would work nicely.

I was wrong.

Too much me

The process involves me too much and requires two computers. It's a system that was designed to use a person's desktop computer for the plucking and processing, synch to a Palm PDA, and utilize the Palm for reading.

But why should I have to synch? My internet tablet has WiFi. It will run a python program. It's got a great e-reader already.

Ah, the Linux version of Plucker desktop uses wxWindows. I can't run it on my internet tablet. Plus it has all the synch-to-Palm conduit stuff.

What's needed is an interface written for the internet tablet that sits on top of the already-written Python plucking code.

Then every morning (and afternoon), the stuff I want to look at would be grabbed, streamlined and made ready for me to read on the train.

Python-meister available?

Would that I could develop this on my own.

But, besides not being a developer, I found Python unintelligible in my two attempts to learn it (and then found Ruby the complete opposite -- clear, elegant, intuitive). I realize I need someone who knows what they're doing to guide me when I get stuck, which is often, even at my noice level (or: because of my novice level).

Still, the Python tools available for the Maemo are so powerful and give me a real reason (and platform) to develop that I look longingly at this project and wonder, What can I do to make it a reality?

If there's a Python-meister who sees in this a project not too complex and which could be incredibly fruitful for the internet tablet community . . . well, I'll sign on as chief cook-and-bottle-washer. Tester, UI guide and documenter. Evangelist.

I'll do everything I can to make the project succeed, apart from the, um, Python part. We await only the emergence of a true code master.

________
* I could connect to a phone's data plan and surf -- I've done that (http://www.internettablettalk.com/2005/12/02/surfing-on-the-go-on-the-road-with-the-nokia-770-and-a-bluetooth-phone/), it's great! -- if I chose to squander my discretionary income on that instead of extravagances like children.Read the full article. (http://www.internettablettalk.com/2007/02/17/the-great-offline-issue/)

ptman

2007-02-17, 17:04

I've hacked together in python an offline reader for google reader that can sync the status back to google reader later. Like:

1. Fetch unread articles from Google Reader
2. Go offline
3. Read some articles, star (mark as interesting) some other, don't read all
4. Go online
5. Push back changes
6. Go to google reader using a desktop web browser
7. The messages you haven't read on the road are still unread, the interesting ones can be found and the ones you have read are no longer shown

It's still very crude, but I use it myself. I have to polish it and release it. Contact me if you are interested.

BR,
ptman

jethro.itt

2007-02-17, 18:00

But, besides not being a developer, I found Python unintelligible in my two attempts to learn it (and then found Ruby the complete opposite -- clear, elegant, intuitive).

You do realize, that you might be the only person on earth with that sentiment... :p

fpp

2007-02-17, 18:02

Didn't want to rub it in, but totally agree :-)

andymulhearn

2007-02-17, 19:30

You do realize, that you might be the only person on earth with that sentiment... :p

Now if he'd said <insert any proramming language> vs perl, I'd agree :D

bugnotme

2007-02-17, 20:31

freeman

2007-02-17, 21:24

My offline time are usually involve reading fetched rss stuff. But one thing I would rather have is built in "save webpage for offline" in the menu somewhere in the browser.

RogerS

2007-02-17, 21:41

You do realize, that you might be the only person on earth with that sentiment... :pOn the contrary -- I've encountered a number of people for whom Python does everything backwards or incompletely that Ruby does beautifully. Of course they've mostly been people in the Ruby community.

Anyway, I'm not dissing Python. I'm only saying that certain things seem natural to me the way Ruby does them that didn't seem natural at all when I was looking at them in Python. But if I were lefthanded, some tools would seem hard to me to use and others (lefthanded scissors) would seem natural. It's perfectly sensible that other people would feel differently.

RogerS

2007-02-17, 21:45

I've hacked together in python an offline reader for google reader that can sync the status back to google reader later.
...
Contact me if you are interested.
BR,
ptmanThis does sound really good.

Plucker, of course, antedates RSS. In the end RSS and Google Reader may be wildly easier than a Plucker/FBReader combination.

But so many things in FBReader are attuned so well to reading that I am loathe to give it up as the e-reader of choice.

crackhead

2007-02-17, 22:10

I understand the desire to pull down full the full text of articles, but I find the built-in RSS reader is handy for grabbing feeds in the morning before I head to the subway. I HAVE to pull stuff down first, because there's no mobile service whatsoever in the subway...and there's likely to be no service for the forseeable future. But, I would certainly be interested in a Python solution like this, too.

jethro.itt

2007-02-17, 22:32

... But if I were lefthanded, some tools would seem hard to me to use and others (lefthanded scissors) would seem natural.

"Ruby: The left-handed Python". I like that!

No, just kidding. I was just surprised to see someone describing Python as non-intuitive, as my five-year romance with Python has been nothing but joy. And I admit I've only had minor exposure to Ruby.

digitaldruid

2007-02-18, 13:51

My offline time are usually involve reading fetched rss stuff.
But one thing I would rather have is built in "save webpage for offline" in the menu
somewhere in the browser.
it would be great if someone portded wwwoffled proxy to
770/800, so that you could browse offline all pages visited online from the cache.

aflegg

2007-02-19, 09:46

Why I don't want to synch[...]

Roger, you do want to sync, you just don't want to sync with a master PC. Neither do I, I want to sync with the network - however you do want to sync:

Then every morning (and afternoon), the stuff I want to look at would be grabbed, streamlined and made ready for me to read on the train.

The point we need to make (and we agree on) is that there are valid use cases for these "Internet" Tablets without an Internet connection.

Cheers,

Andrew

TA-t3

2007-02-19, 13:08

It's been a long time since I tried Plucker, but I'm certain that I didn't use a Plucker GUI - there was a command line utility that did it all, after I had set up an initial .html file which described what I wanted Plucker to download.
A command line version of Plucker should hopefully not be difficult to get running on the N800/770, unlike a wxwindows version.

(As for Python, it's not for me either. I stopped using position-dependent code with Fortran 66, and I'm not going back - but there are other reasons too. Ruby I could do, I think. But I really need Perl! :))

>The point we need to make (and we agree on) is that there are valid use cases for these "Internet" Tablets without an Internet connection.

Indeed!

zoglesby

2007-02-20, 18:31

I found Python unintelligible in my two attempts to learn it

WOW. I have never heard anyone say that in my life. I have found python to be the easiest language that I have ever used.

Texrat

2007-02-20, 18:54

I actually just received my first 2 Python books from Amazon last week. As an old VB (and Pascal, and Lisp) programmer, I'm not too keen on the odd syntax (looks too much like C at first glance, which I despise)... but I intend to swallow hard, grit my teeth and teach myself anyway. Prayers appreciated.

zippy

2007-02-20, 19:36

Even though the syntax is different, Python is a lot like Scheme. Once you get past the syntax, you'll be fine. If you want, think of the tab key as open-paren :)

ptman

2007-02-21, 07:39

I am very interested in this. I was planning to do something similar when I got my hands on an N800. Was it designed for desktops or the Nokia tablet specifically? Is it a new RSS reader interface as well?

Right now it is a terminal application, so it's not designed for either =) I'm planning to fit it really tightly to the maemo platform, as that is it's main intended use. But hopefully I can program it in such a modular way that people can use it on their desktops too.

Right now the main problem is displaying HTML in a hildon gui programmed with Python. There's libgtkhtml, but no python bindings. And I'm not going to redo this in C. In the terminal, I just format the html through links(1).

And fetching the www-pages for offline reading is an interesting idea as well. But it doesn't work really well for, say, Slashdot, where the real content isn't pointed to via the feed.

When it's ready, it can be found at http://paul.totterman.name/maemo.html

BR,
Paul Tötterman

joeaguy

2007-02-22, 04:04

The Nokia is a powerful device with a network connection. I would like to be able to run an app from the device itself to have it grab a bunch of web pages to an offline directory. That way I sync what I want to read offline whenever I happen to have a connection.

There is great web site slurping tool called pavuk that I've used in the past. It can take a url and copy down the whole page to an offline directory, changing image paths, etc, so the whole page can be read offline. It is very configurable and intelligent and includes both command line and X windows based modes.

I downloaded a copy into a Maemo Sandbox, and it compiles and runs without a hitch, even the X version. While this could probably be put onto a Nokia as is, some work could be done to make this a Hildon app. I have never attempted this before, nor have I touched C in a few years, but it would make a cool project. It is also a very powerful program, and could probably use a simple mode that handles most sites well.

http://pavuk.sourceforge.net/

simonvc

2007-02-22, 16:48

bac522

2007-02-28, 14:40

RogerS

2007-03-07, 12:42

Hi, I wrote a distiller that pulls down content (rss feeds + weather + news sites) and distils it into pdb format.

You can get it at http://simonvc.googlepages.com/thedailypluck

I heard it has some problems with the N800 (but works on my N770)

Good Luck
SimonFantastic!

I'll try it with both the N800 and the 770.

For all us offline-on-the-train and FBReader readers, thanks!

Roger

TA-t3

2007-03-07, 13:21

I've used Avantgo (http://www.avantgo.com/frontdoor/index.html) on my pocketPC that does a great job of providing off line content. I did a search on google for Avantgo Linux and come up with this site, http://quark.humbug.org.au/publications/various/avantgo.html but it looks like the source code link is bad. I'm sure with a little bit of googling one could find the source code. Wonder if this could be built to work on the Nokia's?
I use Avantgo on my Palm. The links on that page points to an old version (3.3) of AvantGo for Palm, there are newer versions on avantgo.com. The malsync code is for Linux, the link is bad, but 2.2.0 is out there somewhere. However, this is not Avantgo for Linux, it' just a conduit handler that downloads the data from sync.avantgo.com and pushes it to the Avantgo application on the Palm, which handles the rest. So it's not really an avantgo solution for Linux, it's a replacement for when you use Linux and not Windows to hotsync. (And I do neither, as newer versions of Avantgo can update directly from sync.avantgo.com through BT or Wi-Fi networking.)
I too would like to have Avantgo on the Nokia Internet Tablet, it would be a great application for a device with that screen and networking speed (for the synchronization part).

(As a sidenote, I've always wanted to take a better look at that malsync code to see if it would be possible to make it into a two-stage downloader; e.g. either store the data directly into a .pdb file that could later be pushed to the Palm, or, if not, save the data in some other way so that the malsync->palm interaction could happen later. I can't see why this wouldn't also work on the N800. Particularly with the .pdb file push this would be great to get running on the N800, because it could directly push the file through BT to the Palm with no additional coding (unlike with hotsync, which would need the pilot-link software to be ported too). This could be useful if you're without wi-fi on the PDA (could be made to work with Win PDAs too, presumably.))

w14

2007-03-07, 13:30

The Daily Pluck seems good, but it doesn't allow access to websites that require authetication, for example. (any plans?)

I use Sunrise, which was formerly called JPluck. There's a howto for Ubuntu at http://www.howtoforge.com/comment/reply/1190. It works on Windows as well. Download details are in the Ubuntu howto.

There's a great tutorial here:

http://wiki.mobileread.com/wiki/SunriseXP_tutorial

Mike.

ptman

2007-03-14, 16:47

In case someone is interested, I just put out a first public release of maemo-google-reader. You can get it from my home page http://paul.totterman.name/maemo.html . Please give me some feedback. I know that it's a terminal based program and I would like to make it graphical, but I couldn't find a nice html rendering widget that was accessible from python. libgtkhtml didn't have python bindings available for maemo.

idomcfly

2007-03-14, 17:07

ptman

2007-03-14, 17:16

I don't know if they are available on maemo though

They're not, at least in any repository I searched. Also that gtkhtml-python-binding is for gtk+-1.2 which is so outdated that it's not even funny.

ragnar

2007-04-02, 09:54

This article just popped up:

http://www.37signals.com/svn/posts/347-youre-not-on-a-****ing-plane-and-if-you-are-it-doesnt-matter

bergie

2007-04-02, 18:42

This article just popped up:

http://www.37signals.com/svn/posts/347-youre-not-on-a-****ing-plane-and-if-you-are-it-doesnt-matter

I wrote some N800-related comments about this in my blog:

http://bergie.iki.fi/blog/the_old_offline_vs-online_debate.html

TA-t3

2007-04-03, 11:17

I read that article too, and I think the comments were better than the article.. :)

There's a reason for wi-fi hotspots being called "hotspots".. they're short-range islands in an ocean of non-connectivity. Phone connection is fine as far as it goes, and I use it often, but it doesn't go that far: GPRS, at least, is not wideband, and it's costly. And as soon as you go outside your own country (as many people do all the time in Europe) you'll find that a) it's insanely expensive, and b) your provider may not even have roaming deals for GPRS on the local carrier you happen to connect to.

And even where you're reasonably well-covered with wi-fi hotspots, if you're moving (in a bus, on a train, in a tram) you will want to quickly download or synch to your mobile device while waiting for the transport, then read off-line while on board. (This is of course a totally alien concept for an American, where either you're driving a car and thus not working on your N800, or you're stationary.. ;))

I also read Bergie's blog comments.. on the issue of synching protocols, I was thinking that maybe something could be built on top of 'git', for those who are familiar with it.

geneven

2007-04-03, 14:31

Yes, some sort of Avant-go like syncing like my Rexx 6000 had in 2000 is important. I spend all day in a non-wifi environment. I have wifi at home, but I also have three desktop computers there. The amount of time I spend in places where I have wifi but don't have a desktop computer is usually 0%. Syncing would be great, for the times I don't have wifi or a desktop computer.

_Chris_

2008-01-25, 21:26

The great offline issue - finally solved!?!

Thanks to loki and his version of pyplucker (Forum Post (http://www.internettablettalk.com/forums/showthread.php?t=11149&highlight=pyplucker), Repository (http://maemo.lancode.de/)) you can gather relevant news right on your device, whenever you are online, without the need for syncing.

(I assume here in this post you have heard of the excellent FBReader for Maemo (http://www.fbreader.org/maemo/), which displays ebooks. Coupled with a PC based web site converter like Sunrise (http://sourceforge.net/projects/sunrisexp) (or others) you can convert arbitrary websites into "take away ebooks" on your device. But this has been a two step solution until now...)

After you installed pyPlucker from the above repository, open an X Terminal on your device. You should execute "plucker-setup" now, but some error message appears, which fails on missing rights on the directory ".plucker" and sometimes ".pluckerrc". After you granted those rights for everyone (search for "how to become root" in this forum), execute "plucker-setup" again. Now it works.

You could now invoke the plucker (the web site converter) by loki's command line (for more reference on the plucker check the Documentation (http://code.plkr.org/docs/)). But I prefer to set up a "home.html" file, which acts as a default for plucker and contains all the news I'd like to update onroad.

A sample is already installed in .plucker, you just have to change it to the news you prefer. I print mine for reference:

<HTML>
<HEAD>
<TITLE>Plucker Home Page</TITLE>
</HEAD>

<BODY>

<H1>_Chris_'s News on the go...</H1>

<P><A HREF="http://www.tagesschau.de/mobileTSpic/" MAXDEPTH=2>Tagesschau</A><P>

<A HREF="http://www.heise.de/pda/newsticker" MAXDEPTH=2>Heise Newsticker</A><P>

<A HREF="http://www.golem.de/pda/pdahome.html" MAXDEPTH=2 STAYONHOST>Golem News</A><P>
</BODY>
</HTML>

If you now set some additional defaults in the ".pluckerrc", than averything works smooth:

;; 0 - silent except for errors
;; 1 - progress status
;; 2 - debugging
;;
verbosity = 0

;;
;; Path to the plucker dir.
;;
;;pluckerdir =

;;
;; Document name
;;
doc_name = Chris's news on the go

;;
;; Filename for the document
;;
doc_file = ../MyDocs/.documents/ChrisNews

;;
;; Compression type
;;
;; doc - use the DOC compression (works on all supported versions)
;; zlib - use ZLib compression (doesn't work on 2.x devices)
;;
;; Zlib compression is typically much better than DOC compression.
;;
compression = zlib

;;
;; Bits per pixel for images (0 to means 'no images')
;;
bpp = 8

;;
;; Max width and height for the images. Alternative maximum width
;; and height can also be specified. These values are used for 'big'
;; versions of inlined images that had to be scaled down in size to
;; obey the maxwidth and maxheight parameters.
;;
maxwidth = 400
maxheight = 240
;;alt_maxwidth =
;;alt_maxheight =

;;
;; If an image is smaller or equal to the given limit (in bytes)
;; the image will not be compressed.
;;
image_compression_limit = 300

:) Voila, you are ready to start your first plucker by entering "plucker-build". On my WLAN-connection it takes roughly 3-5 minutes for the above 3 sites, which is ok for me in the morning. I tried once over the mobile (3G), but canceled after 10 minutes...

After it finished, there should be a document called "ChrisNews" (or what you have named it :D) ready to open with FBReader or Plucker.

By the way: since out beloved tablets are not rockets, you should restrict the updates to the absolutely neccessary pages. It helps to add:

0:-:http://heise.ivwbox.de/.*
0:-:http://www.heise.de/ivw-bin/.*
0:-:http://tagessch.ivwbox.de/.*
0:-:http://golem.ivwbox.de/.*

to the end of the file "exclusionlist.txt" in the ".plucker" directory to avoid a lot of web-bug-links...

Ok folks,
happy news-harvesting on the go!

Cheers,
Chris

PS: anyone knows how to make a start menu entry for the shell command "plucker-build"? I would like to start it with a tap... :)

iamthewalrus

2008-01-25, 21:43

PS: anyone knows how to make a start menu entry for the shell command "plucker-build"? I would like to start it with a tap... :)

Add a .Desktop file in /usr/share/applications/hildon

Seems like a very useful script! Do you have any plans to make a graphical frontend and single click install/repo ?

_Chris_

2008-01-25, 21:49

Add a .Desktop file in /usr/share/applications/hildon

Thanks for the fast reply. Edit: I now have the following "plucker.desktop" file in the above location:

[Desktop Entry]
Version=1.0
Encoding=UTF-8
Name=PluckerIt
Exec=/usr/bin/plucker-build
Icon=terminal
Terminal=false
Type=Application
X-HildonDesk-ShowInToolbar=true
X-Osso-Type=application/x-executable

Works in starting the script without an open xterm (so you can't can see when it has finished though...). Can be startet by SimpleLauncher (Just one tap! :D ). Thanks to Maddler for the template...

Seems like a very useful script! Do you have any plans to make a graphical frontend and single click install/repo ?

I thought about writing a Python frontend, where you could enter the webadress, specify outputfilename, image dimensions etc., in fact did already the user interface design, but so far didn't find the time to implement it. Maybe later...

Cheers and thanks for the hint!

_Chris_