maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Development (https://talk.maemo.org/forumdisplay.php?f=13)
-   -   Time for an open maemo fork? (https://talk.maemo.org/showthread.php?t=70680)

paulkoan 2011-03-09 22:41

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by TiagoTiago (Post 963295)
Regarding contacts storing standard, do you see anything wrong with VCard files?

vcard is a standard format for representing a contact, but doesn't necessarily need to be the storage format - like a one for one vcard text file per contact.

You could have the contacts in a sqlite database with any structure provided there was an interface that produced a vcard in response to a query.

Personally, I'd like to see that interface be groupdav provided it can be made efficient enough. That way we can make synchronisation a simple affair for tasks, contacts, calendar, etc.

wmarone 2011-03-10 04:31

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by Funklord (Post 963257)
String parsing interfaces are a big no-no on most embedded platforms since they waste memory and time

These platforms that we call smartphones are far from what is considered "embedded" these days. To reach the blurring point between "embedded" and "phone" from the industrial perspective you either step through the looking glass into the realm of the baseband (where vxworks or some other RTOS rules the day,) or drop down under 16MB of RAM into the dumbphone world where they are trying to wring every last cent out of the system.

At the level we're working at, and with the hardware in these platforms, string parsing is totally trivial and you're better off working on minimizing wake-ups and overall power saving.

Quote:

and how is it better than just using a shell interface?
dbus is common across Linux platforms, and has its own maintenance team. I'm not quite sure what you mean by a "shell interface" though.

Quote:

Originally Posted by Funklord (Post 963650)
It may be fine for n900 and similar platforms with huge amounts of memory, (once we get rid of MMC and maybe have all that 32G as pure, directly accessible nand chips)

It's unlikely you'll ever see raw NAND in that quantity. The eMMC interface simplifies system bring up so much it's ridiculous.

Quote:

I believe the largest part of the phone market needs a root system measured in a few M not G.
Certainly, however that also limits the tasks those phones are capable of. So as a result you have phones on the low-end (dumbphones) with a couple megs of space for the OS, and on the high end you have several hundred or more.

Quote:

Look at openWRT project, it's easy to configure, comes out tiny, and they developed it out of a need for a better router distro. Now every mfg and their mom is using it in at least a few of the higher end routers since it saves them money and users get a better product.
openWRT is awesome, but with only 8MB of storage in my WNDR3700 I can't do a whole. In fact, I only have 1MB free and there's more I'd like for it to do, which will probably involve me performing acrobatics (or contortions) to make it boot from an external 8GB USB stick.

Funklord 2011-03-10 22:39

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by wmarone (Post 964598)
These platforms that we call smartphones are far from what is considered "embedded" these days. To reach the blurring point between "embedded" and "phone" from the industrial perspective you either step through the looking glass into the realm of the baseband (where vxworks or some other RTOS rules the day,) or drop down under 16MB of RAM into the dumbphone world where they are trying to wring every last cent out of the system.

I like the wringing part.

Quote:

Originally Posted by wmarone (Post 964598)
At the level we're working at, and with the hardware in these platforms, string parsing is totally trivial and you're better off working on minimizing wake-ups and overall power saving.

String parsing is also inherently dynamic, along with a higher likelyhood of timeouts, races and security issues.

Quote:

Originally Posted by wmarone (Post 964598)
dbus is common across Linux platforms, and has its own maintenance team. I'm not quite sure what you mean by a "shell interface" though.

Yeah, I remember when dbus first came, but I've always linked it in my mind with very high level stuff like gnome and kde. By a shell interface I mean pipes, fifo, stdio or what ever might be appropriate. For example: mplayer slave mode.

Quote:

Originally Posted by wmarone (Post 964598)
It's unlikely you'll ever see raw NAND in that quantity. The eMMC interface simplifies system bring up so much it's ridiculous.

I'm very interested to hear more if you have insight on the issues?
At what point? bootloader? linux?
I always thought the MTD subsystem was extremely flexible, was thinking of learning how to develop a larger flash memory bus for an arm board.
Do you know why pure nand costs more than nand+ftl?

Quote:

Originally Posted by wmarone (Post 964598)
openWRT is awesome, but with only 8MB of storage in my WNDR3700 I can't do a whole. In fact, I only have 1MB free and there's more I'd like for it to do, which will probably involve me performing acrobatics (or contortions) to make it boot from an external 8GB USB stick.

Stuck a usb sound card and mpd in mine.. but realized I didn't really need it :P

TiagoTiago 2011-03-11 00:50

Re: Time for an open maemo fork?
 
Regarding synchronizing contacts, even when both have been edited between synchs, i can't see the algorithm being all that complicated.

Always store both last modified date and last synch date.

If the number of each type(and subtype) of field is the same, consider changes to be editing them.

If the number is different, look for matches between the two contacts, leave the fields that match alone, the ones that aren't the same on both you treat as being added or removed (for additional advanced functionality, look for similar values between fields of the same type and of similar type (landline, cell and fax numbers, plain, work and home substypes etc) looking for potential changes and optionally ask the user whether they are the same or should be treated as seaparated)


In cases where both contacts have been changed between synchs, consider differences to be conflicts and have the user manually solve the conflicts. In case the synch dates (and times) don't match, ask the user what to do (choose on a case by case basis, use the most recent, oldest, from one source instead of other, the biggest one etc, with an additional option to remember this choice (for this session, forever or no, etc)




Did i miss anything?

arkanoid 2011-03-11 01:02

Re: Time for an open maemo fork?
 
Having truly open fork, i'd suggest moving to a different hardware. Chinese phones look promising..

Funklord 2011-03-11 15:43

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by TiagoTiago (Post 965394)
Regarding synchronizing contacts, even when both have been edited between synchs, i can't see the algorithm being all that complicated.

Always store both last modified date and last synch date.

If the number of each type(and subtype) of field is the same, consider changes to be editing them.

If the number is different, look for matches between the two contacts, leave the fields that match alone, the ones that aren't the same on both you treat as being added or removed (for additional advanced functionality, look for similar values between fields of the same type and of similar type (landline, cell and fax numbers, plain, work and home substypes etc) looking for potential changes and optionally ask the user whether they are the same or should be treated as seaparated)

In cases where both contacts have been changed between synchs, consider differences to be conflicts and have the user manually solve the conflicts. In case the synch dates (and times) don't match, ask the user what to do (choose on a case by case basis, use the most recent, oldest, from one source instead of other, the biggest one etc, with an additional option to remember this choice (for this session, forever or no, etc)

Did i miss anything?

Yes, syncing is not as easy as it first looks, and all approaches have severe down sides.
The best automatic syncing mechanisms use a lot of metadata and complicated algorithms to determine what to do when for example a field has changed on two of the directories being synced.
You have to accept that a date field is NOT reliable in a PIM application. If you rely on it solely, then you will eventually mess up users data, important data.
Last sync date metadata is a bad idea, because the sync protocol becomes a 2 dir sync per design, which also needs to be paired at the very start
We know that in the real world, most syncs are 3-5 way, and that many of them will be directories that all already contain data, but have never seen each other before.
So, merging needs to be optimised for partial, N-way syncing, and assume directories have been created separately by different people.

Each field needs to have at least a unique id and a change counter metadata. The change counter + id is the most reliable source of information for merging, it can be combined with a date metadata, so you can at least guess what to do with very complicated merges.
As I said, it's far from easy. And auto-merging that works for most people may still be wrong for a some use cases.

What most solutions are missing, is a good way for the sync protocol to interrupt the user and with a proper GUI prompt the user for a merge solution.
That's why we often lose so much data when using most of the existing sync protocols.

Funklord 2011-03-11 15:49

Re: Time for an open maemo fork?
 
I almost forgot, a good PIM editor should always include a facility and UI to merge multiple contacts into one.
Then at least it doesn't become too much of a hassle if you accidentally get duplicates, when merging with a new dir.

wmarone 2011-03-11 17:51

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by Funklord (Post 965328)
I like the wringing part.

Seriously, if they could twist them and have pennies(won,yen,yuan) fall out, they would.

Quote:

By a shell interface I mean pipes, fifo, stdio or what ever might be appropriate. For example: mplayer slave mode.
Qt applications don't compile with support for that by default IIRC (for instance, printf() will execute but nothing will appear on the console unless you change how it is built.) I'm not sure why dbus appeared, but I'm guessing there are limitations in the methods you describe that necessitated it... can't say for sure.

Quote:

I'm very interested to hear more if you have insight on the issues?
At what point? bootloader? linux?
For bootloader and kernel you have OneNAND devices (packaged with your RAM and installed on top of the SoC,) which still accept raw NAND commands. Generally, however, this is one or two devices set up in a certain fashion to act like one.

Quote:

I always thought the MTD subsystem was extremely flexible, was thinking of learning how to develop a larger flash memory bus for an arm board.
It is quite flexible, but eMMC lets you hide all the NAND + FTL complexity behind a standard block interface and share a driver with your SD card. It also dodges the issue that a lot of filesystems geared towards raw NOR/NAND devices don't behave well on larger devices (more CPU, more RAM.)

Quote:

Do you know why pure nand costs more than nand+ftl
No, I would have to go look it up. I suspect it may be due to sheer quantity.

Funklord 2011-03-11 23:57

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by wmarone (Post 965908)
Qt applications don't compile with support for that by default IIRC (for instance, printf() will execute but nothing will appear on the console unless you change how it is built.)

I suspect they just redirect it somewhere else, probably since Qt does some of it's own argument parsing among other things, they just want you to do it "their way" instead of using printf() directly etc.
Or... came to think of it, isn't your GUI always a child process of main()?
Hmm.. I should know this since I wrote the Qt logging libraries for some companies.

Quote:

Originally Posted by wmarone (Post 965908)
I'm not sure why dbus appeared, but I'm guessing there are limitations in the methods you describe that necessitated it... can't say for sure.

If they needed that level of flexibility (multicast etc), it would've been easier just to use TCP/IP directly.

Quote:

Originally Posted by wmarone (Post 965908)
For bootloader and kernel you have OneNAND devices (packaged with your RAM and installed on top of the SoC,) which still accept raw NAND commands. Generally, however, this is one or two devices set up in a certain fashion to act like one.

So, the problem is that the MTD subsystem does not have drivers for a bus that does not yet exist, most drivers maybe assume that you've connected it to GPIOs directly?
Obviously a new memory bus would need a new driver. But to make things simpler, you could just a have a tiny nor directly connected, to store the bootloader, kernel and a simple root.
Then when the kernel is up, you use the more complicated bus and filesystem, so you don't need to develop drivers for the bootloader as well.

Quote:

Originally Posted by wmarone (Post 965908)
It is quite flexible, but eMMC lets you hide all the NAND + FTL complexity behind a standard block interface and share a driver with your SD card. It also dodges the issue that a lot of filesystems geared towards raw NOR/NAND devices don't behave well on larger devices (more CPU, more RAM.)

But a block interface does not translate well to flash memory, that's why we lose so much performance and suddenly the thing may crash when there are too many faulty flash blocks.
So, is performance an issue when you scale ubifs to gigabyte sizes?

wmarone 2011-03-12 03:58

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by Funklord (Post 966115)
So, the problem is that the MTD subsystem does not have drivers for a bus that does not yet exist, most drivers maybe assume that you've connected it to GPIOs directly?
Obviously a new memory bus would need a new driver. But to make things simpler, you could just a have a tiny nor directly connected, to store the bootloader, kernel and a simple root.
Then when the kernel is up, you use the more complicated bus and filesystem, so you don't need to develop drivers for the bootloader as well.

I'm not quite sure what you're getting at. In any case, you can boot from NAND directly, pretty much every smartphone does it and an increasing number of basebands are. NOR, IIRC is being dumped due to its price, lack of density, and decreasing support for it in SoCs.

Quote:

But a block interface does not translate well to flash memory, that's why we lose so much performance and suddenly the thing may crash when there are too many faulty flash blocks.
Not really. The problem is utilizing filesystems that are unaware of how NAND behaves and treat it like a rotating disk. Properly using SSDs is a popular research topic in industry precisely because of this problem, and those are exposed solely via a block interface. Exposing the raw NAND interfaces doesn't gain you much, and complicates things as you add devices. As for performance, that's mostly a matter of parallelism and the controller. With raw NAND your controller becomes the CPU via software (and detracts from the experience.) As for faulty blocks, they should be mapped out transparently by the eMMC's logic. If not, and you're seeing block errors, then your eMMC device is either faulty or dying and should be replaced. Hard, of course, for something soldered down, but devices like this aren't designed for frequent turnover of erase blocks.

Quote:

So, is performance an issue when you scale ubifs to gigabyte sizes?
Depends on how many. I haven't done much research into UBIFS, but if you look at some of the ELC presentations, you can get an idea of memory and cpu utilization various flash filesystems incur.

Funklord 2011-03-14 00:33

Re: Time for an open maemo fork?
 
Quote:

Originally Posted by wmarone (Post 966176)
I'm not quite sure what you're getting at. In any case, you can boot from NAND directly, pretty much every smartphone does it and an increasing number of basebands are. NOR, IIRC is being dumped due to its price, lack of density, and decreasing support for it in SoCs.

I don't see NOR going anywhere, especially as small boot devices, since you can execute-in-place, it saves a lot of ram, decreases boot-up times and increases performance.

I wasn't talking about SoCs, rather normal designs, which have a typical mcu, bus and maybe 8-9 memory parts etc.

Quote:

Originally Posted by wmarone (Post 966176)
Not really. The problem is utilizing filesystems that are unaware of how NAND behaves and treat it like a rotating disk. Properly using SSDs is a popular research topic in industry precisely because of this problem, and those are exposed solely via a block interface. Exposing the raw NAND interfaces doesn't gain you much, and complicates things as you add devices. As for performance, that's mostly a matter of parallelism and the controller. With raw NAND your controller becomes the CPU via software (and detracts from the experience.) As for faulty blocks, they should be mapped out transparently by the eMMC's logic. If not, and you're seeing block errors, then your eMMC device is either faulty or dying and should be replaced. Hard, of course, for something soldered down, but devices like this aren't designed for frequent turnover of erase blocks.

Faulty blocks are a natural thing for flash, you're supposed to mark them as "bad" and just move on, using the rest of the flash.
What does the ftl do when there are not enough blocks to left to cover the "virtual size" of the block device? There is no solution, since a block device per definition cannot decrease in size.
Soon we'll all be using mlc-nand, which is very rare to get without bad blocks from the factory.
When using ftl It's very difficult to predict what will happen during brownouts etc. just that fact makes it unsuitable for embedded designs in general.
In my experience the speedups I've seen when using ubifs instead of ftl were huge.
Here's a good explanation of pros and cons of ftl.

Using raw nand with a suitable filesystem allows you to use the flash memory as you'd use a hard drive normally, without worrying about waring it out prematurely. You can switch on things like logging etc. And since it's transparent, you can see if your bad-blocks list is increasing at an alarming rate. (with ftl that would be hidden, or a proprietary interface)

An FTL chip will also utilise at least as much power as you'd have required for the added cpu-time and ram, so there's no difference here.

IMO FTL is just a temporary solution to use flash with old filesystems and operating systems, we don't really need it anymore and should get rid of it.

Quote:

Originally Posted by wmarone (Post 966176)
Depends on how many. I haven't done much research into UBIFS, but if you look at some of the ELC presentations, you can get an idea of memory and cpu utilization various flash filesystems incur.

I checked, ubifs is suitable up to 16GiB according to them.

norayr 2013-02-21 12:45

Re: Time for an open maemo fork?
 
still I believe there is a need for free maemo fork. because mer/nemo is not maemo. I'd like to see free maemo to have more diversity within free mobile os population


All times are GMT. The time now is 11:58.

vBulletin® Version 3.8.8