Reply
Thread Tools
Posts: 422 | Thanked: 244 times | Joined on Feb 2008
#41
Originally Posted by TiagoTiago View Post
Regarding contacts storing standard, do you see anything wrong with VCard files?
vcard is a standard format for representing a contact, but doesn't necessarily need to be the storage format - like a one for one vcard text file per contact.

You could have the contacts in a sqlite database with any structure provided there was an interface that produced a vcard in response to a query.

Personally, I'd like to see that interface be groupdav provided it can be made efficient enough. That way we can make synchronisation a simple affair for tasks, contacts, calendar, etc.
 

The Following 2 Users Say Thank You to paulkoan For This Useful Post:
Posts: 1,746 | Thanked: 2,100 times | Joined on Sep 2009
#42
Originally Posted by Funklord View Post
String parsing interfaces are a big no-no on most embedded platforms since they waste memory and time
These platforms that we call smartphones are far from what is considered "embedded" these days. To reach the blurring point between "embedded" and "phone" from the industrial perspective you either step through the looking glass into the realm of the baseband (where vxworks or some other RTOS rules the day,) or drop down under 16MB of RAM into the dumbphone world where they are trying to wring every last cent out of the system.

At the level we're working at, and with the hardware in these platforms, string parsing is totally trivial and you're better off working on minimizing wake-ups and overall power saving.

and how is it better than just using a shell interface?
dbus is common across Linux platforms, and has its own maintenance team. I'm not quite sure what you mean by a "shell interface" though.

Originally Posted by Funklord View Post
It may be fine for n900 and similar platforms with huge amounts of memory, (once we get rid of MMC and maybe have all that 32G as pure, directly accessible nand chips)
It's unlikely you'll ever see raw NAND in that quantity. The eMMC interface simplifies system bring up so much it's ridiculous.

I believe the largest part of the phone market needs a root system measured in a few M not G.
Certainly, however that also limits the tasks those phones are capable of. So as a result you have phones on the low-end (dumbphones) with a couple megs of space for the OS, and on the high end you have several hundred or more.

Look at openWRT project, it's easy to configure, comes out tiny, and they developed it out of a need for a better router distro. Now every mfg and their mom is using it in at least a few of the higher end routers since it saves them money and users get a better product.
openWRT is awesome, but with only 8MB of storage in my WNDR3700 I can't do a whole. In fact, I only have 1MB free and there's more I'd like for it to do, which will probably involve me performing acrobatics (or contortions) to make it boot from an external 8GB USB stick.
 

The Following User Says Thank You to wmarone For This Useful Post:
Posts: 69 | Thanked: 41 times | Joined on Feb 2010 @ Sweden
#43
Originally Posted by wmarone View Post
These platforms that we call smartphones are far from what is considered "embedded" these days. To reach the blurring point between "embedded" and "phone" from the industrial perspective you either step through the looking glass into the realm of the baseband (where vxworks or some other RTOS rules the day,) or drop down under 16MB of RAM into the dumbphone world where they are trying to wring every last cent out of the system.
I like the wringing part.

Originally Posted by wmarone View Post
At the level we're working at, and with the hardware in these platforms, string parsing is totally trivial and you're better off working on minimizing wake-ups and overall power saving.
String parsing is also inherently dynamic, along with a higher likelyhood of timeouts, races and security issues.

Originally Posted by wmarone View Post
dbus is common across Linux platforms, and has its own maintenance team. I'm not quite sure what you mean by a "shell interface" though.
Yeah, I remember when dbus first came, but I've always linked it in my mind with very high level stuff like gnome and kde. By a shell interface I mean pipes, fifo, stdio or what ever might be appropriate. For example: mplayer slave mode.

Originally Posted by wmarone View Post
It's unlikely you'll ever see raw NAND in that quantity. The eMMC interface simplifies system bring up so much it's ridiculous.
I'm very interested to hear more if you have insight on the issues?
At what point? bootloader? linux?
I always thought the MTD subsystem was extremely flexible, was thinking of learning how to develop a larger flash memory bus for an arm board.
Do you know why pure nand costs more than nand+ftl?

Originally Posted by wmarone View Post
openWRT is awesome, but with only 8MB of storage in my WNDR3700 I can't do a whole. In fact, I only have 1MB free and there's more I'd like for it to do, which will probably involve me performing acrobatics (or contortions) to make it boot from an external 8GB USB stick.
Stuck a usb sound card and mpd in mine.. but realized I didn't really need it :P
 
Posts: 1,522 | Thanked: 392 times | Joined on Jul 2010 @ São Paulo, Brazil
#44
Regarding synchronizing contacts, even when both have been edited between synchs, i can't see the algorithm being all that complicated.

Always store both last modified date and last synch date.

If the number of each type(and subtype) of field is the same, consider changes to be editing them.

If the number is different, look for matches between the two contacts, leave the fields that match alone, the ones that aren't the same on both you treat as being added or removed (for additional advanced functionality, look for similar values between fields of the same type and of similar type (landline, cell and fax numbers, plain, work and home substypes etc) looking for potential changes and optionally ask the user whether they are the same or should be treated as seaparated)


In cases where both contacts have been changed between synchs, consider differences to be conflicts and have the user manually solve the conflicts. In case the synch dates (and times) don't match, ask the user what to do (choose on a case by case basis, use the most recent, oldest, from one source instead of other, the biggest one etc, with an additional option to remember this choice (for this session, forever or no, etc)




Did i miss anything?
 
Posts: 251 | Thanked: 70 times | Joined on Nov 2009
#45
Having truly open fork, i'd suggest moving to a different hardware. Chinese phones look promising..
 
Posts: 69 | Thanked: 41 times | Joined on Feb 2010 @ Sweden
#46
Originally Posted by TiagoTiago View Post
Regarding synchronizing contacts, even when both have been edited between synchs, i can't see the algorithm being all that complicated.

Always store both last modified date and last synch date.

If the number of each type(and subtype) of field is the same, consider changes to be editing them.

If the number is different, look for matches between the two contacts, leave the fields that match alone, the ones that aren't the same on both you treat as being added or removed (for additional advanced functionality, look for similar values between fields of the same type and of similar type (landline, cell and fax numbers, plain, work and home substypes etc) looking for potential changes and optionally ask the user whether they are the same or should be treated as seaparated)

In cases where both contacts have been changed between synchs, consider differences to be conflicts and have the user manually solve the conflicts. In case the synch dates (and times) don't match, ask the user what to do (choose on a case by case basis, use the most recent, oldest, from one source instead of other, the biggest one etc, with an additional option to remember this choice (for this session, forever or no, etc)

Did i miss anything?
Yes, syncing is not as easy as it first looks, and all approaches have severe down sides.
The best automatic syncing mechanisms use a lot of metadata and complicated algorithms to determine what to do when for example a field has changed on two of the directories being synced.
You have to accept that a date field is NOT reliable in a PIM application. If you rely on it solely, then you will eventually mess up users data, important data.
Last sync date metadata is a bad idea, because the sync protocol becomes a 2 dir sync per design, which also needs to be paired at the very start
We know that in the real world, most syncs are 3-5 way, and that many of them will be directories that all already contain data, but have never seen each other before.
So, merging needs to be optimised for partial, N-way syncing, and assume directories have been created separately by different people.

Each field needs to have at least a unique id and a change counter metadata. The change counter + id is the most reliable source of information for merging, it can be combined with a date metadata, so you can at least guess what to do with very complicated merges.
As I said, it's far from easy. And auto-merging that works for most people may still be wrong for a some use cases.

What most solutions are missing, is a good way for the sync protocol to interrupt the user and with a proper GUI prompt the user for a merge solution.
That's why we often lose so much data when using most of the existing sync protocols.
 

The Following User Says Thank You to Funklord For This Useful Post:
Posts: 69 | Thanked: 41 times | Joined on Feb 2010 @ Sweden
#47
I almost forgot, a good PIM editor should always include a facility and UI to merge multiple contacts into one.
Then at least it doesn't become too much of a hassle if you accidentally get duplicates, when merging with a new dir.
 
Posts: 1,746 | Thanked: 2,100 times | Joined on Sep 2009
#48
Originally Posted by Funklord View Post
I like the wringing part.
Seriously, if they could twist them and have pennies(won,yen,yuan) fall out, they would.

By a shell interface I mean pipes, fifo, stdio or what ever might be appropriate. For example: mplayer slave mode.
Qt applications don't compile with support for that by default IIRC (for instance, printf() will execute but nothing will appear on the console unless you change how it is built.) I'm not sure why dbus appeared, but I'm guessing there are limitations in the methods you describe that necessitated it... can't say for sure.

I'm very interested to hear more if you have insight on the issues?
At what point? bootloader? linux?
For bootloader and kernel you have OneNAND devices (packaged with your RAM and installed on top of the SoC,) which still accept raw NAND commands. Generally, however, this is one or two devices set up in a certain fashion to act like one.

I always thought the MTD subsystem was extremely flexible, was thinking of learning how to develop a larger flash memory bus for an arm board.
It is quite flexible, but eMMC lets you hide all the NAND + FTL complexity behind a standard block interface and share a driver with your SD card. It also dodges the issue that a lot of filesystems geared towards raw NOR/NAND devices don't behave well on larger devices (more CPU, more RAM.)

Do you know why pure nand costs more than nand+ftl
No, I would have to go look it up. I suspect it may be due to sheer quantity.
 

The Following 2 Users Say Thank You to wmarone For This Useful Post:
Posts: 69 | Thanked: 41 times | Joined on Feb 2010 @ Sweden
#49
Originally Posted by wmarone View Post
Qt applications don't compile with support for that by default IIRC (for instance, printf() will execute but nothing will appear on the console unless you change how it is built.)
I suspect they just redirect it somewhere else, probably since Qt does some of it's own argument parsing among other things, they just want you to do it "their way" instead of using printf() directly etc.
Or... came to think of it, isn't your GUI always a child process of main()?
Hmm.. I should know this since I wrote the Qt logging libraries for some companies.

Originally Posted by wmarone View Post
I'm not sure why dbus appeared, but I'm guessing there are limitations in the methods you describe that necessitated it... can't say for sure.
If they needed that level of flexibility (multicast etc), it would've been easier just to use TCP/IP directly.

Originally Posted by wmarone View Post
For bootloader and kernel you have OneNAND devices (packaged with your RAM and installed on top of the SoC,) which still accept raw NAND commands. Generally, however, this is one or two devices set up in a certain fashion to act like one.
So, the problem is that the MTD subsystem does not have drivers for a bus that does not yet exist, most drivers maybe assume that you've connected it to GPIOs directly?
Obviously a new memory bus would need a new driver. But to make things simpler, you could just a have a tiny nor directly connected, to store the bootloader, kernel and a simple root.
Then when the kernel is up, you use the more complicated bus and filesystem, so you don't need to develop drivers for the bootloader as well.

Originally Posted by wmarone View Post
It is quite flexible, but eMMC lets you hide all the NAND + FTL complexity behind a standard block interface and share a driver with your SD card. It also dodges the issue that a lot of filesystems geared towards raw NOR/NAND devices don't behave well on larger devices (more CPU, more RAM.)
But a block interface does not translate well to flash memory, that's why we lose so much performance and suddenly the thing may crash when there are too many faulty flash blocks.
So, is performance an issue when you scale ubifs to gigabyte sizes?
 
Posts: 1,746 | Thanked: 2,100 times | Joined on Sep 2009
#50
Originally Posted by Funklord View Post
So, the problem is that the MTD subsystem does not have drivers for a bus that does not yet exist, most drivers maybe assume that you've connected it to GPIOs directly?
Obviously a new memory bus would need a new driver. But to make things simpler, you could just a have a tiny nor directly connected, to store the bootloader, kernel and a simple root.
Then when the kernel is up, you use the more complicated bus and filesystem, so you don't need to develop drivers for the bootloader as well.
I'm not quite sure what you're getting at. In any case, you can boot from NAND directly, pretty much every smartphone does it and an increasing number of basebands are. NOR, IIRC is being dumped due to its price, lack of density, and decreasing support for it in SoCs.

But a block interface does not translate well to flash memory, that's why we lose so much performance and suddenly the thing may crash when there are too many faulty flash blocks.
Not really. The problem is utilizing filesystems that are unaware of how NAND behaves and treat it like a rotating disk. Properly using SSDs is a popular research topic in industry precisely because of this problem, and those are exposed solely via a block interface. Exposing the raw NAND interfaces doesn't gain you much, and complicates things as you add devices. As for performance, that's mostly a matter of parallelism and the controller. With raw NAND your controller becomes the CPU via software (and detracts from the experience.) As for faulty blocks, they should be mapped out transparently by the eMMC's logic. If not, and you're seeing block errors, then your eMMC device is either faulty or dying and should be replaced. Hard, of course, for something soldered down, but devices like this aren't designed for frequent turnover of erase blocks.

So, is performance an issue when you scale ubifs to gigabyte sizes?
Depends on how many. I haven't done much research into UBIFS, but if you look at some of the ELC presentations, you can get an idea of memory and cpu utilization various flash filesystems incur.
 

The Following 2 Users Say Thank You to wmarone For This Useful Post:
Reply

Tags
i hate you all, i love you all, just shoot him, just shoot me, nokia defiled, popcorn time, what the fork?


 
Forum Jump


All times are GMT. The time now is 16:36.