[ANNOUNCE] Auto-Complete Editor

[Rob1n]

I think I've now managed to decode enough of the auto-complete dictionary format to implement a basic editor - it works for me anyway

So, I'd like to announce the snappily-named "AutoComplete Editor", coming shortly to an extras-devel repository near you. It allows deletion and addition of terms from the custom auto-complete dictionary. I'm not sure how well this'll handle international character sets, so I'd appreciate some feedback from anyone using those.

For future reference, the custom auto-complete dictionary (/home/user/.osso/dictionaries/.personal.dictionary) format is:

An 8 byte header, consisting of a 3-byte hex sequence 80 00 01 (though I've also seen 01 00 01), followed by a single byte indicating the number of different dictionaries, followed by a single byte indicating the file size (in 256-byte multiples, so 0x04 indicates a 1kb file), followed by 0x00, followed by a 2-byte sequence indicating the position the padding starts.

This is followed by one or more 8-byte dictionary entries, consisting of a 2-byte sequence indicating the dictionary language (or 0x00 followed by a single byte), then a 2-byte sequence indicating the start position of the dictionary entries, followed by a 2-byte sequence indicating the number of entries in the dictionary, followed by a 2-byte hex sequence 00 00.

The dictionary data is stored as a single byte indicating the string length, followed by the string itself. The multiple entries follow straight on from each other, with no other delimiters.

Here's a screenshot for 0.0.4 (there's been no major changes to the default layout since then):


Current releases are:

0.0.12 in extras-devel. This allows:

• Editing auto-complete dictionary
• Deleting entries with/without warnings, or instantly by selection
• Moving entries to a blacklist
• Editing of entry blacklist
• Auto-application of blacklist without launching GUI (suitable for scripting or scheduling)
• Overriding of character set used to encode/decode dictionary entries

For those having issues, please see this post for how to generate a clean dictionary file. If this doesn't show the same issue then please consider sending me the original dictionary - all received files will be kept completely confidential and will be deleted as soon as I've fixed the relevant issue. Any dictionaries can be emailed to me at: maemo at robinhill.me.uk

[DaSilva]

Sounds good, I will try it as soon as it is available with umlauts

[pelago]

Sounds interesting and useful, thanks.

If you haven't done so already, I think an interesting option to implement would be for your app to check the "apparently fixed" things, and if they don't match your expected values, asking the user to contact you, just in case these bytes aren't fixed after all.

[ivgalvez]

Thanks, thanks, thanks, thanks!

For me the auto-complete feature have been turned completely useless due to misspelled words added to the dictionary, mixing of Spanish and English words, etc.

[Rob1n]

Originally Posted by pelago

If you haven't done so already, I think an interesting option to implement would be for your app to check the "apparently fixed" things, and if they don't match your expected values, asking the user to contact you, just in case these bytes aren't fixed after all.

A very good suggestion - I'll look at adding that in.

[Rob1n]

Originally Posted by Rob1n

A very good suggestion - I'll look at adding that in.

Version 0.0.2 is now available, with this check.

[maddler]

Just tried it but got error:

File "/opt/AutoCompleteEditor/AutoCompleteEditor.py", line 17, in <module>
w.loadData()
File "/opt/AutoCompleteEditor/ACE_gui.py", line 157, in loadData
self._dict = ACEFile()
File "/opt/AutoCompleteEditor/ACE_file.py", line 34, in __init__
self.read()
File "/opt/AutoCompleteEditor/ACE_file.py", line 119, in read

Wasn't able to catch the whole stuff.
Any clue?

[pelago]

Out of interest, how did you determine the format anyway? It's a shame it isn't documented openly by Nokia.

[Rob1n]

Originally Posted by maddler

Just tried it but got error:

File "/opt/AutoCompleteEditor/AutoCompleteEditor.py", line 17, in <module>
w.loadData()
File "/opt/AutoCompleteEditor/ACE_gui.py", line 157, in loadData
self._dict = ACEFile()
File "/opt/AutoCompleteEditor/ACE_file.py", line 34, in __init__
self.read()
File "/opt/AutoCompleteEditor/ACE_file.py", line 119, in read

Wasn't able to catch the whole stuff.
Any clue?

That's an entry count mismatch error - it's failed to read the correct number of entries before hitting the padding start point. Would it be possible for you to email/PM me your ~/.osso/dictionaries/.personal.dictionary file?

[Rob1n]

Originally Posted by pelago

Out of interest, how did you determine the format anyway? It's a shame it isn't documented openly by Nokia.

Basically I just removed the dictionary & rebooted (making sure I was starting with an empty file). I then just use Notes and added a single new word at a time, copying the dictionary file out each time. I then looked through the different files and noted the differences.

From there it's just a matter of trying to figure out what each means - the format used for the dictionary entries was fairly straightforward (once I realised that the delimiter between the words was the length indicator), so it was just the header changes that I needed to look at. Some of those are pretty trivial (the number of entries increments by one each time, so it's clear what that is) but some of it is pretty much guesswork - the file size (in 256-byte multiples) for example - it's a pretty random thing to store and I don't really see why it's necessary, but that's the only thing I could figure it as being.