Macroing Using 770's Handwriting System

[Ceklund]

Sign Language MACROING System for 770

PURPOSE:
By drawing assigned shapes anywhere on the screen (with the rocker key held down first) we can have the 770 execute 100's of quick tasks we pre-arranged via the Handwriting "teaching" mode for the Macro system ... in order to stay in Full Screen mode with no interfaces cluttering our experience and yet do everything even quicker and easier than with the interfaces!

To use the Macro system, we would enable the Handwriting Sign Language system from the desktop at the Home option bar (where we turn on/off rss reader, internet radio, etc....) Once enabled, the Hardware Enter button at the center of the 770's joystick no longer = Enter... it's now only the Sign Language shift key for real-time MACROING. The entire screen is now ready to be drawn on AS LONG AS THE HARDWARE ENTER IS BEING HELD DOWN.

FUNCTION: Excute tasks, or macros, and switch between Opera windows (like Alt-Tab for Windows.)

PURPOSE: ... to allow us to go to Full Screen mode in browser or other software and still perform ALL needed functions at faster than current speeds w/o having to drop back down to perform actions in the standard mode.

HOW IT WORKS: We would train the S.L. commands using the 770's already-existing Training tool for it's Handwriting Recognition system. We draw a shape and assign it to a database of shapes... say 100's of slots big... from which we can set up tricks to help us work and play better.

EXAMPLE:
Say I'm working in two Internet browser windows... to switch between them, I would hold down Hardware Enter and draw a line forwards left to right.... Bam! the other window comes up to the front. And on this window is a picture of a 10-speed bike I'm looking for, but I only save bike images to the Bike folder on my 512 mmc. Saving on the 770 is too time consuming, so I've taught the Macroing system that an upside down V shape is going to have the following rules (assignable from a cool list of rules and options): 1) Find the image that sits below last drawn shape the user just drew, 2) Save that image to “/mmc/Pictures/Bikes,” 3) Do not override pics with same names... append random character and re attempt save./ Now I can assign flags like this to dozens of macros as their own unique Macro. Thus, just as one example, every type of picture I see I can instantly send--AS I'M BROWSING--to it's proper folder w/o having to damage my browsing experience! Real-time, TOTAL control of my system w/o switching out of Full screen mode! I see a picture of flowers, I send that picture automatically to the "mmc/Pictures/Flower" folder by drawing a 3-leaf clover. I send the picture of a cute woman to "mmc/Pictures/Models" etc... all without having to even see the Save interface. It just captures the image, and sends it to the location you assigned in the Macro, executing any rules you assigned while letting you know by displaying the results on screen.

EXAMPLE RESULTS: In pop up window along top, it says:
Picture "Bike3" saved to "mmc/Pictures/Bikes" as "Bike3(c)" ...

This lets you know it was trying to overwrite an already-present picture called "Bike3" but instead changed the name to "Bike 3(c)" so it wouldn't overwrite the one that was there, according to your rules.

In the browser, ALL of the surfing buttons could be macro'd to shapes... Home, Update, Reload, "Save Webpage As", Back, Forward, whatever you want... all Macro'd up so we'd never need that real-estate munching task bar along the bottom again.

Types of Example options available for Macros:

1. Save item under drawn last-drawn shape to a user-definable mmc location, using a list of rules we choose.
2. Switch between up to 12 Opera browsing windows, each with their own drawn shape we assign. (This actually is already setup in 2006 beta, all windows open with their own icons attached to them for ready identification, but the Macro would let you switch between them in Full Screen mode without having to see the icons at all.)
3. Assign different sounds to Macroing commands for instant feedback on it's success.
4. Click down (simulate a user tap) at location x/y ...
5. Entering pre-defined text at last place clicked by previous command (#4).
6. Hard Returns for accepting typed-out text from previous command (#5).
7. Assign user-shapes to Bookmarks for instant loading of any and every bookmark w/o having to even look through them!

The list could grow if the idea is accepted.

[Ceklund]

------ Additional Comment #1 From Maemo QA 2006-06-30 20:38 [reply] -------

Thanks for your feedback.

You mean this would be like gestures in desktop Opera browser?
Marked as an enchantment request.

------- Additional Comment #2 From Christopher Eklund 2006-06-30 21:41 [reply] -------

I'm not familiar with Gestures per se, but it sounds like it could be similar.
The 770 already has the software to do this built in: go to the button with a
page on it with iines, go to Tools>Teach. Now choose 'Shortcuts' from the
drag-down bar.... and there you see 16 slots where you can draw ANY shape you want to and apply to that shape a portion of text. It's half done already!

With my enhancement ideas implemented, we could be free of the interface
entirely. This enhancement would also appeal to asian cultures that rely on
huge character sets and image-based words, etc....

You see a picture on screen you want to save. When you built your Macros, you
told your 770 that a closed triangle shape has the following properties:

1)Save to current memory card at '/mmc/Documents/Images/Photos/Car_Images/'
2)If image name already exists for currently-being-saved image, append a random letter or character (the other option is *number*.)
3)When saving the image, reSize image down to 1280 x 986 BUT if image is smaller than this size already, do not scale up! ('keep aspect ratio' is option.)

All we the user did was draw one simple closed triangle over the Firebird car
picture, and the system does all the previous after first asking if the correct
image was captured like so:

'Macro 62,Triangle: Save image to mmc/Documents/Images/Photos/Car_Images using option 'Resize' | option 'Rnd Char Append'+Aspect Ratio'? We then choose Yes or No, in which case we redraw the shape for the interpreting system and try again.

What if we want to rename the file entirely while also saving it under the
triangle Macro? The interpreter does not perform any action until the Hardware
Enter button (our shift key) is released. SO... We draw the triangle first
then we draw a zigzag (which we setup in Macroing to warn system that next part is going to be a filename change)... followed by our new name, then a zigzag to close, leaving room for other commands after. It would look like this: [ Triangle Zigzag CoolFirebird Zigzag ] where [ ] equals when Hardware Enter btton was pressed/released, where Triangle and Zigzag ar the actual drawn shapes, not these words.

If someone is going to develop this furter, I will of course be ready to answer any
questions on other improvements to this concept.

Thank you for your time.

------- Additional Comment #3 From Christopher Eklund 2006-06-30 22:13 [reply] -------

P.S., this is NOT a mere Opera enhancement, it is a total OS enhancement. The same Macros accessible from ANY program... once you engage the Macro System, the system is always listening for the opportunity to apply its rules.

What if you're in the PDF viewer and you want to save an image you see there?
What if you're entering info into D.eli.cious and you have Macros set up for
most of your text information and all you want to do is send it into the
textfields quickly?

What if you're trying to download a video and when you click the link it goes to
a view window instead, so you go back to the link and do the following: [Square]
drawn over the link itself, where a Square shape was defined in Macro system as Find Link Shape was drawn over and execute a 'File Save As...' command, which would then treat any link the same way PC's do with File Save As.... not going into the link but capturing it to a place we choose ('predefinable mmc location' or 'choose on the fly' are the options.)

What if you see a URL that should go into the RSS Feed Reader but it's not
clickable and the 770 doesn't recognize it? Merely highlight the URL, then
execute the following: [Star] over the highlighted text, where Star was defined
in Macro System as 'Send Highlighted Text to RSS Feed Reader as Feed' which then opens that system and attempts to load it as a feed.

The same thing for a page of URL's inside a RSS Feed that you can't click on...
highlight and execute following: [Wrectangle with Circle Inside] over the
highlighted text, where Wrect w/Cir was defined as 'Open New Opera Window With Highlighted as URL' !

Do you see how far we could take this? I do. And it would be fun... no HH
device would ever have been so fluid and easy as this could become.

Later.

[cybe]

I'm using an excellent mouse-gesture app on Windows called Stroke-It which I highly recommend...

[Odin]

I'm not familiar with Gestures per se, but it sounds like it could be similar.
The 770 already has the software to do this built in: go to the button with a page on it with iines, go to Tools>Teach. Now choose 'Shortcuts' from the drag-down bar.... and there you see 16 slots where you can draw ANY shape you want to and apply to that shape a portion of text. It's half done already!


There are some signal hooks built-in to the OS. There may be a "short-cut activated" signal that would allow a script to be run.

See: stefans.datenbruch.de/nokia770/

[Ceklund]

I've had some correspondence with a man who is developing ideas related to glyph recognition in the handwriting system for the 770 about this idea. Here is what he wrote about this idea:

I was immediately reminded of Opera "gestures" (which I use). I have a couple of comments:

I'm not sure I like drawing the gesture with one hand while holding down the center button with the other. Better to define one of the buttons as a gesture enablement toggle.

[More] ideal would be to have both text and gestures available anywhere on the screen, even in fullscreen mode, not just in the input method's pop-up area.

The protocol to communicate the gesture data would be a challenge. It's helpful to sponge off an existing standard, and
"/usr/X11R6/include/X11/keysymdef.h" defines [or gives] "XK_F1 through XK_R15" 60 different function keys, plus countless other keysyms such as XK_ogonek.

I suppose the handwriting input method could be convinced to emit arbitrary user selected keysyms for arbitrary glyphs. The X clickstream includes, in addition, the location of the cursor.

It would be helpful if the user could define a hotspot for a gesture, e.g. the location of a V might be the bottom corner, not the start or end points.

An alternative is to send dbus events. I think the Maemo input methods actually do that, i.e. sending the same data in two ways. Of course the challenge is to get applications to respond to these things. In your example, Opera should know to copy the cache data for the bicycle image to a specific directory. Better, it should call a user provided script with custom behaviors: knowing the directory, and knowing it should alter the filename rather than overwrite. For generic behaviors such as changing speaker volume, there's already a daemon that does this, but it would not be useful for the example because it has no way to find the original data in the (memory) browser cache.

--James F. Carter
I'm not sure why you don't like holding down the center joystick button as a "shift key" and want it to be a toggle instead? Can you explain why you would like it as a toggle? My concept is that it's like a shift key... you can draw a shape over 3 different pictures on the same page... and tell each picture where to be saved independently, and when you let go of the center key, the interpreting system is then free to try to apply the rules you have defined for each of the shapes.

If the interpreter makes a mistake you will let it know... in the above example, a little window would pop up with 3 line items (1 for each pic) and it would have an empty checkbox beside each pic's name and the name of the glyph or symbol as it interpreted it in another column. The question in the popup window's Titlebar would read: "Check correct items:" You then checkmark 1 and 3, but leave 2 unchecked because you decided it interpreted that one incorrectly, so 1 and 3 are instantly saved, and then it pops up a window to draw on, titlebar reading "Redraw intended shape for 2nd item now..." You redraw, it asks if correct, you checkmark, and OK, and it saves according to those rules.

It takes a long time to describe, but we're talking 1-2 seconds for this whole transaction, and you never had to open a request dialogue and wait for the mmc card to populate the folders, you never had to click through dozens of layers to get to your folder, you never had to rename the file, never had to resize that image with an independant xterm operation that only a mastermind can unlock the secrets of. You just told the system "I want these rules applied to this shape, and I want it to save to this folder on the mmc" and it does it all while you're still in Full-Screen mode not having to see keyboard interfaces, etc.

Toggling the button could still work almost the same (it just wouldn't feel the same I guess), and it would be less stressful on left-handed people not to have to hold down the center button because then they couldn't write. Is this perhaps what you were thinking?

[Ceklund]

He has replied with....

[As to why I would like a toggle instead of holding it down like a shifter....] I'm imagining myself doing the process, and my imaginary wrist gets cramped when holding down the key. It's just a human engineering preference....Yes, you could hold up sending the glyph events until the center key were released, but the relevant app(s) would pretty much have to treat each one independently. I think I would prefer that each glyph (gesture) event should be sent (and acted upon) as soon as the glyph was recognized.

[In reponse to why the process of opening the mmc card takes so long, he replies]: There's something nasty in the GTK file selection widget; the slowdown isn't on the MMC card; the widget puts too much work into creating its list (and I'd prefer a list, not these useless icons).
--James F. Carter