I like the prospect of this!

I don't know how pocketsphinx works but you said it needs dictionary to recognise words. does this mean that a dictionary can contain regional dialects or accents?
I have a Geordie accent, which, along with others accents such as Scottish or Welsh, means that most speech recognition engines have a major problem understanding me unless I speak queens-english to match it's dictionary of pronunciation.

My last attempt to do anything speech-related was when i had a SideWinder Headset and voice software.. I was able to type the phonetic version of how it would sound eg If I said "Sword", it would be "Saud" phonetically..