then plug in/out your head phones. You should see an output in Xterm showing a reaction when you plug them in. If not, you could have a problem in the headphone socket, check it for debris.
Try plugging in a TV and then a head-set (with mic) they should all state what they are, if they all report correctly then I would assume a software issue so you might have to think about re-flashing.
did you try pluging in another pair of headphones on the N900? When my headphone cable got damaged n900 would not recognise them but my mp3 player whould. Then it got damaged more and they stoped functioning.