When BT is in 'invisible' mode it isn't transmitting at all. It is only a receiver. It would not be possible to detect that it is there unless you are at the time communicating with it, so for a 3party person it does not make any difference if it is 'on' in receiver mode, or not, when you yourself isn't currently communicating through BT.

So an attacker would first have to collect info about your phone while you're using BT actively, and could then at a later time try to connect to your listen-only BT. In both cases they would have to get close to you (10 meters) (yes I know about the special long-range narrow-angle BT cracker's antenna.. not very practical for war-driving really).

So yes, there's a possibility for breaking into your BT phone, but not as easy as war-driving for open (or wep-protected) wi-fi access points: The area of near-enough for potential break-ins is vastly larger for wi-fi than for BT, and can usually be done in one go instead of first collecting info and then coming back later.

I choose to consider that risk very low, so I ignore it and leave my BT on (in invisible mode), but of course everyone is free to judge the risk differently. (BT in visible mode is an entirely different story of course -- that can be extremely problematic. For example, a student's pub I visited tried to automatically push some kind of advertisements to every BT device visible in the building. Very annoying.)