I have over 700 contacts, almost all with images, synced with Hermes and Outlook.... And most have IM enabled. Mine takes about 2 seconds to load up. Something is clearly not right with your setup.

Maybe try disabling IM and/or seeing what plugins are enabled? Do you have a way to sync the contact list out somewhere (like to Outlook?). If so, you could do that, nuke the local contact db, and re-sync them all back in. Assuming maybe the db is corrupt in some way?