Would you mind sharing your experiments with PulseAudio, in case you still know what you did then? Even if it wasn't successful, it may help.

PS: Thanks a lot for moc!

PPS: If I remember correctly, even in /proc/asound/devices the usb device still showed up after removing it... so I guess you're right about pulse audio not likely to recover.