I noticed xorg using a bunch of cpu too. It never seemed to affect battery life or performances, though, so I'd just leave it be.