I would be surprised if you are really able to damage the back cover, it seems to be much stronger than it may appear at first. Also, after a couple of times removing the back cover will feel natural and not quite so stiff anymore.

But yes, on first contact it probably makes most people a little nervous. Don't be afraid to use some force.