C++ :: Reference Semantic For Pimpl Class?

I have a basic class with a pimpl, with agressive instantiation:

Code:
class MyClassImpl;
class MyClass {
public:
MyClass() : pImpl(new MyClassImpl) {}

[Code] ....

Classic/Standard. However, I do hate having to use pointer semantics for all of my operations. So I thought: If I never ever manipulate the actual pointer itself, why not just keep a reference?

Code:
class MyClassImpl;
class MyClass {
public:
MyClass() : impl(*new MyClassImpl) {}
~MyClass(){delete &impl;}
private:
MyClassImpl& impl;
};

The only downside that I see, is that I can't swap or move, but this particular object is not meant to be swapped or moved.