In a larger scale of rights overall, your rights to do something should end when it restricts other people's rights. No one person should be above another. Not really how the world works, but it's the idealistic goal.