When you create a perspective guide, make sure you creating it in a new "ruler layer." I usually add new "ruler layer" for every extra perspective guide and ruler guide. If you create the perspective guide on a drawing layer, it will stick with it, and you can't take it off.

To turn it off from snap to the ruler, you can either press " Ctrl 7", or click on the "snap to perspective ruler" icon.

For remove the guide, you can delete the ruler layer. Or you can click on the visible/eye icon to make it invisible.