I've slid pyrmids on my line through the sinker eye, but that's only when I've gone to the water without any snap swivels. It works, but I only do that in a pinch. I've never really considered not using a snap swivel on purpose. I don't know if would damage your line any worse than using one.
However, not using a snap swivel does take more time when you want to change weights.
We have the same problem at LIP. Not rocks, but a huge snag with rigs, line and sinkers all over it. I've been told it's a huge cypress stump, but I don't know fer sure.
Anyway, what we all do when we retrieve our rigs is lift the rod tip straight up and crank like hell. It doesn't always work, but it does the trick most of the time.
But I guessing you already know that trick.