Zoloft and anxiety and panic attacks

I've been on Zoloft for about 4 years now (100mg) a day and I really want to stop taking it. At the time I was put on it I was just getting myself out of a really bad situation and I needed to be put on it for severe anxiety and panic attacks. I'm in a totally different situation now and am doing fine, I don't want to rely on medication anymore to make me feel "ok". I actually kind of feel like it's counteracting in my system. The only thing really holding me back is the fear of crazy side effects after you stop taking it. I heard you can even have a heart attack =/ Has anyone gone through this or going through it right now?