You were given some excellent comments to your post from the three ahead of me. I will try to add a few thoughts without being redundant.
I understand where you are coming from with wanting to remove people from your life that drain or manipulate you. And I would agree that since you feel so deeply, it probably is best to part ways with the manipulators as they are probably narcissists and can be damaging to yo. But if you find yourself parting ways with everyone you will soon find yourself pretty lonely.
I at many times over the years just wanted to be alone and found myself being irritated by just about everyone. Like you, I just felt that my job and relationships just drained me. There was just constant exhaustion and irritability. And I found it hard to find any joy in my life. It was not until I learned about grounding that I was able to get back to being my old self enjoying most of my relationships.
We empaths are sponges as we soak up everyone’s energies and take in the good, the bad, and the ugly. At a certain point you, the sponge have soaked up about as much as you can take. You're maxed out and feeling anxious, exhausted, irritable etc... That's when you know it is time to expel all that energy that you have absorbed so that you can feel better, feel normal, and be more comfortable around others. I'm not sure if you know about the subject of grounding. If you need some help with that please let me and others know.
I did want to say that some of my favorite people in my life can be a bit taxing at times and drain me. They aren't bad people but are very emotional and love to come to me to feel better by unloading their stories of struggle & drama. They feel better after being with me and I feel worse and tired. But as empaths that is what we do. I know I have to put a limit on how much time I spend with those types of people who need me to remove their "stuff". And I know when they leave me that I need to do some grounding to get me back to normal.