I have trouble with that stuff, too. It's very hard for me to be around rude, mean or harsh people. I bet he did psychic attack on you. It's like a whole other level of being mean. It attacks your spirit and soul and sort of sticks there until you can get it out.
There are a few things you can do. You can call on god and angels or whoever you believe in, and demand that they remove it from all the cells of your body. Or you can go inside and imagine the bad energy he put in you has a form. See what it looks like, a black slime, a dark chunk of black concrete, whatever you see. Then you can imagine gathering it all up and tossing it forcefully out of your body and soul and aura, like taking out the garbage.
I struggle with this, too. There are certain people in my life who do this to me from time to time as a method of punishing me for not giving myself to them to control. There is also a person I work with who is a very unhappy person and does this to me sometimes. I go home from work and have nightmares and all that. I've been reading and practicing the book The Most Important Person in the Room to help me learn some tools to not take on that stuff so much, and it's helping. The truth is, just as there are light workers in this world, there are dark workers as well to create a balance. He was probably a dark person. Just as a lot of people here are healers and want people to feel happy and send positive healing energy almost without knowing it, some people do the opposite and send bad energy. I bet he was one of those.
I hope this advise helps. That bad stuff doesn't belong in your body. I hope you are able to remove it soon and feel whole and calm and learn how to receive less of that, because it's not yours and it doesn't belong in you. Hugs! Good luck!