I too need time to recharge my batteries. I watch my 4 grand children during the week, so by Friday afternoon, I'm ready for some "me" time. I love Saturdays, doing what I want, but by Sunday, I usually start feeling alone. I have learned this of myself-I enjoy time alone, but for about 1 1/2 days.
I don't live in your shoes, so I don't know if it is fear based that your counselor is talking about or just energy overload. If it is fear based, then yes, face your fears and just do it. But if it is your Empath feelings saying that every time you go there, you end up on energy overload, then listen to your body. Going to the mall is something I NEVER want to do-I'm not afraid to go, but I am afraid of all the energy that I end up picking up.
Bottom line-get to know yourself. Know your limits, what makes you happy, how much time you need to be with self to be happy, know the difference between fear "based on phobias" and "fear based on the knowledge that it will end in an energy surplus." Protect yourself from energy overload.
Have you taken the free empath training course on this site? It helps you to learn about shielding and how to protect your body from energy overload. It is a great help to your sanity.
Sending you lots of healing love.....