From the point of view of someone who was a student not all that long ago (at least it doesn't seem that long ago), who has done a little bureaucratic university work, and who knows professors and others who work in academia, it may be the environment. When I was in school, most people I knew were constantly stressed and trying to find happiness in unhealthy ways. While drinking, frequent casual sex, and some drug use are probably normal and possibly healthy to some degree, constantly trying to escape feelings of anxiety, inadequacy, etc combined with a lack of meaningful social connections (thanks age of social media) is normal, but certainly not emotionally healthy. For perspective, I had what is considered to be a difficult major. Out of the 20 or so people in my year with that major, at least 3/4 of them took illegal drugs to function. Probably 4 or 5 had any sort of meaningful romantic relationship during those 4 years, and most people's idea of fun was just going to a bar and getting blacked-out drunk (fine on occasion, but have some variety in your life and don't use it to deal with life). I've even had roommates threaten to kill themselves because they were so stressed out. The people in the "easier" majors didn't seem much happier. Most people told me that the were happy, but they just weren't; I could feel it.
Although I suppose it depends on exactly where you were working, my experience is that school's funding is being slashed and staff members are not exactly feeling appreciated, and sometimes not as if their job is safe. Unappreciated and scared people get grumpy (to put it mildly) and take it out on others. Plus, they give off a lot of negative feelings.
Between less grant funding, fewer tenured positions, and schools hiring part time adjunct faculty to teach classes to save money, faculty is feeling pretty grumpy in general as well.
So, yes, I'd say that the college environment may certainly be a more depressing place to work than many others, particularly for an empath.