Too much "me" time?

So recently the hubs and I have had a few "discussions" about how I should balance my time.  My job as a Catering Sales Manager can be pretty time consuming as it is, which leaves little room for other activities aside from family time...even on some weekends.  The time at work alone has already caused some tension, but hey, if you want me to help provide for the family, then don't complain about my job --- let's face it, I'm not stripping, I'm not doing anything illegal, and I'm actually doing something I enjoy, so what gives with the attitude?

Anywho, because my job has been a little bit more demanding lately, it makes it that much harder for me to enjoy a happy hour here and there or even go to the gym without hearing "Why aren't you coming home to your family?".  I don't get it... I am not trying to avoid my family by any means...I LOVE being with my family, but if all I did was bust my as catering (literally) to my clients at work and then catering to my family, I would go insane.  Even the world's best moms & supermom's need their "me" time...So how much "me" time is allowed?

Just recently a younger coworker asked if I would go to Happy Hour with her and I knew that unless I was leaving work early, I would not be able to justify coming home late again to the hubs (and by late, I mean an hour later).  This coworker is like a little sister and is actually our babysitter, so I feel really bad that she's practically been begging me to hang out with her outside of work and outside of seeing me before I leave the house.  When I asked Commander Hubs if it would be OK...his response was, "Party Time is for the Weekends". OK - I understand that, but I'm not trying to party, I'm going to have a drink with a colleague...

To make it even harder, in my profession, networking is everything...it really isn't what you know, it is who you know and who knows you, so there are times when WORK calls for me to go to a networking event, but to Comm. Hubs, I'm out partying.  So it has gotten to the point where when I truly just want to enjoy a glass of wine or two with a friend and catch up, I feel like I'm neglecting my family and choosing them over my duties as a mother and a wife...is that the case? I truly am conflicted because on one hand, I know that I need my girl time (none of my friends have kids, so play dates are not an option and I would like to have girltime away from my husband and son running around), but on the other I feel like I do not have enough time to balance it all...

Superwoman powers would be greatly appreciate...kthanks.