Back when I was working a corporate job, that was the one thing I hated. Unfit people trying to talk fitness, diet etc..all while I was in their presence.
I'm getting another corporate job soon though, I have a feeling this time I will be better equipped at letting that go over my head and not caring...because it seriously used to drive me crazy.
I remember a month where I was going through a phase in my wardrobe and weeding out my more loose fitting shirts.. I recall something like this at work:
Me in a loose shirt, coworker comment: "oh you stopped working out, hey man..you used to look pretty big"
Me in a fitted shirt, coworker comment: "oh you started working out again, you get big so quickly"
next week me in a loose shirt, coworker comment: "oh you stopped working out again"
next day, me in a fitted shirt, coworker comment "oh wow, when you workout you grow too quickly, are you on steroids"
My advice is to just not let it even get to you, they don't know any better. Would a dentist waste his time getting irate over people talking about teeth, or how to maintain teeth? He/she would just let them be. That's how you have to be.