So what's the best way to get rid of rumors you are doing some work?

[rah]

I worked one place as a middle manager where it was expected that you attended meetings all day exclusively.
And half of your meeting time was agreeing on when the next meeting would be. The most screwed up company that I ever worked at. Even with a 6 digit salary and large bonuses, I couldn't take it anymore and left after a year.