Jokes and humor defuse tense situations. If you want to put your audience at ease, you make a light joke. What if you don’t want to put your audience at ease? What if unease and discomfort are specifically your aims? What if your goal is precisely to reframe something that’s normally dismissed as trivial?
Since women are expected to do the emotional work of making our interlocutors feel comfortable, when we fail to do it, our refusal seems particularly stark. When we care more about communicating our message clearly than communicating it comfortably, our refusal is part of the message.
When our interlocutors accuse us of being humorless, what they are saying is that their comfort is more important than our message.