I don't know if there is a bigger double-edged sword in comedy than the fact that, in general, the audience is vehemently discouraged from speaking while the comic is performing. I actually talked about it once before in a column about "victims" whom we should stop feeling sorry for, in which I brought up the Daniel Tosh rape joke controversy and pointed out that, had the audience member in question adhered to accepted comedy club etiquette, that incident never would have happened.
I still stand by that, but it's worth noting that in that same column, I made it clear that I thought the joke itself was indeed pretty awful. I'm not alone there -- even Tosh himself apologized eventually, and that's precisely why the controversy didn't destroy his career.
Comics take the stage with an inherent advantage over the audience, in that out of everyone in the room, only the person with the microphone is allowed to speak. It's a rule; talk too much while in the audience, and you run the very real risk of being kicked out.
But here's the thing: That's not a rule intended to uphold free speech. It's meant to ensure that the other people in the crowd who paid money to be there don't have their night ruined by some cackling drunkard. What it's not meant to be is a license for comics to drop all standards of human decency when addressing what, in the big scheme of things, is a minor nuisance. Comics like to use the analogy of a person who inevitably gets attacked by the animal they're taunting at the zoo. You brought it on yourself, you know?