When everyone agrees that a remark was a joke, the teller and the listener laugh together. When everyone agrees that a remark was an attack, then the teller is gratified, the listener is hurt or angry, and they're quite likely to quarrel. The actions and perceptions are all congruent.
When the listener perceives an attack, however, it is the teller's response that makes the distinction. A teller who meant to share a joke, but picked the wrong one and upset their listener, will validate the listener's perception and backpedal. "Sorry, dude, I didn't realize that was a touchy topic and not funny for you. Next time I'll tell a different kind of joke." A teller who meant to attack will instead attempt to discredit the listener's perception and obscure the attack. "What's the matter with you, can't take a joke?" The actions and perceptions are muddled.
The first case, a misplaced joke, is relatively straightforward to solve even if it is awkward and embarrassing. Everyone who tells jokes will occasionally misplace one, just because humor is quirky, people have sore spots that aren't obvious, and social context is always changing. You mop up the mess and move on. Sometimes hidden under this area is the less socially graceful version of not apologizing but also not telling that kind of joke to that person again.
The second case is where it gets complicated and interesting. The attacker attempts to avoid recriminations by downplaying the attack. This can also happen if the remark was nasty but the teller didn't realize it, and after being alerted to the fact, also tries to evade responsibility instead of fixing it. But the distinction is easily made between jokes and attacks: a teller will apologize for a joke but defend an attack. Because the joke missed the goal, but the attack hit the goal, the teller's objective is different in those two cases. Therefore, the action of dismissing it as a joke disproves the attacker's own argument. Confirmation is found in the fact that many verbal attackers, having identified an effective attack, will use it repeatedly and aim to keep it just inside the bounds of what they can get away with, like serving a ball to bounce just inside the foul line.
This observation is useful in truncating the often-lengthy argument over what "is" or "is not" a joke. "If you meant it as a joke, when the listener didn't laugh, you would have apologized. Since you're not doing that, laughter was not the goal and your remark was not a joke." That's compelling because everyone knows that the social goal of a joke is laughter. If the teller claims that other people laughing makes it a joke, point out that if everyone but the target is laughing, that's not a joke, that's bullying.
When an argument is about the "isness" of things, proofs based on the generally known features of the thing -- rather than on personal interpretations -- can be very effective.