I seem to recall this question generating a LOT of heat a number of months ago. Someone posted about smacking a lady's hand away, and the chivalry police attacked.

Smacking away a person's hand seems reasonable enough, though.
Mind you, how we react to people messing with us will identify us.