Whenever I get the "skirt" tossed at me I shrug it off. Attempting to correct the comment is useless, the people who call it a skirt know it's a kilt so reminding them is pointless.
Luckily it doesn't happen often, and it's easy to deduce the character of the person by the comment. Friendly jokers notwithstanding ;)