I'm pretty lucky I guess; either due to my burliness of girth, or the inexplicable (but undeniable) ex-cop vibe I tend to give off, I hear very few disparaging remarks . . of course, running my own business in the town I grew up in also gives me an edge.
But I do indeed hear "the skirt remark", from time to time - sometimes droll, sometimes pleasantly baiting, sometimes exploratory.
And if they aren't trying to obviously irritate me, what I prefer for a response is: "It's a kilt, not a skirt; if I wore something under it, then it would be a skirt!"
That nearly always elicits nods & murmurs of amused enlightment, or outright, good-natured laughter, and a resulting sense of appeasement.
Occasionally, a male of equivalent size to myself will actually follow-up my response with a genuinely derogatory remark, and then I know that it's time to either begin assertively ignoring him, or go nose-to-nose for a little MYOB reminder . . all depending on the venue of the when & where.