This is a common question that has been answered many times here.

First, a kilt IS a form of skirt, a MAN'S.

Second, solid colored Kilts are most easily confused with women's skirts by the uninformed.

Third, Some people are just ignorant, and choose to irritate you by calling it a skirt. (You have the confidence to wear it, they don't, so they try to make you uncomfortable!)

Wear it proud, don't let it bother you! You will find how many answers others have given here, and many work well.