I can only say that I wear my kilt because I want to, not in response to what others may think or say - good or ill.
In other words, I wear it for ME.
After over three years of wearing the kilt full time I can say that the delight has not worn off.
Friends and townsfolk are well used to seeing me kilted, and I consider that a bonus. There will always be idiots who snicker and make remarks, hey, that's human nature!

I wouldn't be concerned, you want YOU want when you want it, you are not harming anyone.