I do my best to keep my 'dress up' and 'dressed out' kilts separate. The 'dressed out' (better) kilts I normally wear with shirt and tie, but otherwise wear with anything from a woolly jumper to the ocassional t-shirt. 'Dress up' kilts I wear with a fluffy shirt to Ren faires and Halloween parties, and even the back-woods cretins I'm usually surrounded by can see the difference.

I suppose, in my experience, it's more about what you wear *with* the kilt than that you wear the kilt.