She's right that people don't wear the tux as daily wear, but you can't dress down a tux very well. I wouldn't go around in the Prince Charlie and bow tie all the time either, but you can dress down the kilt.

Show her the pics on this board, especially those of casual kilt wearing. If nothing else, showing her this board will let her know that you're not the only "strange" one out there.