I've been around quite a few people that were as you said; befuddled or confused by the whole kilt thing. Strange as it may sound to us, there seem to be a lot of folks who seem to have never seen a kilt before.
I met up with a young friend from out of town for lunch one time, and she had a girlfriend with her. Her friend kept looking at me like my junk was hanging out or something. This was a 21 year old college girl, and she had no idea what I was wearing, never saw one on TV, never seen a pipe band, not even Braveheart! We had to explain it to her.

Most folks of course know what it is, but have likely never seen one up close, so they ask some dumb questions. A good way to deflect some of it is to say something like "I wear it to express my Celtic heritage" most folks won't want to press on about heritage.

The bigger problem in my book is the folks that know a little too much about them