I never get upset with people on this one.
I usually say something like "ah...that will just have to remain a mystery", and let it go.
They don't mean to be offensive I don't think. They're just curious because of all the stories and they speak before thinking.
I think if we fire back something in anger we're doing a lot more harm than necessary.

Or try this....just smile and wag your finger