If/when I wear military-issue garb, or stuff that looks military-issue (e.g. surplus), it's usually from the generation before I was eligible to serve (i.e. OD greens, not woodland camo, and I don't even have and wouldn't think of getting any current issue or desert/digital camo). I also usually don't wear those things out in public and don't wear them in a military style (i.e. I'll leave shirts unbuttoned/untucked, and p*nt legs unbloused), I don't wear them in combination (I'll wear just a shirt, or just the pants, or just a jacket, but none of them together) and most of the time they're not ironed, so they're even less obviously military.

I do have an O.D. M-65 Army field jacket of my uncle's that I put my Dad's unit patches and name tape on (the right shoulder has the 83rd "OHIO" Infantry Division and the left has the 5th Army, which I don't think match up for that era - Dad was in the regular Army reserves for 8 years then transferred to the KANG). That's the closest I come to looking even close to military. When people do ask or make a comment I'll explain that "this was my uncle's/father's".

Most of the time, I'm wearing the mil surplus stuff when I go camping or working around the house or in the yard. It's more durable than most of my other clothes, and I'm not as concerned about getting stains and stuff on most of it, since it's already stained/torn from when they wore it to do yard work, etc.