I think perhaps "You wear it well" is the same sort of thing as when you've been talking with someone for a couple of minutes and they say, "I didn't even notice at first that you were wearing the kilt." Some people might be self-conscious about wearing it that the kilt wears them, not the other way around.

Andrew.