I'd personally not wear it with the charlie. I wear a crested polo shirt if I'm in slacks or jeans, but when kilted I tend to stay away from crested jackets etc, as my own theory regarding this is that I have the clan tartan on, a crested kilt pin, one of my sgians is crested, (& when required, so is my plaid brooch) & there's more than enough there to show my affiliation to those who need to know. In fact, at mum's funeral the piper came around the corner, saw us all gathered on the street, & called "A Gordon!" He picked us from a good 200 yards away. (He was a Gordon himself.)