If it looks good pin it on, if not than don't. You live in the US so do what looks best. I have seen a lot of people wear their casual kilts both ways.