I don't see a problem with that. The Argyl is much like a sports jacket in many ways, being semi formal. Anything you can wear with a sports jacket should be fine.