Personally, I would only wear a fly plaid with evening wear.

For example, I wear it with my Prince Charlie jacket, but not with my Argyll, as seen below.



I guess I'd never wear it without a jacket.