Matt...

I think the buttons may have something to do with it.

Immitation Stag buttons (for those of you who don't know, they're pretty much standard on tweed jackets) lend themselves, IMHO, to daywear. If you switched the buttons to plain BLACK buttons or to the square chrome buttons of an Argyll / PC, it might dress it up a little.

Then again, chrome buttons might just look 'odd' and take away the VERSAILITY of wearing the jacket for daywear to a degree. Thoughts?

Perhaps this combo with plain black buttons would be up your alley: