My Sheriffmuir is charcoal tweed, which definitely helps me to dress it down if I want to. I think it would be a bit more difficult to successfully dress it down if it were black, especially if it had bright chrome buttons. A navy doublet might be dressed down more easily.

This is about as dressed down as I have worn my Sheriffmuir.