If you go back in history to the days before everything was mass-produced, a person would wear whatever he/she could make, buy or barter.
You have the option of accessorinzing with everything matching perfectly. You also have the option of mixing things up, and no one will fault you for it, unless, of course, there are no pictures.