Of course you can wear your fedora. Some people won't like it, but those folks are not always representative of everyone. Go with what works for your personal style.