Well, I get a little picky about putting a few types of brown with gray or with black., but thinking over my nicer clothing, I don't know that I have that much of that kind of brown.

These are the colors of my suits:
Navy blue,
Black,
Dark gray (that one has a waistcoat that I use with the other three suits),
and a gray/brown suit that has little bits of the two colors all through it.

Then I have three sport jackets: light gray, tan (with brown buttons), and medium brown.

I have a medium brown and a very dark brown kilt jacket.

Going by that, I could probably do alright starting with a gray fedora, then think about getting a brown one at some point.