MacMillan is right. Up until maybe 40-50 years ago, there was no concept of "casual" clothes. A person had work clothes and nice clothes. The only time you saw a person in work clothes is when they were either working or on their way to or from a job. They dressed up to go out in public.

People used to wear at least a jacket and tie just to go to the diner or to a ball game. And that's assuming they had any free time; many of the lower classes didn't. The only time you saw jeans on someone was when they were working at manual labor, or possibly a social outcast.

The main difference between the classes was that the commoners had the one set of good clothes and the well-to-do would have several of more expensive materials.