I'd hesitate to wear a vest without a jacket. It'll look like you stepped away from the bar for a moment and need to get back. It's the look of someone who's currently on the clock.