Although the combination looks like it would work, I would consider bringing a spare shirt simply because you'll be seen drinking whisky in a shirt that clearly associates you (and thus the activity of drinking whisky), with your employer.

Consider a police officer... he'd be required to wear his uniform while on duty. However, what would the public's perception be if he were then seen when off duty, drinking whisky while still in his uniform? Dressed as he is, he's not seen as a private citizen enjoying a cocktail, but as a police officer who is (assumed to be) drinking on duty.