I have both, and typically wear them together... however, whenever I reach my desitnation, the jacket invariably comes off, as wearing both is typically warm and a wee bulky when indoors.

So, I suppose my vote would be to start with the vest, as i'd find that of more use.