Here again we have confusion with "Semi-Formal". Some people in tuxedos, some in suits and women in full length dresses... That is certainly a minefield to navigate and get it right!

That being the case, I rescind my previous advice about the sweater/jumper. If all the other men will be wearing jackets, I would think that you should as well.

Do I understand that you are in the band that will be performing that night? If so, you can remove the jacket for your performance and then put it back on whey you are done and cool down. But, I (personally, my style/choice) would not wear a vest in that circumstance without my jacket.