I do it all the time. Make sure your sporrans are on leather straps - not chains, and made so they can swivel around on the strap.

I put mine to the side when working, running etc. sometimes even round to the back!