I've swum a few times in a PV kilt. I'd definitely recommend taking off the kilt while you swim. It'll just get in the way and weigh you down, as others have said. I've only swum short distances, but I wouldn't want to swim more than a few hundred yards in it.

Andrew.