Can I suggest that it evolved from the natural amusements of chaps engaged in hard manual work, with a link to the actuality of that work: which in turn evolved into formal sports with actual rules.
Today there are competitions for throwing bales of hay from pitchforks-natural for a farm worker to show off their physical prowess.
Or skills such as archery, and shooting.
Hammers-rocks, I think obvious.
So I can imagine chaps working with timber competing to see who could throw a log futhest-this evolving into caber tossing.
Bookmarks