Twist is measured by comparing the angle of a straight line between the leading edge (luff) and trailing edge (leech) with that of the boom.
[1] Some twist is desirable, but too much can induce weather helm or ruin the slot between the mainsail and jib.
Manipulation of the traveler can counteract this because it always pulls the boom laterally.
The vang on the other hand does the opposite, it always changes the height of the boom.
Less twist is generally desirable in light air conditions and slackening the luff tension will tighten the leech of the sail.