The idea seemingly derived from the commemorative golden spike that was temporarily driven at the completion of the U.S. transcontinental railroad in 1869.
Most often the myth is used as a practical joke or fool's errand played on junior sailors, exploiting their naivete and natural curiosity about their new surroundings.
After scouring the entire ship without success, it eventually dawns on the junior that he has been the butt of a joke.
The prank was also commonly employed as a means of homosexual seduction: the recruit accompanied to the bowels of the ship, and invited to bend over to see the mythical rivet better in the dark.
[1] Like many other hazing rituals, the "golden rivet" myth is perpetuated for the amusement of senior crewmembers at the expense of their naive and gullible junior crewmates.