In 1849, at the age of twenty-nine, Manly joined the California Gold Rush, traveling overland from Wisconsin.

Manly and his associate John Haney Rogers trekked 250 miles on foot across the Mojave to Rancho San Fernando near Los Angeles, California to scout an evacuation route for the families trapped in Death Valley.

There they procured food and horses from Mexican villagers and were able to save their party by leading them to Rancho San Francisco.

[3] Manly worked the gold fields for several months, then returned to his farm in Wisconsin via steamboat, crossing the isthmus of Panama overland.

In 1851, Manly returned to California by the same method, arriving in San Francisco shortly after the fire of 1851 and continued in the goldfields until the fall of 1859, by which time he had saved enough money to buy land 250 acres (1.0 km2) in the Communications Hill, San Jose area,[4] paying $16 an acre, $4,000 in all, where he planted a farm.

By this time, his diary was "lost" (probably also burned, perhaps purposely) but he recreated events from memory and in 1886, Manly published "From Vermont to California" in Santa Clara Valley, a monthly agricultural review.