The native of Nashville, Tennessee, appeared in 268 games played over all or part of five seasons in Major League Baseball, from 1954 to 1958, for the New York Giants and Baltimore Orioles, mostly as a third baseman or shortstop.

[1] He was called up to the Giants during their 1954 World Series championship season on August 4 and, in his debut, he bounced into a double play as a pinch hitter off Paul Minner of the Chicago Cubs.

In 1955, Castleman made the Giants' roster out of spring training and hit his first two Major League home runs on May 4–5, but he got into only 15 games and was sent back to the Triple-A Minneapolis Millers in late May.

He succeeded Hank Thompson as the Giants' regular third baseman, hit 14 home runs (tied for third on the club) and started 97 games at the hot corner.

In March 1958, his contract was sold to the Orioles and he spent the entire season on Baltimore's roster as a reserve infielder, starting 64 games at shortstop, second only to Willy Miranda.