The centerpiece of the parcel is an 1854 Gothic Revival stone home known as Whitby Castle which was designed by American architect Alexander Jackson Davis.

The stone building which functions as the facility's clubhouse and houses a restaurant was originally intended as a residence for the Chapman family.

[5][6] The land currently occupied by the Rye Golf Club was originally purchased by the Chapman family of New York in the 1800s and used as a private residence.

The club was built on 150 acres of the former George Park and William Porter Allen estates overlooking Long Island Sound.

[9] Construction on the golf course moved slowly with some initial overall club projects like the lawn tennis courts and harborage for yachts deferred.

[11] In its early days, the Rye links drew famous people like Yankee great Babe Ruth in 1933 [12] and amateur golf champion George Von Elm.

[13] Other celebrities to hit the greens included golfers Ed "Porky" Oliver and Gene Sarazen who played an exhibition match there in the summer of 1940 as part of a 60-day tour sponsored by Golf Magazine.

[25] By 1959, the club's Board of Governors represented residents throughout Westchester including New Rochelle, Mount Vernon, and Port Chester.

[26] Annon was succeeded by 1952 National PGA Champion "Whistling Jim" Turnesa who was hired in 1959 with a January 1, 1960 start date.

The proposal, the goal of which was to expand public access to recreation,[30] was a topic of great debate and political discussion for many years starting as early as 1961.

Though the original purpose for the city's acquisition of the property was for the exclusive use of its citizens,[34] one no longer needs to be a Rye taxpayer to qualify for membership in the club and there is a separate non-resident category.

A new way to fill it with saltwater was solved by digging a 1,700 foot (0.52 km) pipeline from Long Island Sound to pump water in.

[38] Though the two sports are not reflected in the club's name, swimming and diving are activities that members have excelled at in competitions throughout Westchester County.