The Armed Forces Retirement Home was established as the northern branch of the United States Military Asylum in 1851.

Four American Presidents, James Buchanan, Abraham Lincoln, Rutherford B. Hayes and Chester A. Arthur, summered at the home.

While no fighting took place on the site during the American Civil War it was used as a signal post.

Development of the asylum began in 1852 when the Riggs home was enlarged and the placement of a flagstaff, which marks the establishment of a military installation.

The grounds also include recreational areas, statuary, war relics, and other features.