Once part of a row of late-eighteenth-century mansions, the building recalls the time when New York's merchant families lived at Manhattan's southern tip, near the river, in order to have an unobstructed harbor view and to be in close proximity to their shipping interests.

In order to create a unified facade, a colonnaded portico was added, reportedly using masts from his fleet of merchant ships which he was converting to steam power.

The architect of the eastern (original) half is unknown, but the western extension, next to the church, is attributed to John McComb, Jr.[4] With the exodus of wealthy families northward, the building housed the office of the Ithaca Line.

[7] On a visit to Queenstown, the main port of embarkation, Charlotte Grace O'Brien was appalled at the conditions faced by immigrant women, who encountered overcrowded, overpriced lodgings and robbery.

[7] O'Brien found little effort to provide food or drink or accommodation at the Castle Garden entry facility and illiterate young women being tricked into prostitution through spurious offers of employment.

Ireland also contacted Cardinal John McCloskey, Archbishop of New York, about providing a chaplain specifically for immigrants arriving at Castle Garden.

In 1885, the Watson House at 7 State Street was purchased from Isabella Wallace for the Mission of Our Lady of the Rosary to serve as a way station for young immigrant women.

It provided a home on State Street for 70,000 girls whose friends did not show up on the day of arrived, or who had no one expecting them, or who were unable to proceed on their journey.