During his lifetime, Dudley held the posts of Commissioner of Revenue, Judge of the Court of Admiralty, Master in Chancery, representative for Kildare & Wicklow in the Protectorate Parliament and MP for Naas, Bannow and Fethard (Co Wexford).

The young man lost his money in the South Sea Bubble and in 1723 the castle was sold for £62,000 to the Right Hon William Conolly, speaker of the Irish House of Commons.

In 1767, Boyle sold his interest in the property to Nicholas Hume-Loftus, second Earl of Ely, a descendant of Adam Loftus, the original builder of the castle; a small head rent to the Conolly family remained.

[5] Nicholas died within a few years, probably as an indirect result of great hardships which he had suffered in his youth, and the estate passed to his uncle, Hon.

Henry died in 1783 and was succeeded by his nephew Charles Tottenham, who subsequently became Marquess of Ely as a reward for his vote at the time of the Union.

To quote a contemporary account from 1838:[5] Crossing the Dodder by a ford, and proceeding along its southern bank towards Rathfarnham, a splendid gateway at left, accounted among the best productions of that species of architecture in Ireland, invites the tourist to explore the once beautiful grounds of Rathfarnham Castle, but they are now all eloquently waste, the undulating hills covered with rank herbage, the rivulet stagnant and sedgy, the walks scarce traceable, the ice-houses open to the prying sun, the fish-pond clogged with weeds, while the mouldering architecture of the castle, and the crumbling, unsightly offices in its immediate vicinity,…The castle, so long the residence of the Loftus family, and still the property of the Marquis of Ely, subject, however, to a small chief rent to Mr. Conolly, is an extensive fabric,.....The great hall is entered from a terrace, by a portico of eight Doric columns, which support a dome, painted in fresco with the signs of the Zodiac and other devices.

This room was ornamented with antique and modern busts, placed on pedestals of variegated marble, and has three windows of stained gloss, in one of which is an escutcheon of the Loftus arms, with quarterings finely executed.

Several other apartments exhibited considerable splendour of arrangement, and contained, until lately, numerous family portraits, and a valuable collection of paintings by ancient masters.

But, when it is mentioned, that this structure has been for years a public dairy, and the grounds to the extent of 300 acres (1.2 km2) converted to its uses, some notion may be formed of their altered condition.

In 1986, the Jesuits sold Rathfarnham Castle but before leaving, they removed the stained glass windows, designed by Harry Clarke, from the chapel and donated some to Tullamore Catholic Church, which had been destroyed by fire in 1983.

The castle consisted of a square building four stories high with a projecting tower at each corner, the walls of which were an average of 5 feet (1.5 m) thick.

In commemoration of regaining ownership and of Henry being made Earl of Ely in 1771, the Loftus family constructed another entrance for the castle in the form of a Roman Triumphal Arch.

[9] Not far from the Golf Club's clubhouse was a small Palladian temple with a surrounding colonnade built of granite and brick, another relic of Lord Ely's occupation of Rathfarnham and likely constructed in the last quarter of the 18th century.