Active in the Anglo-Catholic wing of the Church of England and a member of the Alcuin Club, he energised the development of The English Hymnal (1906) and was chairman of its editorial board.

Riley's London house, at 2 Kensington Court, contained an altarpiece by Ninian Comper, a major designer of Anglo-Catholic church furnishings.

Later in his life, he moved to Jersey in the Channel Islands, where he purchased Trinity Manor in 1909, thereby acquiring the feudal title of Seigneur de La Trinité.

Finding the manor house in a ruined condition, he undertook an elaborate restoration (or "imaginative reconstruction", which has been criticized as turning the building into a French-style château[2]).

[3] Riley also bought the historic property L'Ancienneté in Saint Brélade, and removed architectural features of interest to incorporate into Trinity Manor, carefully recording the provenance of items and nature of alterations made in his project.