Rigaud used his wide connections to recruit promising pupils whose fathers he knew.
Despite his many lasting achievements, Rigaud was unhappy at the school, which he subsidized from his own pocket.
Resentment at the cost of the new school persisted in the town, where there was also a feeling that he preferred boarders to day boys drawn from the local area, and matters came to a head when radicals in Ipswich brought an unsuccessful prosecution against him in April 1856 for punishing a day boy with undue severity.
In 1858 Rigaud was chosen Bishop of Antigua; he was consecrated on 2 February and went out to his diocese almost immediately.
A massive stone memorial marks his grave in the grounds of the cathedral there.