While at Lydd Fisher associated with some Anabaptists, attending their meetings and offering them the use of his pulpit, in which he was stopped by the churchwardens.
In 1654 William Coton and John Stubbs, while on a visit to Lydd, stayed at Fisher's house, and convinced him of the truth of quakerism.
On 17 September 1656 Fisher attended the meeting of Parliament, and when Oliver Cromwell stated that to his knowledge no man in England had suffered imprisonment unjustly, he attempted a reply.
He was active in Kent, where according to Joseph Besse he was roughly handled in 1658, and in 1659 he was pulled out of a meeting at Westminster by his hair and beaten.
During the following year Fisher and Stubbs made a journey to Rome, travelling over the Alps on foot, where they testified to several of the cardinals, and distributed copies of Quaker literature.
Anthony Wood states that when Fisher returned, he was well dressed; suspected of being a Jesuit and in receipt of a pension from the Pope, he was imprisoned and he seems to have undergone some further persecution.
Shortly after his discharge he was again arrested at Charlwood, Surrey, and committed to the White Lion Prison, Southwark, where he was confined for about two years.
The Rusticks Alarm to the Rabbies, or the Country correcting the University and Clergy (1660) is, according to Christopher Hill, "a remarkable work of popular Biblical criticism, based on real scholarship", in which Fisher "virtually abandoned any hope of unity of interpretation, and so of any external unity [of the church].