He held the office, also in succession to his father, of acting librarian to the College of Physicians (Edinburgh), for which he prepared a catalogue in 1863.
Thereafter his chief labour was expended on editing, with careful glossaries and indices, the works of early Scottish poets, viz.
In 1885 he re-edited David Laing's Remains of Early Scottish Poetry, prefixing a bibliographical notice of his predecessor.
Small's nephew was Andrew Munro, (1869-1935), a Scottish fellow, lecturer in mathematics and bursar at Queens' College, Cambridge from 1893 to 1935.
In 1924, the estate of John Small's sister, Jemima, left £5,000 to establish a fund in the name of both her father and her brother for the purchase of books and to subsidize general purpose expenses within the library.