This did not take up all of his time and he was granted permission to attend the drawing room at the Tower of London, where he trained under Clement Lempriere and John Peter Desmaretz and became a skilled military artist and plan-maker.
In 1745 he was taken on by Captain Thomas, Royal Engineers, a subordinate of the Duke of Cumberland, and together they repaired and extended Vilvorden Castle's fortifications – they produced a plan of the new scheme which is now in the British Library.
He returned to England the following year and was sent to Plymouth, from which he worked up the notes of his friend Kane William Horneck (died 1753) on the defences of Antigua after the latter's premature death.
They both fell from their positions ten years later – Smelt refused a pension, but did gain the post of Deputy Ranger of Richmond Park and remained an intimate of George and his wife Queen Charlotte.
His wife was a niece of lieutenant general Joshua Guest of Lydgate in Lightcliffe, Yorkshire – they had two daughters, Anne and Dorothy.