Anne's brother Henry came down to Lisbon to bring her back to England.
However, on their way back, their ship was captured by a French frigate and the siblings were taken prisoners.
Anne remarried on 2 August 1799 to Charles Culling Smith, having two more children: On 25 July 1814 Lady Anne's daughter Georgiana Frederica FitzRoy (born 3 October 1792) married Henry Somerset, Marquess of Worcester, who had served the Duke of Wellington as an aide-de-camp during the Peninsular War.
On 29 June 1822 Lord Worcester married Lady Anne's other daughter Emily Frances (born 3 March 1800), greatly to the annoyance of Wellington, who was never close to his sister afterwards.
Lady Anne Culling Smith died on 16 December 1844, at Hampton Court.