He was the only son of Eleanor (née Carter) Anderson and Robert Vyner, MP for Okehampton, Lincoln, and Thirsk.
From her first marriage, he had at least two half-siblings, including Charles Anderson-Pelham, 1st Baron Yarborough and Francis Evelyn Anderson who both served in Parliament.
His grandfather, who represented Great Grimsby and Lincolnshire in Parliament, inherited Swakeleys and his extensive Lincolnshire estates in 1707 (which had been acquired by his father's great uncle, Sir Robert Vyner, 1st Baronet, who was Charles IIs banker).
He entered Parliament for Lincolnshire in 1794, on the vacancy caused by the elevation of his half-brother Charles Anderson-Pelham to the peerage as Baron Yarborough.
After Robert's death in 1872, the estates passed to his grandson, Henry Frederick Clare Vyner.