[17] Ogilvie's first appearance back at Stevenage in his second loan spell came on 28 January 2017, in a 2–0 win over Grimsby Town.
[19] On 7 July 2017, Ogilvie signed a new two-year contract and was loaned to League One club Gillingham until January 2018.
[20] He made his Gillingham debut in the opening game of the season, in a 0–0 draw against Doncaster Rovers.
[23] After two loan spells at the club Ogilvie joined Gillingham on a permanent deal in May 2019 signing a two-year contract.
[27] On 2 August 2021, Ogilvie joined Gillingham's League One rivals Portsmouth on a free transfer, signing a two-year contract with the potential for a third.