David Mills in A Dictionary of British Place-Names gives the meaning of Irby as 'settlement or village of the Irish'.

[5] In the 11th-century Domesday Book, Irby's population of 11 villagers, 7 smallholders, 52 freemen, in over 70 households, was considered 'very large'.

[8] Holles was not an absentee landlord, spending much of his time in Irby, and the baptisms of at least five of his children are recorded in the parish registers.

[8] He made numerous improvements to the estate and was in the process of constructing a new manor house when he died in 1591.

[8] Writing in the 1600s, when the estate was owned by Denzel's son John Holles, 1st Earl of Clare, cousin Gervase Holles described it as:[9] ...a place happy in the sweetness of the air and very delightful by the pleasant hills and dales, where there are dry and inviting walks both summer and winter, with a welcome prospect towards the sea; affording as good hawking and hunting, and as good conveniency for training and airing young horses, as can be found anywhere.In 1840, Irby and the surrounding parish had 263 inhabitants and a notable local industry was chalk quarrying in the area of the Dale.