[2] Lilly's father and grandfather were both involved in farming and ranching, but the severe 1950s Texas drought forced his family to move at the end of his junior year at Throckmorton High School, where he received All-District honors in football.

As a sophomore with the Horned Frogs, he was a part of the team that won the Southwest Conference (SWC) championship, where he was mainly a backup (one start).

Lilly was a consensus All-American as a senior in 1960 and was a 2006 inductee to the East-West Shrine Game Hall of Fame.

His 29-yard sack of Dolphin quarterback Bob Griese (at the time an NFL record) is regarded as one of the most memorable plays in Super Bowl defensive history.

His greatest assets were his pass-rushing skills and his ability to slice plays open with his agility and instincts.

NFL Films did an eight-minute feature on Lilly and called him the "unblockable, unstoppable, force of the Doomsday Defense".

In the first play of the 1973 NFC Divisional playoff game three weeks later, on December 23 versus the Los Angeles Rams (Cowboys 27-Rams 16), he reinjured the same hamstring.

He entered the Hall of Fame along with former teammate Herb Adderley (for two seasons), as well as David "Deacon" Jones and Jim Otto.

Although the Cowboys do not have a practice to retire jersey numbers, Lilly is the only player to wear #74 in team history (with the exception of preseason games).

His interest in photography began when he was named to the College Football All-America Team in 1961, which was sponsored by Kodak.

Lilly spent two weeks in 1967 as part of the Texas National Guard at the Rhein-Main Air Base in West Germany.

After his retirement from professional football, Lilly moved to Waco, Texas, where he successfully operated a beer distribution business until 1982, when he saw the impact of a traffic accident caused by drunken driving and decided to sell the company and launch his landscape photography career.

[12] He lived in Las Cruces, New Mexico, from 1984 to 1989, where he opened a photo art gallery, complete with both color and black-and-white darkrooms.