He played college football for the Kent State Golden Flashes and was signed by the Pittsburgh Steelers as an undrafted free agent in 2002.

Harrison did not pay attention to his grades or college entrance tests, and by his senior year he had become disruptive to his football team.

After his second TD, he ran down the opposing team's sideline, and was ejected and later suspended for one game for making obscene gestures to the fans.

Due to his off-the-field issues, powerhouse football programs like Ohio State, Notre Dame, and Nebraska rescinded their scholarship offers.

After graduating, he returned in 2010 to be inducted into Kent State's Varsity "K" Hall of Fame along with San Diego Chargers tight end, Antonio Gates.

[9] The Steelers signed Harrison as an undrafted rookie in 2002, making him the first Kent State alumnus to play at linebacker for the team since Hall of Famer Jack Lambert.

Harrison spent two years on and off the practice squad for the Steelers, being released three times, and also was briefly on the active roster towards the end of the 2002 season, playing only special teams.

Shortly thereafter, he was signed a fourth time by the Steelers during training camp in 2004 after Clark Haggans sustained an injury in an offseason weightlifting accident.

His biggest highlight of the year was in a game against the San Diego Chargers, where he intercepted a Drew Brees pass for a 25-yard return.

Harrison gained some attention and popularity when he restrained a Cleveland Browns fan during a 41–0 Pittsburgh win on Christmas Eve.

In the 2007 offseason, with longtime head coach Bill Cowher resigning after 15 seasons and Mike Tomlin taking over the reins, the Steelers controversially cut Joey Porter for salary cap reasons.

Although the Steelers drafted two linebackers with their first two picks that year (Lawrence Timmons and LaMarr Woodley), Harrison was appointed the starter in place of Porter.

The decision would ultimately pay off, as Harrison would go on to have a breakout season, making All-Pro Second-team and earning his first trip to the Pro Bowl as a starter on the AFC squad.

[19] On January 5, 2009, Harrison was named the AP NFL Defensive Player of the Year for the 2008 season, beating out Cowboys linebacker DeMarcus Ware for the award.

[21][22] During Super Bowl XLIII, Harrison made his most famous play when he intercepted a pass from Cardinals quarterback Kurt Warner at the goal line and ran back the length of the field for a 100-yard touchdown at the end of the first half.

In a memorable scene, Harrison collapsed in the endzone and spent several minutes regaining his breath as his teammates celebrated.

[26] The week before, Easterbrook named Harrison to his annual All-Unwanted All-Pros due to his struggles earlier in his career of having been cut four times, three by the Steelers.

[27] On February 7, 2009, Harrison was parodied on Saturday Night Live's Weekend Update by cast member Kenan Thompson.

Although Heath Miller, LaMarr Woodley, and Casey Hampton were named Pro Bowlers as well, Harrison was the only starter from his team.

[30] On October 17, he knocked out two Browns wide receivers; Mohammed Massaquoi, and his former college teammate, Josh Cribbs.

In Super Bowl XLV, Harrison recorded a sack on Aaron Rodgers, but lost 31–25 to the Green Bay Packers.

[40] On September 23, 2014, rumors surfaced that Harrison was set to return to the NFL after Mike Tomlin noted in a press conference that there were significant injuries suffered by members of the Steelers defense.

[46] On January 8, 2017, he recorded ten combined tackles, sacked Matt Moore 1½ times, and forced a fumble in a 30–12 AFC Wildcard victory over the Miami Dolphins.

Following injuries to Ryan Shazier and Tyler Matakevich and the subsequent move of Arthur Moats from the outside to the inside linebacker position, Harrison played 11 snaps in Week 14 against the Baltimore Ravens.

[59] Harrison elected not to visit the White House with the rest of the team after the Steelers won Super Bowl XLIII.

Skipping the visit gained some media attention, as Harrison said that Barack Obama (who supported the Steelers in the game and considers his second favorite NFL team after his hometown team, Chicago Bears)[60] would have invited the Arizona Cardinals had they won: "This is how I feel — if you want to see the Pittsburgh Steelers, invite us when we don't win the Super Bowl.

Harrison was arrested in March 2008 and was charged with simple assault and criminal mischief stemming from a domestic altercation with his girlfriend, Beth Tibbott.

On May 23, 2009, Harrison's son, James III, sustained an injury to his thigh when the family's pit bull became agitated and bit him.

[72] On July 13, 2011, a controversial magazine article titled "James Harrison: Confessions of an NFL Hitman" was published by Paul Solotaroff in Men's Journal.

Harrison also made comments towards his quarterback, Ben Roethlisberger, and other players in the National Football League such as Brian Cushing.