His parents emigrated to the United States when he was young and settled in Oneida County, New York.

After continuing to study law in Buffalo, New York, for a few years, he decided to relocate to St. Paul, Minnesota, in 1857.

He opened a law firm with his brother, Charles Duncan Gilfillan, though the American Civil War brought their partnership to an end.

When Chief Justice Thomas Wilson resigned from the Minnesota Supreme Court in 1869, Gilfillian was named his interim replacement him by Governor William Rainey Marshall.

When Chief Justice Samuel J. R. McMillan resigned in 1875, Gilfillian was again named as an interim replacement by Governor Cushman Kellogg Davis.