During its construction, numerous cave-ins and landslides occurred because of the fragile and weak rock the tunnel passed through, and at one point, an outbreak of cholera forced work to stop.

[13] Claudius Crozet was appointed as chief engineer of the Blue Ridge Railroad and developed a plan to cross the mountains using a series of four tunnels.

Although the eastern section of the ridge was composed of firm and sturdy rock,[18] the majority of the tunnel passed through a mixture of frail slate that was formed at a steep angle, soapstone, and clay.

[23] The weakness of the earth, especially in the western sections, necessitated the implementation of a strong arch throughout approximately 500 feet (150 m) of the tunnel and a durable brick lining.

In a letter to the Virginia Board of Public Works, Crozet states that the epidemic began at the eastern end of the tunnel where around 25 of the total 150 workers perished from the disease.

[28][29] As was done over Rockfish Gap to the west, a temporary track approximately 0.5 miles (0.8 km) long was proposed by the Virginia Central Railroad's Chief Engineer, Charles Ellet, Jr., and constructed around the Brookville Tunnel as the digging continued.

[37] In a November 1849 report, Crozet describes the design to be used on the tunnels: The shape I have adopted is that of an ellipse, formed by bisecting, on each side, the horizontal ordinates of a circle of 32 feet diameter, down to its center, below which the sides are to be made vertical, if no arch is necessary lower down; otherwise, the ellipse is to continue down to the bottom and to be shaped all round with bricks, stones, or iron, as may appear most eligible.

[40]Portions of the Brookville tunnel were lined with brick for added strength, with the brickwork originally contracted to Joseph Dettor, a local brickmaker.

[25] After examining the offers of multiple brickmakers,[43] the Virginia Board of Public Works contracted Robert Harris to provide the bricks for the Brookville Tunnel.

When the earth slip occurred during the Civil War, these remaining timbers acted as levers, furthering the cracking and structural instability of the tunnel.

[47] Costs at the Brookville Tunnel were greatly increased from the expected by the frail and insecure earth encountered, contrary to what originally appeared to be solid rock.