It was established in 1811 as the first prison in the state and the second of its kind in the country and the original buildings faced towards East Madison Street above the east bank of the Jones Falls stream and adjacent to the old stone walls of the Baltimore City Jail (now renamed the Baltimore City Detention Center), earlier established in 1801, rebuilt in 1857–1859, and later in 1959–1965.

Before its opening, convicted criminals were put in county jails or a workhouse where they were employed in public projects such as road building.

Compared with other prisons, convicts were treated reasonably well and were kept in hygienic conditions with an ample supply of food.

During the 1800s, the prison underwent several construction phases, with a massive granite stone central tower with the landmark pyramid roof, including a new western and south wing in the late 1890s facing Greenmount Avenue towards the south and along Forest Street to the west.

Part of this modern (late 19th Century) construction included new cells for solitary confinement at night.

He initiated many changes for the Penitentiary including re-building the majority of the prison and adding several new buildings.

New cells were larger, lit with electric lights, better ventilated, had sliding steel doors and, for the first time ever, flushable enamel toilets.

During his time in control, Weyler had a reputation for being a good and fair warden, but this image was completely destroyed with the publication of the "Report of Maryland Penitentiary Penal Commission" of 1913 "charging his administration with mismanagement, cruelty and corruption."

He was appalled at the methods of punishment including "chaining," which involved hand cuffing inmates and hanging them by the wrists slightly above the ground.

He was found guilty of stealing up to three thousand pounds of bread crumbs a week to feed the animals on his farm.

Eventually Weyler retired to avoid the outcome of the commission and thus he left with no legal charges.

The wardens who followed included Patrick J. Brady and Edwin T. Swenson, who ran the prison for long, stable periods of time.

It was decided that the existing warden wasn't capable of maintaining order among the inmates which led to the rule of army colonel, Claude B. Sweezey.

Additional facilities were constructed with a stone castle-like structure as another "penal farm" for men in Hagerstown in the early 1930s, which became the Maryland Correctional Institution.

It occurred when inmate John E. Jones fought the correctional officers escorting him back to his cell.

Inmate Lascell Gallop was working in the kitchen and threw an acid solution in an officer's face.