The size of this delivery led to the enlargement of the Ayer Itam gardens and Residency grounds; as a result, Smith was instructed by Lieutenant Governor Leith to return to supervise this enterprise and cease collecting further specimens in the region.

[3] Captain James Low (writing in 1836) described the Gardens during this period as: "... embracing one hundred and thirty acres of land, lying on the slopes which skirt the base of the hill near Amie's Mills, a romantic spot and well watered by a running stream now called Ayer Putih.

"[3] Following further collecting trips, Smith returned to George Town in 1805, accompanied by 71,266 nutmeg and 55,263 clove plants together with additional canary nut and sugar palm specimens.

By now, George Town had been elevated to the status of India's fourth presidency, and Lieutenant Governor Leith had been succeeded in 1803 by Colonel Robert T. Farquhar.

Raffles' friend, the Danish naturalist Dr. Nathaniel Wallich, recommended the appointment of Penang Free School headmaster and amateur botanist, George Porter, on a salary of $100, to manage these gardens.

[6] Located at 579 m, with an average annual rainfall in the 1890s of 381 cm, the site of the Botanic Gardens embraces over 29 ha, comprising a significant middle portion, of the Waterfall River valley below the Waterfall which cascades from a height of over 120 m. The heavy rainfall often resulted in management problems, and land slips in the "steepest part of the grounds".

[6] The third Botanic Gardens were eventually sited in the Waterfall River valley on land acquired by the colonial government in 1884–85 for this purpose.

[6] On his appointment as the first superintendent of the Botanic Garden, Curtis was presented with a tropical valley, including a nutmeg plantation with associated structures, and a prominent location on the trail to and at the foot of the "Great Waterfall".

[6] On accepting the position, Curtis proposed a long-term strategy as to the development of the gardens and its potential role as a botanical repository and clearing house.

[7] Curtis' immediate actions were to develop a plant nursery and undertake a programme of works to create a pleasurable recreational and botanical garden in the valley.

This included proposals to extend and develop the existing "Waterfall Gardens", the construction of road circuits, the erection of plant-houses for the propagation and cultivation of various species, and the provision of recreational venues.

In his 1885 report, Curtis commented on "the poor gravelly soil in the valley" which required that considerable attention should be given to the preparation of the ground for tree planting.

The circular road circuits he had constructed carefully weave through the valley opening up views, framing vantage points, and providing surprises to the visitor.

Curtis' design was motivated by his objective to take advantage of and exploit the natural landscape in the first instance, and then locate plantings in functional or species family associations.

In March 1903, health problems caused him to take early long service leave and he formally retired on 7 December 1903, to be replaced by Walter Fox.

Frederick Flippance, who was appointed assistant supervisor in 1921, was faced with the formidable task of relaying the gardens, in addition to his advisory role on roadside trees and other planting work in the municipal area.

The Japanese excavated tunnels near the lily ponds and turned them into ammunition works and storage facilities,[4] and a torpedo assembly station.

[12] There is also the pinang palm (Areca catechu) that lent its name to the island of Penang, and the black lily (Tacca integrifolia)[12] with its unique purplish-black coloured flowers.

The candle tree (Parmentiera cereifera),[11] the endemic slipper orchid (Paphiopedilum barbatum)[12] and the ginger, Geoctachys penangensis,[12] can also be found in the gardens.

[12] The garden fauna include long-tailed macaques, dusky leaf monkeys, black giant squirrels as well as many insects and butterflies.

[12] The path around the Lily Pond offers access to tropical rainforest, a short distance from the Botanic Gardens gate.

[9] The waterfall and reservoir are private property although they can be visited by arrangement with the Penang Water Authority (Perbadanan Bekalan Air Pulau Pinang).