Yu Garden was first conceived in 1559 during the Ming Dynasty by Pan Yunduan as a comfort for his father, the minister Pan En, in his old age.Pan Yunduan began the project after failing one of the imperial exams, but his appointment as governor of Sichuan postponed construction for nearly twenty years until 1577.[6] The garden was the largest and most prestigious of its era in Shanghai, but eventually its expense helped ruin the Pans.

The garden was inherited by Zhang Zhaolin, Pan Yunduan's granddaughter's husband, and then passed to different owners. A section was briefly organized by Zhang Shengqu as the "Academy of Purity and Harmony" and the Ling Yuan, today's East Garden, was purchased by a group of local leaders in 1709. A group of merchants renovated the increasingly decrepit grounds in 1760 and in 1780 the West Garden was opened to the general public.

The gardens suffered damage numerous times during the 19th century. During the First Opium War,the British army used the Huxinting Teahouse as a base of operations for several days in 1842. During the Taiping Rebellion, the Small Swords Society ran its headquarters in the Dianchun Hall; by the time Qing troops recovered the garden, the original structures had nearly all been destroyed. They were damaged again by the Japanese in 1942 before being repaired by the Shanghai government from 1956 to 1961. They were opened to the public in 1961 and declared a national monument in 1982.