The Pavilion of Prince Teng was first built in 653 AD, by Li Yuanying, the younger brother of Emperor Taizong and uncle of Emperor Gaozong of Tang. Li Yuanying was enfeoffed as Prince Teng in 639 and spent his early years in Suzhou. In 652 he was assigned the governorship of Nanchang where the pavilion served as his townhouse. The Pavilion of Prince Teng is the only existing royal architecture in southern China. Twenty years later, the building was rebuilt by the new governor. Upon its completion, a group of local intelligentsia gathered to compose prose and poetry about the building. The most famous of these is the Preface to the Pavilion of Prince Teng by Wang Bo. This piece made the Pavilion of Prince Teng a household name in China down to the present day.

In 2013, Jiangxi province launched the promotional campaign to attract more tourists during the golden week. Visitors who can recite the entire masterpiece within 10 minutes are given free admission.

Along with the Tengwang Pavilion and the Yellow Crane Tower, Yueyang Tower is one of the "Three Great Towers" of China.