Northern Zhou finally allowed him to return to Chen in 560, but as he wrote impolite letters to Emperor Wen, Emperor Wen felt threatened (as he viewed the letters as implied demands for the throne), and he sent his trusted general Hou Andu to escort Chen Chang.
When Chen Baxian was invited by Xiao Ying (蕭映) the Marquess of Xinyu, who was also the governor of Guang Province (廣州, modern Guangdong), to serve on Xiao Ying's staff, around 540, it appeared that both Lady Zhang and Chen Chang accompanied him to Guang Province, but when he was subsequently commissioned in 544 to campaign against the rebel Li Ben in modern northern Vietnam, he sent them back to his home commandery of Wuxing (吳興, roughly modern Huzhou, Zhejiang).
Chen Baxian sent the officials Xie Zhe (謝哲) and Cai Jingli (蔡景歷) to assist him in governance, and the scholar Du Zhiwei (杜之偉) to teach him in his studies.
In 554, Western Wei forces attacked Jiangling and captured it, and around the new year 555, they put Emperor Yuan to death.
However, because the Liang general Wang Lin, who had by that point supported the Liang prince Xiao Zhuang as emperor, controlled the parts of Yangtze River that Chen Chang was required to travel on to reach Jiankang, Chen Chang was unable to proceed much on his journey, and he had to stop at Anlu (安陸, in modern Xiaogan, Hubei).
Emperor Wen summoned his general Hou Andu, suggesting that perhaps he should yield the throne to Chen Chang and accept a princely title.