The Wall Street financial services firm said that its current chief executive Jamie Dimon would succeed Mr Harrison as chairman, combining it with his current role at the head of the company.
Mr Harrison has been an employee of the company for nearly 40 years, and played a part in the merger of JP Morgan with Bank One, from which Mr Dimon was recruited.
Commenting on his imminent departure, Mr Harrison said that he was optimistic about the future of JP Morgan.
"I am very proud of the leading role we have played in the transformation of global banking and the creation of our world-class franchise," he said.
"I could not be more excited about the future of our firm and the leadership that Jamie Dimon is providing."
Mr Harrison will also retire in his role as director of the bank before the new year.