Learning disabilities treatment care is 'shocking'
Sir Stephen Bubb had recommended moving people to community care and a charter of rights, after the abuse scandal at Winterbourne View care home in 2011.
In his final report, he criticised lack of progress and said a commissioner should be appointed to drive reform.
The government is understood to be looking at all the recommendations.
In 2011, the BBC's Panorama uncovered serious patient abuse and neglect at the Winterbourne View private hospital, near Bristol.
Sir Stephen - who is chief executive of the Association of Chief Executives of Voluntary Organisations - was then asked by NHS England to examine how to address "serious shortcomings" in the support for those with learning disabilities.
In November 2014, Sir Stephen published his report - Winterbourne View - Time for Change - that said many people were being kept in hospitals far from home for far too long.
He made 10 recommendations, including closing large "inappropriate in-patient facilities" in favour of care services for people in their own community and the introduction of a legal charter of rights for them and their families.
But he later reported an "absence of any tangible progress".
'Advocate for change'
Speaking on BBC Radio 4's Today programme, as he published his final report on Monday, Sir Stephen said: "I have been really shocked by what I have heard about seclusion, around over-medication, around the use of physical restraint.
"It's an intolerable way to treat people with learning disabilities.
"The right place for people is with their families and in the community, supported properly."
But Sir Stephen said that what needed to be addressed was "not just around closing institutions", it was also about how society treated people in the education and health systems.
"That's why my major recommendation today is for the establishment of a commissioner for learning disabilities - someone who will drive and act as an advocate for change and for making more progress," he said.
The latest report, titled Time For Change - The Challenge Ahead, says some 3,500 vulnerable disabled people are still in institutions - 30%, or 900, more than had been thought.
It suggests that supporting people in their own community will require 10,000 extra staff, who will need to be trained to an established standard.
With at least 1,300 people expected to move out of hospital care by 2019, the report also says there is a "critical need" to develop housing for vulnerable people who, Sir Stephen says, should be exempt from proposed housing benefit caps.
'Too little progress'
The Department of Health is understood to be looking at all of Sir Stephen's recommendations, including the establishment of a commissioner.
Health minister Alistair Burt said: "It was only recently that NHS England announced a major programme to move people with learning disabilities out of hospital and into their communities...
"This, combined with the increase in specialist staff including nurses, will transform care.
"We are not complacent and will work with the NHS, local government and others to make sure their plan is delivered."
Luciana Berger, Labour's shadow minister for mental health, said there had been "too little progress" since the Winterbourne scandal.
"Tory ministers must avoid repeating their mistakes, accept these proposals and deliver the transformation in our learning disability services that is urgently required," she said.