The prisons inspector called for "urgent" changes at one of the country's best-known jails.
Anne Owers found conditions at Wormwood Scrubs in west London had deteriorated since her last inspection. Her report revealed that one in five prisoners given mandatory drugs tests failed them.
Illegal drugs led to "gang activity" and "encouraged intimidation", she found. The prison was at the centre of a scandal over abuse of inmates by guards.
But Ms Owers said: "This inspection confirmed that there was no evidence of an abusive or negative staff culture. However, disappointingly, it also found that progress had been halted and, indeed, that there had been an appreciable drift in all our key areas - safety, respect, purposeful activity and resettlement. This is a direction of travel that needs urgently to be reversed."
The prison holds around 1,200 prisoners including nearly 500 foreign nationals.
Mark Leech, editor of the Prisoner's Handbook, said the 300 new prisoners arriving every month put Wormwood Scrubs under "enormous pressure".
"Those numbers create pressures that are impossible to deal with properly. We have neither the space nor the staff to cope with such numbers and it's inevitable that prisons like Wormwood Scrubs will start to fail in such circumstances.
"The solution is for the Government to now open temporary cellular accommodation in disused Army camps."
Liberal Democrat justice spokesman David Howarth said: "This report is a sad reflection of the Government's prison policy. Unfortunately the situation it describes is the norm rather than the exception.
"The Government should focus on measures that will actually reduce the prison population, such as removing prisoners with mental health and drug problems, and greater use of restorative justice."
