The Pentagon sent a U.S. veteran of the "dirty wars" in Central America to oversee sectarian police commando units in Iraq that set up secret detention and torture centers to get information from insurgents. These units conducted some of the worst acts of torture during the U.S. occupation and accelerated the country's descent into full-scale civil war.

Colonel James Steele was a 58-year-old retired special forces veteran when he was nominated by Donald Rumsfeld to help organize the paramilitaries in an attempt to quell a Sunni insurgency, an investigation by the Guardian and BBC Arabic shows.

After the Pentagon lifted a ban on Shia militias joining the security forces, the special police commando (SPC) membership was increasingly drawn from violent Shia groups such as the Badr brigades.

A second special adviser, retired Colonel James H Coffman, worked alongside Steele in detention centers that were set up with millions of dollars of US funding.

Coffman reported directly to General David Petraeus, sent to Iraq in June 2004 to organize and train the new Iraqi security forces. Steele, who was in Iraq from 2003 to 2005, and returned to the country in 2006, reported directly to Rumsfeld.

The allegations, made by US and Iraqi witnesses in the Guardian/BBC documentary, implicate US advisers for the first time in the human rights abuses committed by the commandos. It is also the first time that Petraeus – who last November was forced to resign as director of the CIA after a sex scandal – has been linked through an adviser to this abuse.

Coffman reported to Petraeus and described himself in an interview with the US military newspaper Stars and Stripes as Petraeus's "eyes and ears out on the ground" in Iraq.

"They worked hand in hand," said General Muntadher al-Samari, who worked with Steele and Coffman for a year while the commandos were being set up. "I never saw them apart in the 40 or 50 times I saw them inside the detention centers. They knew everything that was going on there ... the torture, the most horrible kinds of torture."

Additional Guardian reporting has confirmed more details of how the interrogation system worked. "Every single detention center would have its own interrogation committee," claimed Samari, talking for the first time in detail about the US role in the interrogation units.

"Each one was made up of an intelligence officer and eight interrogators. This committee will use all means of torture to make the detainee confess like using electricity or hanging him upside down, pulling out their nails, and beating them on sensitive parts."

There is no evidence that Steele or Coffman tortured prisoners themselves, only that they were sometimes present in the detention centers where torture took place and were involved in the processing of thousands of detainees.

The Guardian/BBC Arabic investigation was sparked by the release of classified US military logs on WikiLeaks that detailed hundreds of incidents where US soldiers came across tortured detainees in a network of detention centers run by the police commandos across Iraq. Private Bradley Manning, 25, is facing a prison sentence of up to 20 years after he pleaded guilty to leaking the documents.

Samari claimed that torture was routine in the SPC-controlled detention centers. "I remember a 14-year-old who was tied to one of the library's columns. And he was tied up, with his legs above his head. Tied up. His whole body was blue because of the impact of the cables with which he had been beaten."

Gilles Peress, a photographer, came across Steele when he was on assignment for the New York Times, visiting one of the commando centers in the same library, in Samarra. "We were in a room in the library interviewing Steele and I'm looking around I see blood everywhere."

The reporter Peter Maass was also there, working on the story with Peress. "And while this interview was going on with a Saudi jihadi with Jim Steele also in the room, there were these terrible screams, somebody shouting: 'Allah, Allah, Allah!' But it wasn't kind of religious ecstasy or something like that, these were screams of pain and terror."

The pattern in Iraq provides an eerie parallel to the well-documented human rights abuses committed by US-advised and funded paramilitary squads in Central America in the 1980s. Steele was head of a US team of special military advisers that trained units of El Salvador's security forces in counterinsurgency. Petraeus visited El Salvador in 1986 while Steele was there and became a major advocate of counterinsurgency methods.

Steele has not responded to any questions from the Guardian and BBC Arabic about his role in El Salvador or Iraq. He has in the past denied any involvement in torture and said publicly he is "opposed to human rights abuses." Coffman declined to comment.

An official speaking for Petraeus said: "During the course of his years in Iraq, General Petraeus did learn of allegations of Iraqi forces torturing detainees. In each incident, he shared information immediately with the US military chain of command, the US ambassador in Baghdad ... and the relevant Iraqi leaders."

The Guardian has learned that the SPC units' involvement with torture entered the popular consciousness in Iraq when some of their victims were paraded in front of a TV audience on a programme called "Terrorism In The Hands of Justice."

SPC detention centers bought video cameras, funded by the US military, which they used to film detainees for the show. When the show began to outrage the Iraqi public, Samari remembers being in the home of General Adnan Thabit – head of the special commandos – when a call came from Petraeus's office demanding that they stop showing tortured men on TV.

"General Petraeus's special translator, Sadi Othman, rang up to pass on a message from General Petraeus telling us not to show the prisoners on TV after they had been tortured," said Samari. "Then 20 minutes later we got a call from the Iraqi ministry of interior telling us the same thing, that General Petraeus didn't want the torture victims shown on TV."

Othman, who now lives in New York, confirmed that he made the phone call on behalf of Petraeus to the head of the SPC to ask him to stop showing the tortured prisoners. "But General Petraeus does not agree with torture," he added. "To suggest he does support torture is horseshit."

Thabit is dismissive of the idea that the Americans he dealt with were unaware of what the commandos were doing. "Until I left, the Americans knew about everything I did; they knew what was going on in the interrogations and they knew the detainees. Even some of the intelligence about the detainees came to us from them – they are lying."

Just before Petraeus and Steele left Iraq in September 2005, Jabr al-Solagh was appointed as the new minister of the interior. Under Solagh, who was closely associated with the violent Badr Brigades militia, allegations of torture and brutality by the commandos soared. It was also widely believed that the units had evolved into death squads.

The Guardian has learned that high-ranking Iraqis who worked with the US after the invasion warned Petraeus of the consequences of appointing Solagh but their pleas were ignored.

The long-term impact of funding and arming this paramilitary force was to unleash a deadly sectarian militia that terrorized the Sunni community and helped germinate a civil war that claimed tens of thousands of lives. At the height of that sectarian conflict, 3,000 bodies a month were strewn on the streets of Iraq.