When developing "evidence-based" guidelines, the World Health Organisation routinely forgets one key ingredient: evidence. That is the verdict from a study published in The Lancet online, Tuesday, 8 May 2007.

The medical journal's criticism of WHO could shock many in the global health community, as one of WHO's main jobs is to produce guidelines on everything from fighting the spread of bird flu and malaria control to enacting anti-tobacco legislation.

The study was conducted by Dr Andrew Oxman and Dr Atle Fretheim, of the Norwegian Knowledge Centre for Health Services, and Dr John Lavis at McMaster University in Canada. They interviewed senior WHO officials and analysed various guidelines to determine how they were produced. What they found was a distinctly non-transparent process.

"It's difficult to judge how much confidence you can have in WHO guidelines if you're not told how they were developed," Oxman said. "In that case, you're left with blind trust."

WHO issues about 200 sets of recommendations every year, acting as a public health arbiter to the global community by sifting through competing scientific theories and studies to put forth the best policies.

WHO's Director of Research Policy Dr  Tikki Pang said that some of his WHO colleagues were shocked by The Lancet's study, but he acknowledged the criticism had merit, and explained that time pressures and a lack of both information and money sometimes compromised WHO work.

"We know our credibility is at stake," Pang said, "and we are now going to get our act together."

WHO officials also noted that, in many cases, evidence simply did not exist. Data from developing countries are patchy at best, and in an outbreak, information changes as the crisis unfolds.

To address the problem, they said, WHO is trying to develop new ways to collect information in poor regions, and has proposed establishing a committee to oversee the issuance of all health guidelines.

The Lancet study — conducted in 2003-2004 through analysing WHO guidelines and questioning WHO officials — also found that the officials themselves were concerned about the agency's methods.

One unnamed WHO director was quoted in the study as saying: "I would have liked to have had more evidence to base recommendations on." Another said: "We never had the evidence base well-documented."

Pang said that, while some guidelines might be suspect and based on just a few expert opinions, others were developed under rigorous study and so were more reliable.

For example, WHO's recent advice on treating bird flu patients was developed under tight scrutiny.

Oxman also noted that WHO had its own quality control process. When its 1999 guidelines for treating high blood pressure were criticized for, among other things, recommending expensive drugs over cheaper options without proven benefit, the agency issued its "guidelines for writing guidelines," which led to a revision of its advice on hypertension.

"People are well-intended at WHO," Oxman said. "The problem is that good intentions and plausible theories aren't sufficient."

It remains to be seen how WHO's 193 member countries will react to The Lancet study, released just before WHO's governing body — the World Health Assembly — meets next week at UN headquarters in Geneva to decide future health strategies.

"If countries do not have confidence in the technical competence of WHO, then its very existence is called into question," said Horton, the journal's editor. "This study shows that there is a systemic problem within the organisation, that it refuses to put science first."

WHO Director-General Dr Margaret Chan, who took over the position this year, will be under pressure to respond to the study's criticism.

"We need a strong WHO," which in recent years "has seen its independence eroded and its trust diminished," Horton said. "Now is a fabulous opportunity for WHO to reinvent itself as the technical agency it was always meant to be."