Wikipedia talk:Identifying reliable sources (medicine)/Archive 1

Archive 1

Archive 2

Archive 3

→

Archive 5

Intent?

I'm curious as to the intent of this page; do you seek to create an exhaustive list of magazines and books to use as resources? Or is this page a group of examples, in which case I wonder why it is needed in addition to the normal pages like WP:RS? (Radiant) 13:39, 14 November 2006 (UTC)

This page started as a place to put the "reliable sources" portion of the WP:MEDMOS, which I felt were not style issues and should go elsewhere. It is not intended to have exhaustive list and certainy not for all books/journals as there are simply too many that are of excellent quality. The only reason to list a handful of top journals is to highight those that are regarded as the cream.

Recent discussions on talk WP Medicine have asked:

• Are primary sources better than secondary sources? In particular, editors who cite papers covering basic science possibly being interpreted for its affects in the human population.
• Is XYZ a reliable web site that we should consider either using as a reference or providing external links?

This page could be used to capture consensus on those issues. I could also serve as a resource of sites useful to WP Medicine editors. It is currently very much a work-in-progress. I hoped that others might contribute. Perhaps the links to it are too subtle at present. Colin°^Talk 14:17, 14 November 2006 (UTC)

• I'm no expert on the matter of medicine, but if you want consensus on this page you're definitely going to need more input. While you're technically correct about source matters not belonging in the MOS, if that page is more active than this one it's not a big problem - it is common on Wikipedia for pages to have a somewhat broader scope than their title suggests. (Radiant) 15:10, 15 November 2006 (UTC)

Previous version at RS

WP:RS was substantially rewritten on 1st December 2006 (diff). The following text, which had existed for a while and which we had borrowed for these guidelines, was lost:

Physical sciences, mathematics and medicine

Cite peer-reviewed scientific publications and check community consensus

Scientific journals are the best place to find primary source articles about experiments, including medical studies. Any serious scientific journal is peer-reviewed. Many articles are excluded from peer-reviewed journals because they report what is in the opinion of the editors unimportant or questionable research. In particular be careful of material in a journal that is not peer-reviewed reporting material in a different field. (See the Marty Rimm and Sokal affairs.)

The fact that a statement is published in a refereed journal does not make it true. Even a well-designed experiment or study can produce flawed results or fall victim to deliberate fraud. (See the Retracted article on neurotoxicity of ecstasy and the Schön affair.)

Honesty and the policies of neutrality and No original research demand that we present the prevailing "scientific consensus". Polling a group of experts in the field wouldn't be practical for many editors but fortunately there is an easier way. The scientific consensus can be found in recent, authoritative review articles or textbooks and some forms of monographs.

There is sometimes no single prevailing view because the available evidence does not yet point to a single answer. Because Wikipedia not only aims to be accurate, but also useful, it tries to explain the theories and empirical justification for each school of thought, with reference to published sources. Editors must not, however, create arguments themselves in favor of, or against, any particular theory or position. See Wikipedia:No original research, which is policy. Although significant-minority views are welcome in Wikipedia, the views of tiny minorities need not be reported. (See Wikipedia:Neutral Point of View.)

Make readers aware of any uncertainty or controversy. A well-referenced article will point to specific journal articles or specific theories proposed by specific researchers.

In science, avoid citing the popular press

The popular press generally does not cover science well. Articles in newspapers and popular magazines generally lack the context to judge experimental results. They tend to overemphasize the certainty of any result, for instance presenting a new experimental medicine as the "discovery of the cure" of a disease. Also, newspapers and magazines frequently publish articles about scientific results before those results have been peer-reviewed or reproduced by other experimenters. They also tend not to report adequately on the methodology of scientific work, or the degree of experimental error. Thus, popular newspaper and magazine sources are generally not reliable sources for science and medicine articles.

What can a popular-press article on scientific research provide? Often, the most useful thing is the name of the head researcher involved in a project, and the name of his or her institution. For instance, a newspaper article quoting Joe Smith of the Woods Hole Oceanographic Institution regarding whales' response to sonar gives you a strong suggestion of where to go to find more: look up his work on the subject. Rather than citing the newspaper article, cite his published papers.

Which science journals are reputable?

One method to determine which journals are held in high esteem by scientists is to look at impact factor ratings, which track how many times a given journal is cited by articles in other publications. Be aware, however, that these impact factors are not necessarily valid for all academic fields and specialties.

In general, journals published by prominent scientific societies are of better quality than those produced by commercial publishers. The American Association for the Advancement of Science's journal Science is among the most highly regarded; the journals Nature and Cell are notable non-society publications.

Keep in mind that even a reputable journal may occasionally post a retraction of an experimental result. Articles may be selected on the grounds that they are interesting or highly promising, not merely because they seem reliable.

arXiv preprints and conference abstracts

There are a growing number of sources on the web that publish preprints of articles and conference abstracts, the most popular of these being arXiv. Such websites exercise no editorial control over papers published there. For this reason, arXiv (or similar) preprints and conference abstracts should be considered to be self-published, as they have not been published by a third-party source, and should be treated in the same way as other self-published material. See the section above on self-published sources. Most of them are also primary sources, to be treated with the caution as described in various sections of this guideline.

Researchers may publish on arXiv for different reasons: to establish priority in a competitive field, to make available newly developed methods to the scientific community while the publication is undergoing peer-review (a specially lengthy process in mathematics), and sometimes to publish a paper that has been rejected from several journals or to bypass peer-review for publications of dubious quality. Editors should be aware that preprints in such collections, like those in the arXiv collection, may or may not be accepted by the journal for which they were written — in some cases they are written solely for the arXiv and are never submitted for publication. Similarly, material presented at a conference may not merit publication in a scientific journal.

Evaluating experiments and studies

There are techniques that scientists use to prevent common errors, and to help others replicate results. Some characteristics to look for are experimental control (such as placebo controls), and double-blind methods for medical studies. Detail about the design and implementation of the experiment should be available, as well as raw data. Reliable studies don't just present conclusions.

Popular science and medicine books

Responding to "How do you know what the point is? Discuss this in Talk before deleting."

I wish to remove the line that says "Some well known and respected popular science authors include Richard Dawkins and Stephen Jay Gould." Can the author of this sentence please explain the point of including just a couple of names out of potentially hundreds of worthy authors? Colin°^Talk 14:27, 13 April 2007 (UTC)

Well I originally gave three names in this edit but Sandy removed Oliver Sacks. Colin°^Talk 14:27, 13 April 2007 (UTC)

I don't care much for Richard Dawkins myself and am now questioning the value of providing a short list. This page needs general guidelines instead Colin°^Talk 14:27, 13 April 2007 (UTC)

Yeah. I see what you mean. Just remove it. Colin°^Talk 14:27, 13 April 2007 (UTC)

Done. Colin°^Talk 14:27, 13 April 2007 (UTC)

(Apologies for the sarcasm – just a bit of fun).

Nice debate, sound conclusion :-) As an example, Sacks is infamous for his non-standard and sensationalist views on TS (I've heard other physicians opining on his writing). While we're on the topic: I don't understand either of these edits. [1] We need to get back to, Avoid citing the popular press, as they usually get it wrong. [2] The Merck Manual is an utter and total inaccurate wreck when it comes to TS; I hope it's better in other areas. SandyGeorgia (Talk) 14:33, 13 April 2007 (UTC)

The reason for including a few names is that the text gives the reader no way to distinguish between reliable books and unreliable books, or primary sources and tertiary sources. Giving a few examples gives them an idea of what the text is referring to. The reason for choosing 2 or 3 examples out of potentially hundreds of worthy authors is that you have to stop somewhere. Nbauman 15:17, 13 April 2007 (UTC)

OK, maybe we can approach that from a different angle, then. Instead of giving examples of book authors who may be reliable (and I certainly disagree on using Sacks except in very limited contexts, as I did on TS)—what if we instead give an example of a popular or vanity press book that shouldn't be used ? There are legions, for example, in the realm of ADHD. Patty Duke's book is often used to cite bipolar articles—is that reasonable? In other words, maybe we can say what we "shouldn't" use instead of what we should ??? SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)

That path may lead to conflict. Nbauman is right that it would be helpful to clarify primary from secondary and tertiary. I recently read a good book on Aspirin (ISBN 0747570833) that is thoroughly sourced. It certainly isn't a primary source but is a bit of a mix between secondary and tertiary. It is probably a reasonable source for historical material on the aspirin page but since the author isn't a scientist or doctor, would be a poor choice for the pharmacological material. Very few books are primary sources. Those that are tend to be quoted as a source of original ideas, which is why Sacks and Dawkins were on the list, though the latter hasn't much to say that would concern this wiki project. Colin°^Talk 15:38, 13 April 2007 (UTC)

Not sure how to fix this, so will offer an example (not one I'd want to highlight, though). Comings, Tourette Syndrome and Human Behavior. He owns the vanity press, Hope Press. He couldn't get his work accepted by peers, so he self-published. (For a scathing review of his methodological flaws, see this article.) He's a recognized physician in the field (more infamous than famous),so someone could argue that he's a reliable source. I suppose he would be covered under WP:RS self-published sources, but that's the kind of example I'm concerned about. In terms of wacky theories, anyone can get a book published, so maybe we can find an example that's not self-published as Comings is. And, the problem with Sacks is his focus on sensationalized aspects and his personal views. SandyGeorgia (Talk) 17:01, 13 April 2007 (UTC)

Scientists aren't always the best people for describing the scientific literature and qualities like reliability. The people with the expertise to do that are science librarians. Science librarians work with scientists, and many science librarians have had experience as bench scientists and then went on to be trained as librarians. There are many published reference books on library science that cover the subject of this entry much better than this entry does, and have solved many of the problems that these editors are struggling with. I wish some medical librarians would work on this entry. Nbauman 15:17, 13 April 2007 (UTC)

I don't think we have a medical librarian on board, but I do know someone who may be able to look at your concerns. SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)

I read the book reviews in Science, New England Journal of Medicine, and BMJ. They often review popular books and recommend them strongly. For example, the NEJM last month reviewed "Skin: A Natural History" and a book about Jack Kevorkian. BMJ has been particularly supportive of books that give the patient's perspective. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book, who are we to tell people not to read them? Nbauman

Essay or guideline?

This page is currently a bit too much like an essay than a guideline. That might be fine if we want to keep it an essay, and we may. If not, then the text needs to be condensed and keep to the point. There's a lot more that can be said on this topic and I really welcome other contributors. It might be best to let the text expand a bit before we start refining it back down. That said, if something important got deleted or watered down, then we should bring it back. Colin°^Talk 14:44, 13 April 2007 (UTC)

Newspapers

I deleted the claim that broadsheets can be reliable sources of medical information, while tabloids are not. Some tabloids are excellent sources of medical information, and conversely.

Here's a good example from today's New York Daily News, about the automobile accident of New Jersey Governor Jon Corzine, who was not wearing a seat belt.

[3] A difficult recovery that could take months, by Christina Boyle, New York Daily News, April 14th 2007.

This is what teachers call a "teachable moment," an opportunity to teach an important message because the subject has everyone's attention. This story explains exactly how someone is hurt in an auto accident if they're not wearing a seat belt, and it explains Corzine's injuries in meaningful detail. I've read hundreds of accident reports in the engineering literature, and this tabloid news story covers all the essential points. Nbauman 14:12, 14 April 2007 (UTC)

Looks reasonably well reported - but that is just the issue. All the journalist did was parrot what she was told by the doctor, while probably simplifying the language and possibly making mistakes along the way. I'm not sure how such a story could form the basis of a source on a medical article - just on a bio of the poor man himself. I appreciate that this is just an example where you considered the reporting to be of a good quality.

Perhaps US tabloids are different to UK ones. Some quality broadsheets employ journalists that write original material that may be worth citing on Wikipedia. Though, like the article above, I can't think of a circumstance where I'd do so on a medical article other than in a history/bio section. The journalist Ben Goldacre has a column Bad Science that frequently points out the shortcomings in the UK newspapers, broadsheet and tabloid.

So while I'd support your deletion of broadsheets, I wouldn't support the claim that "Some tabloids are excellent sources of medical information". Not from my experience, anyway. Colin°^Talk 15:01, 14 April 2007 (UTC)

I have to agree. Reporting on a current event is different than sourcing a medical article. A report like that might be useful somewhere in an article about seat belt safety, but shouldn't be the sole basis for medical statements. SandyGeorgia (Talk) 15:18, 14 April 2007 (UTC)

You say, "All the journalist did was parrot what she was told by the doctor, while probably simplifying the language," as if that were a trivial task.

That's not all she did. First, she had to identify one of the important scientific issues, which in this case was, "How do people get injured in automobile accidents, and how do seat belts prevent those injuries?"

Second, she had to find a doctor who was authoritative and knowledgeable, just as a journal editor would have to find an authoritative author for a review article. She didn't get any old doctor, she interviewed an academic doctor at a major critical care center.

Third, she had to get the doctor's intended meaning and quotes right. That's not easy. If she didn't do that right, the doctor would write a letter of correction. I've read a lot of the peer-reviewed auto safety engineering literature, and in my reading of this article, she seems to have gotten it right.

You say, "and possibly making mistakes along the way." You don't know that. That's speculation based on no evidence. The question at issue is whether she made mistakes. In my reading, she didn't.

The reason this news story is so important is that it performs a function of public health education. Most of the medical professional societies in the specialties that deal with automobile trauma have argued for educating the public about the dangers of automobile accidents and the protective effects of seat belts. This article is directly fulfilling that purpose. She quoted a doctor saying so directly:

Had Corzine been belted in, his injuries would have been much less severe. "A seat belt keeps your body fixed to the seat so you wouldn't move forward so much," Shou said. "A seat belt can significantly decrease the chances of these type of injuries."

This is exactly the kind of statements doctors say they want newspapers to print.

But go back to the original question, which is, are tabloid newspapers accurate and reliable? This example from the New York Daily News is clearly accurate and reliable, like all the other medical news I've read in the Daily News. According to the BMJ, and my own sampling, the U.K. broadsheets are not always that accurate. Neither is the New York Times. So any generalized conclusion about the accuracy of tabloids vs. broadside is contrary to fact. Nbauman 17:04, 14 April 2007 (UTC)

I don't think we disagree that much. We just have different experiences. It is great that your New York Daily News is so good. Our tabloids are truly awful. I agree many of the broadsheets are awful too - even the journalists that are supposed to be the house science or medicine reporter. Colin°^Talk 17:30, 14 April 2007 (UTC)

So, let's find a way to make our statements as generally correct as possible; this "tabloid" may have gotten it right, but unfortunately, most don't. There are situations in articles where quoting the popular press is fine, but to the extent possible, peer reviewed literature would be preferred. In my experience, *most* of the time, the popular press gets it wrong in subtle or blatant ways. Can we find some wording around the notion that there are acceptable uses, but peer-reviewed medical literature should be consulted ? SandyGeorgia (Talk) 18:00, 14 April 2007 (UTC)

I've removed the line:

Even peer-reviewed journals like the New England Journal of Medicine cite articles in newspapers like the New York Times and Wall Street Journal.

This may be true but I've yet to see an example. If they do, surely it is more for certain social, historical or biographical information rather than for medical facts? Some examples would help. I've also removed the line:

Newspapers should be judged on the facts, not on prejudices.

Which IMO is not written in a suitable tone.

Finally, I've added a line to clarify where I think there is consensus for using and not using newspapers. Colin°^Talk 11:37, 16 April 2007 (UTC)

Colin wrote: This may be true but I've yet to see an example.

See the example that I cited below of the NYT article on Guidant being quoted in the NEJM article on Guidant. Nbauman 13:27, 16 April 2007 (UTC)

Yes but that's current-affairs. It's a story about a certain individual and the citation backs up the story. And, as you note later, the "Perspective" section of NEJM might not be peer reviewed (and hence the term "peer reviewed journals" is a simplification). The current text, that you've restored, gives the impression that medical journals routinely cite newspapers. They may very occasionally cite them for news but not for medical facts. Please reconsider your restoration of that text. I'm not going to edit-war over this. Can anyone else help us reach consensus? Colin°^Talk 13:54, 16 April 2007 (UTC)

I can't work on it today (or maybe even tomorrow) because I'm flooded from the Nor'easter, but one way to address Nbauman's concerns may be to go back to the top of the article and make it all more about what the highest-quality and medical sources do right, and less about what the others may do wrong (taking care not to overgeneralize). Then we can revisit the newspapers and other sections in that context. If we approach it that way, we may also reduce some redundancy. For example, "The quality of newspaper coverage of medicine ranges from excellent to irresponsible, and they should be verified like any other sources" can be said of most of the other sources (not just newspapers), so we should rephrase in a way that we're not just repeating WP:RS and/or singling out newspapers, rather emphasizing what are the best sources and why. I hope this makes sense; I haven't had much sleep :-) SandyGeorgia (Talk) 15:48, 16 April 2007 (UTC)

I'm sure this guideline will be rewritten more than once before we get it right. There are so many poor sources, that I can't see how the guideline would be useful without mentioning them. After all, people are going to quote this guideline when disputing a source, not when congratulating someone for using a top quality one. One thing that might help is to clarify that this guideline is for the medical facts in medical articles. The social, biographical, current-affairs, etc information can come from sources that meet the general WP:RS. Colin°^Talk 17:11, 16 April 2007 (UTC)

Nbauman, earlier you said "who are we to tell people not to read them". I think this may be one source of our disagreement. This is not an article to advise people what to read (whether for pleasure or for research for a WP article). It is solely concerned with what we should cite. Certain books, newspapers, magazines and blogs may be a reliable source of medical information. They are not suitable for citation in an encyclopaedia (wrt to medical facts). This page is not read by Wikipedia's readers - it isn't for them. It is for editors. Are you trying to help improve the quality of our sources, or defending popular journalism? Colin°^Talk 17:11, 16 April 2007 (UTC)

Colin, I'll restate that. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book as a reliable source, I would accept that book as a reliable source for Wikipedia. Why not? Nbauman 18:38, 17 April 2007 (UTC)

They aren't "recommend[ing] a popular book as a reliable source" for medical facts in an encyclopaedia. They are saying that a book is to be recommended for reading. Such books might be suitable sources for Wikipedia (particularly those that have footnotes, citations and a scholarly comprehensive style). There are usually sources that would be a better choice. However, I return to newspapers, which is our source of dispute. Newspapers are good for news; old newspapers are good for old news. If my doctor offered advice based on what he read in the Daily Mail that day, I'd report him to the authorities. Wikipedia should be no different. Colin°^Talk 22:01, 17 April 2007 (UTC)

OK, here's a newspaper article that got the clinical facts more accurately than the peer-reviewed medical journals did. If your doctor had depended on the Wall Street Journal rather than the New England Journal of Medicine in considering whether to prescribe Ketek, he might have saved your life. Saving patients' lives is after all one of the purposes of medicine. You might rather die than have your doctor use information from newspapers rather than peer-reviewed journals, and I admire you for sticking to principles, but it seems a bit pedantic.

[4]Infected Data: Fraud, Errors Taint Key Study Of Widely Used Sanofi Drug; Despite Some Faked Results, FDA Approves Antibiotic; One Doctor's Cocaine Use; Company Defends Safety, By ANNA WILDE MATHEWS, Wall Street Journal, May 1, 2006.

... Now documents including internal Aventis emails reviewed by The Wall Street Journal are raising questions about a key clinical trial -- called study 3014 -- of more than 24,000 people that the company submitted to the FDA seeking approval for the drug.

The doctor who treated the most patients in the study, Maria "Anne" Kirkman Campbell, is in federal prison after pleading guilty to defrauding Aventis and others. An indictment says Dr. Campbell fabricated data she sent to the company. The documents show that Aventis was worried about Dr. Campbell early in study 3014 but didn't tell the FDA until the agency's own inspectors discovered the problem independently....

The full extent of the study's problems has never been made public. Its results were cited last month in an article in the New England Journal of Medicine that suggested Ketek is as safe as other antibiotics. Five of the six authors of that article disclosed that they received consulting fees from Sanofi-Aventis, and the sixth was an Aventis employee at the time of the study. Nbauman 02:15, 20 April 2007 (UTC)

Avoid citing the popular press

The section, In science, avoid citing the popular press is complete personal opinion, completely unsourced, completely overgeneralized, and completely wrong. For example:

[5]Annals of Medicine: The Bell Curve; What happens when patients find out how good their doctors really are? by Atul Gawande, The New Yorker, Dec. 6, 2004

Gawande is an MD, and in addition to the New Yorker he writes for the New England Journal of Medicine. Does the author of this section believe that Gawande does not cover science well when he writes for the New Yorker, but does cover science well when he writes for the NEJM? Similarly, Gina Kolata, a PhD, used to write for Science magazine before she moved to the New York Times. I could give many similar examples.

Many articles in the popular press have identified problems in medicine that have been ignored by the peer-reviewed literature, for example defective heart defibrillators, or financial conflicts of interest in the committees that set guidelines and recommend drugs. Peer-reviewed journals often cite newspapers in the footnotes.

The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them.

I believe that every article in the popular press should be evaluated on its own merits. You can't replace critical evaluation with a rule of thumb like 'The popular press generally does not cover science well."

This section should be completely rewritten. Nbauman 17:24, 14 April 2007 (UTC)

OK, as an example, if that author also writes for peer-reviewed journals, we should 1) be able to quote him from peer-reviewed work, and 2) be able to justify that the particular author is a reliable source, even if the popular press normally isn't. Remember, reliability of sources, like everything else on Wiki, is subject to consensus. Again, I think we can find compromise wording to the effect that there may be times when a reliable source is quoted in the popular press, but we should consult medical consensus and make sure they got it right. SandyGeorgia (Talk) 18:02, 14 April 2007 (UTC)

I think the consensus statement should start out by saying that the accuracy and reliability of the popular press varies greatly, and the accuracy and reliability of articles may vary within one publication. The popular press can be helpful in finding and organizing information, but everything in the popular press is subject to verification from the peer-reviewed literature. Still, the popular press sometimes reports important information that the peer-reviewed literature does not.

There are library science publications, such as Magazines for Libraries, that evaluate the popular press.

The popular press exposed the dangers of Vioxx before the NEJM did. The Wall Street Journal recently exposed a conflict of interest at the NEJM in which the editor who wrote an editorial supporting a policy of the National Kidney Foundation was on the board of the Boston chapter of the NKF, as I recall.

You just have to examine the facts on an case-by-case basis, and sometimes all you can do is give the opposing viewpoints. Nbauman 18:39, 14 April 2007 (UTC)

Well, you got me on the Vioxx. hits much too close to home :'-( Colin is a better writer than I am, so I'll stay out of tweaking the wording, and leave that to the two of you, but I agree with some of your points now. We just need to word it in a way that doesn't open the door too wide. SandyGeorgia (Talk) 18:56, 14 April 2007 (UTC)

I'm going to think a bit more about this one before changing any text. I'll read the Bell Curve thing later too. Hope someone else joins the debate. The current Newspaper's section reads more like a polemic than a guideline. Remember the context: medical facts. This isn't a guideline for current-affairs, history, biography, sports, etc. I'd love to see an example where citing a newspaper on a medical issue was better than citing an alternative source.

Your arguments that some articles in newspapers are OK can be extended to blogs, personal web sites, drug-company adverts, press releases, vanity press and all the other things that we guide against. The "is an MD" attribute doesn't make someone the holder of the truth. There are plenty nutty MDs out there.

You are in the privileged position of being able to judge whether an individual source (on certain topics) is good or not. Many of our editors, and almost all of our readers are not. This is why we have WP:V and WP:RS. The "truth" is secondary on WP.

WP should never use investigative reporting as a source. Newspapers get it wrong at least as often as right. They like to boast when right. Such reporting has only two editorial controls: "are our lawyers happy" and "will it sell papers". We all know of medical campaigns by newspapers that make the professionals despair. Wikipedia is in no rush. Better to report well established facts than up-to-date nonsense.

Many editors come to WP in order to evangelise the world about their treatment or their conspiracy theories as to the cause of illness. You only have to look at the more controversial sections of the autism topics on Wikipedia to find that such editors love quoting the tabloids.

You mention (in the guideline and above) that peer-reviewed journals "often" cite the popular press. Really? What percentage makes "often"? Are we talking about medical journals?

The following gets me worried:

"The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them."

This has been considered. The section on secondary sources says "Journalists writing in the popular press, and marketing departments who issue press releases tend to write poor secondary source material."

The sort of press-releases that hospitals/labs do, which are written to stimulate funding and press interest rather than advance medical science, are the last thing WP should be using as a source. And when newspapers get hold of these, they either regurgitate verbatim and uncritically, or else they mess with it. I'll try to find an example if you like. The latest cutting edge unproven research is really not what an "encyclopaedia" is about IMO.

Sorry about the rant. Your userpage says your are a professional writer. Then I'd love to know your opinion and get your support for WP:MEDMOS, which is being discussed nearby! Colin°^Talk 20:06, 14 April 2007 (UTC)

Librarians have lists of magazines that they have determined to be reliable, with the caveat that the ultimate responsibility for making a decision rests with the reader. One of the standard reference works, for example, is Magazines for Libraries. Some magazines are more reliable than others. The New Yorker is consistently reliable. So is the news section of the Wall Street Journal, though other sections, like the editorial and personal health columns, are not. There's no simple guideline that anyone could apply to determine whether a source is reliable. The only answer is that popular magazines are diverse, their quality varies, and they are subject to verification like anything else.

As for opening the door to blogs -- that's a problem, but some blogs are reliable sources. There should be a burden of proof that they're reliable, but if they meet that burden, OK. Best example I can think of is CancerGuide: Steve Dunn's Cancer Information Page (which BTW has a good discussion of how to evaluate reliable sources of information, from bench to clinical trial).

You wanted an example of a newspaper that cited the popular press. Here's one -- footnote 2. The academic publications didn't even cover this until the New York Times broke the story. The issue here was that Guidant knew their ICDs were short-circuiting, but kept selling the existing stock even though they had solved the problem with newer models. (I still don't understand why.) The Guidant entry in Wikipedia doesn't seem to have cited them. I'm excerpting these articles since you may not have subcriptions.

[6]The Controversy over Guidant's Implantable Defibrillators,

Steinbrook R,

N Engl J Med 2005; 353:221-224, Jul 21, 2005. Perspective

On October 4, 2001, Joshua Oukrop, a Minnesota teenager with hypertrophic cardiomyopathy and a high risk of sudden death from ventricular fibrillation, received an implantable cardioverter–defibrillator (ICD). The device was a Ventak Prizm 2 DR Model 1861 manufactured by Guidant (Indianapolis). After it was implanted, Oukrop's physicians at the Minneapolis Heart Institute Foundation checked it every three months (most recently on January 31, 2005) and found no problems.

On March 14, 2005, Oukrop, then a 21-year-old college student, collapsed and died in a remote area of southeastern Utah during a spring-break bicycling trip with his girlfriend.1,2 An autopsy revealed no clinically significant pathology beyond his massive left ventricular hypertrophy. His physicians were stunned by his death. ICDs have been shown to be almost invariably successful in preventing sudden death in young patients with hypertrophic cardiomyopathy, as long as they do not have end-stage heart failure — which Oukrop did not.3 When the manufacturer analyzed his ICD, it determined that the device had short-circuited internally while trying to deliver high-voltage therapy and had been permanently disabled (see diagram). Moreover, its memory had been destroyed, making the time of failure impossible to pinpoint....

2. Meier B. Maker of heart device kept flaw from doctors. New York Times. May 24,2005:A1.

[7]Maker of Heart Device Kept Flaw From Doctors

By BARRY MEIER, New York Times May 24, 2005

A medical device maker, the Guidant Corporation, did not tell doctors or patients for three years that a unit implanted in an estimated 24,000 people that is designed to shock a faltering heart contains a flaw that has caused a small number of those units to short-circuit and malfunction.

The Potential for Defibrillator Failure

The matter has come to light after the death of a 21-year-old college student from Minnesota, Joshua Oukrop, with a genetic heart disease. Guidant acknowledges that his device, known as a defibrillator, short-circuited. The young man was in Moab, Utah, on a spring break bicycling trip in March with his girlfriend when he complained of fatigue. He then fell to the ground and died of cardiac arrest.

Guidant subsequently told his doctors that it was aware of 25 other cases in which the defibrillator, a Ventak Prizm 2 Model 1861, had been affected by the same flaw. Guidant said it had changed its manufacturing processes three years ago to fix the problem. The physicians say that had they known earlier, they would have replaced the unit in their patient because he was at high risk of sudden death. His death is the only one known....

Nbauman 21:16, 14 April 2007 (UTC)

But there's a theme to your examples, Nbauman, which may help us solve this dilemma. They are tied to current events. The news media stories came about because of current events, as opposed to being reports per se about the conditions or studies about the conditions. I may not trust most newspapers to accurately tell me about the most recent studies on hypertrophic cardiomyopathy, but maybe I can trust some newspapers to report a story about a current event involving that condition correctly. SandyGeorgia (Talk) 21:30, 14 April 2007 (UTC)

The Bell Curve

I've not finished reading this but already spotted something that highlights why newspapers make poor secondary sources. Page two discusses various operations and the variable success rate amongst surgeons. One example given:

A Scottish study of patients with treatable colon cancer found that the ten-year survival rate ranged from a high of sixty-three per cent to a low of twenty per cent, depending on the surgeon.

The article doesn't fully cite the study, which is typical and understandable (though perhaps not for the online edition, which has no space concerns). So we don't know:

1. Why, on an article in an American magazine, with mostly American examples and statistics, does the author pick a Scottish study? Are they desperate to find an example to back their case? Or perhaps this study is internationally famous?
2. How big was the study? How many patients? How many hospitals? Did it just cover NHS hospitals or private ones too?
3. How was the study conducted?
4. What threshold did they use for "treatable"? Did all the hospitals use the same threshold when choosing patients?
5. What other factors influenced patient outcome? Perhaps patients living in poor areas did worse than richer ones and this affected the local hospitals they went to?
6. Perhaps there was a hospital variation, not just the surgeon. A patient's recovery depends on many professionals' input, not just the surgeon.
7. Did they have other non-surgical therapies too?
8. The article makes a case that the range of success follows a bell curve. But this example only quotes two extremes. We have no idea what shape graph this study produced.

And so on. Most "for-professionals" articles would give more information than this, plus a citation. Without this traceability from Wikipedia to secondary source to primary source, our readers are limited in how much they can learn should they ask questions concerning the reliability of the data and how the author chooses to use it to make their case. The best popular science/medical books provide citations. So should online newspapers. Colin°^Talk 23:09, 14 April 2007 (UTC)

Gawande was citing a classic, frequently-cited study that every oncologist would know. [8]McArdle et al, BMJ 1991;302:1501-5, Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival. University Department of Surgery, Royal Infirmary, Glasgow.Nbauman 06:03, 15 April 2007 (UTC)

Every oncologist might think "Ah, she's citing McArdle and Hole 1991" but almost none of our readers would. I was going to try to look up the paper myself but you've saved me the trouble. Unfortunately, it only goes to strengthen the argument that the primary source should be cited by WP rather than the newspaper (which may be less than helpful in finding it). For example, I can now cite:

• McArdle C, Hole D (1991). "Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival". BMJ. 302 (6791): 1501–5. PMID 1713087.

which fortunately has the full text available free online. Even from the abstract, our reader can tell more than the newspaper provided. We can also see there were two follow-up letters, one critical and one supportive. PubMed also tells me that McArdle and Hole continued their research. Which is just as well: these were "patients with colorectal cancer presenting over the six years from 1974 to 1979". The quality of an operation performed 30 years ago is of diminishing interest to those going under (or holding) the knife today. They published a follow-up paper in 2002 (with a much larger number of patients, operated over the years 1991 and 1994) that confirmed that variability amongst surgeons was still a problem:

• McArdle C, Hole D (2002). "Outcome following surgery for colorectal cancer: analysis by hospital after adjustment for case-mix and deprivation". Br J Cancer. 86 (3): 331–5. PMID 11875693.

So it could be argued that the first might be a historical classic, but the second is of more relevance today. They have also answered some of my questions about poor patients doing less well:

• Hole D, McArdle C (2002). "Impact of socioeconomic deprivation on outcome after surgery for colorectal cancer". Br J Surg. 89 (5): 586–90. PMID 11972548.

I see they have refined their conclusions about surgeon variability, by showing that surgeon speciality is a better guide than just volume of work:

• McArdle C, Hole D (2004). "Influence of volume and specialization on survival following surgery for colorectal cancer". Br J Surg. 91 (5): 610–7. PMID 15122614.

I could go on (for example, to check that those who cite this paper do so favourably) but in just a couple of minutes, I've found so much more high quality source material to help improve a WP article on either colorectal cancer or issues of surgeon competence. Colin°^Talk 07:00, 15 April 2007 (UTC)

---

Let's return to the original question. Is the New Yorker a reliable source of medical information or not? Nbauman 13:47, 15 April 2007 (UTC)

Back to what I said above; there are some things that can possibly be cited to some of the news media (I've done it in Sociological and cultural aspects of Tourette syndrome), but a medical, peer-reviewed source is always preferred, as demonstrated in Colin's example. The news media rarely gives us the full story. Let's make sure our wording gives a strong preference to peer-reviewed sources. I believe (haven't checked) we don't say never cite the popular press; we urge caution. Perhaps we can expand/contract that to say they shouldn't be preferred over the highest-quality, peer-reviewed medical sources, but do have some usefulness. I think (?) your concern is that we may be "bashing" the popular press; perhaps we can better flesh that out. SandyGeorgia (Talk)

Right, I don't want to bash the popular press. Most users of Wikipedia use the popular press (many academic doctors read the popular press, as demonstrated from their footnotes), so we should help them place the popular press in its proper context. I think the context is that their accuracy and reiability varies greatly. Some magazines are accurate enough that we can presume them to be accurate, but they're always open to challenge, and peer reviewed literature usually (but not always) trumps the popular press. There are evaluations of the popular press, in the links I added under Newspapers. Nbauman 18:21, 15 April 2007 (UTC)

Gawande's work for NEJM is peer-reviewed. His journalism is not. In general, peer-reviewed science is the preferred source for any medical content, with all other sources inferior to it. Within peer-reviewed science, I think we should adopt EBM grading. A meta-analysis or systematic review is much more powerful as a source than individual trials, case-control studies, case series or case reports. JFW | T@lk 20:14, 15 April 2007 (UTC)

JFW, can you help work on the wording with Colin here? SandyGeorgia (Talk) 20:24, 15 April 2007 (UTC)

Some of those ideas are already very briefly mentioned but not in such an explicit order. JFW is much, much more knowledgeable on this front than me so I'd rather defer to him. I can tidy the English if required :-). I'd really like JFW to contribute to this, since he's often the one defending these things on the project or article talk pages. Colin°^Talk 20:47, 15 Apri

I don't know. Is the Perspectives section of the NEJM peer-reviewed? Nbauman 00:05, 16 April 2007 (UTC)

That's a good point. Not all forms of writing in these journals get the same treatment. For example, an obituary may be no better checked than one in a newspaper. There are several aspects to judging the quality of a source, not just the journal title or type. Colin°^Talk 07:38, 16 April 2007 (UTC)

TV, radio

Re popular press, now that TV and radio news programs have websites, I see a need for some guidance re citing them. See for example fetus in fetu, where one editor cites ABC and MSNBC news as sources of medical data (incidence, treatment). --Una Smith 15:20, 9 July 2007 (UTC)

What wording do you propose (can we make it more general, and not focus on any specific aspect of the popular press)? SandyGeorgia (Talk) 15:26, 9 July 2007 (UTC)

Core journals

This section needs a paragraph explaining that niche journals have low impact factor regardless of quality, because their readership is small; niche journals can be evaluated (and to some extent compared to "core" journals) based on their average article halflife, meaning the number of years over which an article is cited. --Una Smith 15:28, 9 July 2007 (UTC)

If at all possible, avoid promoting the idea of rating an article by the author's authority: where they work, their title, their rank. Judge the person by the quality of the work, not vice versa. --Una Smith 15:28, 9 July 2007 (UTC)

Proposed Guideline?

This page was set up as a Proposed Guideline. It would appear that WP only accepts such a proposal for a finite time before marking it Historical or Rejected. Can we decide what we are going to do with it? My gut feeling is that there currently isn't enough traffic or discussion on this for it to move forward quickly enough to become a formal guideline before someone retires it again. If someone else wants to beat a drum to round up some contributors, then great. I was hoping that we might get contributions from editors with experience writing or training reading medical articles. I do believe the project needs these guidelines

1. To have somewhere to point at when the issue arises.
2. To have a central place to discuss what kind of sources are good for medical info and how they should properly be used.

But it may be that those needs can be met by taking the banner off and leaving it as an informal guideline in Project space. Thoughts? Colin°^Talk 11:35, 15 May 2007 (UTC)

The editor who tagged it rejected tagged dozens at the same time; I doubt he read the ongoing discussion at WP:MEDMOS. I think we can push for guideline status, with just a bit more review. Since we were busy on MEDMOS, we haven't really pushed. SandyGeorgia (Talk) 11:45, 15 May 2007 (UTC)

If I may chime in... proposals tend to be marked "historical" if there's no active debate going on. This is a matter of convenience, not a fixed difference. In particular, if active debate starts once more (e.g. through advertising the page) then it's active and no longer historical. I see no substantial dissent on this talk page, although I lack the expertise in medicine to vouch for the quality of the page. If a few medically-inclined people can confirm that it's (1) accurate and (2) useful, then it would make a worthwhile guideline. Guidelines are not a big deal and are certainly allowed to have exceptions. >Radiant< 11:55, 15 May 2007 (UTC)

(A) If there is discussion regarding development of this guideline it should be reflected here

(B) There is insufficient breadth of contribution here to demonstrate a consensus for acceptance

(C) The onus to demonstrate consensus is on the proponents and in default a proposal is rejected without demonstrable consensus

(D) Yes, this was included in a cleanout of dead proposals, and for now I remove my advocacy of rejection. --Kevin Murray 12:07, 15 May 2007 (UTC)

Our efforts (and discussion) had been focused first at WP:MEDMOS; we'll get this moving again. SandyGeorgia (Talk) 12:10, 15 May 2007 (UTC)

Not ready for guidelines

I don't think this guideline is ready to be adopted. Much of it is unsourced, unsupported personal opinion. For example, "The popular press generally does not cover science well." Who says so? What is their evidence? Isn't this an overgeneralization? In some cases, the Wall Street Journal has turned out to be more reliable than the New England Journal of Medicine.

Why doesn't the entry cite the extensive literature on how the popular press covers science (which finds that the quality of coverage varies from very good to very bad)? What about the library literature, such as Magazines for Libraries?

A fundamental problem is that this is just a list of sources and judgments about them. It wouldn't be much help in resolving real disputes that go on in real articles. Define the problem. I would suggest that you look at some actual controversial medical entries, and examine the disputes that come up over reliable sources, etc. Look at Dichloroacetic acid, or Diabetes.

People do cite the popular press all the time. What should we do about it? Most of the peer-reviewed literature isn't available free on the Internet, so it isn't verifiable to someone who doesn't have a subscription. What do we do about that?

What are the disputes and how should they be resolved, in terms of reliable sources? Nbauman 14:26, 15 May 2007 (UTC)

Maybe the proponents could provide some examples of articles which would be removed if this guideline was adopted and why they are inappropriate for inclusion at WP. --Kevin Murray 14:32, 15 May 2007 (UTC)

Kevin, not sure what you mean by articles which would be removed; we're talking about sources here ? SandyGeorgia (Talk) 16:01, 15 May 2007 (UTC)

I agree 100% that this is not ready to become a guideline right now. That's why I now think our use of "Proposed" may be a little premature. I had thought the label was OK for saying "I propose we have some guidelines on this. Here's a start I've made...". But it is being interpreted as "Here's a set of guidelines I propose. Discuss...". I don't really mind what the banner says (but not "rejected", please) or if we don't have one. It would be nice to have some kind header/intro that said:

Wikiproject Medicine would like to propose some guidelines on choosing and using sources for medical articles. These guidelines are at an early stage so all contributions are welcome. Please be bold and be willing to discuss.

I think Nbauman makes some good points, but WP guidelines do not need to be sourced. Opinion is fine if there is consensus.

I'm finding some of the newspaper arguments are starting to repeat so would welcome input from others. Can we please stay focused that these are guidelines for medical facts. When has The Wall Street Journal ever been "more reliable" for medical facts than the NEJM? I'm not talking about breaking some medical scandal a few weeks early. I've just tried searching their online site for medical info and have been unable to find anything other than articles about how some drug approval or loss of patent is affecting some company's share price, or a paragraph on new research opportunities (that affect a company's share price). Colin°^Talk 15:21, 15 May 2007 (UTC)

The statement "In some cases, ... has turned out to be more reliable than" just doesn't work. I'm sure you can find cases when it was "more accurate than" for a given topic and moment in time. But "reliable" implies one can regularly, not just occasionally, depend on it. If this was an essay on "Accurate sources" then a survey of the "extensive literature" on different sources and their quality might make an interesting read.

The other aspect we need to consider is "useful". I think the example on the Bell Curve above showed that newspapers aren't as useful a source as a journal, even if the information is technically accurate.

I'll have a look at your other points later to see what I can find. Colin°^Talk 15:58, 15 May 2007 (UTC)

Popular press example

Here's an example of using the popular press, which I cleaned up using peer-reviewed or medical sources just last week — from an area I'm familiar with.

• Hair pulling disorder gene found. BBC (29 September 2006)

Note the headline — gene found !! This is what the popular press does.

When in fact ... The peer reviewed sources are available, and the finding is reported in a way that is more "scientifically" and "medically" correct. The BBC merely parroted some portions of the Duke Medical News, while adding nothing clarifying or illuminating.

• Zuchner S, Cuccaro ML, Tran-Viet KN, et al. "SLITRK1 mutations in trichotillomania." Molecular Psychiatry. 2006 Oct;11(10):887-9. PMID 17003809
• "Hair-pulling Disorder Caused by Faulty Gene in Some Families" (Press release). Duke Medical News. 2006-09-27. Retrieved 2007-05-08. {{cite press release}}: Check date values in: |date= (help)

Note the more cautious and accurate headline in the Duke Medical press release, and The researchers estimate that the SLITRK1 mutations account for 5 percent of trichotillomania cases. This gene is not significant in and of itself (it is not *THE* gene that has been discovered as some earth-shattering event), as much as it provides a vehicle for future research directions. There is no need to cite the "hyped" BBC version, when the peer-reviewed article can be found in a medical library (and we don't choose our sources based on whether they are easily available online, free or not - we choose the best and highest-quality sources period, even if they're not available online, which should be the peer-reviewed medical literature).

Further, if the issue of citing the popular press is the only problem with these guidelines, we can work on the wording. But, whenever peer-reviewed sources are available, they could at least be preferred over the popular press, which tends to hype results and take them out of context or proportion. SandyGeorgia (Talk) 16:12, 15 May 2007 (UTC)

Ah Sandy, I'm very disappointed in you ;-) You didn't spot the spelling mistake in the press release that has interesting consequences. The first time the press release mentioned the gene, they spell it SLITKR1, not SLITRK1. Now Google for [Trichotillomania SLITKR1]. You will find all the news sources that uncritically used the press release (267 hits). If you Google for [Trichotillomania SLITRK1 -SLITKR1], you wont find any popular news articles (except a couple of very short ones) but a fair amount of research.

The press release is also somewhat flawed, though not as much as the popular press version. For example, at the start they say scientists "have identified gene mutations that cause trichotillomania" but much later weaken this to "The SLITRK1 gene could be among many other genes that are likely to interact with each other and environmental factors to trigger trichotillomania". They don't say how many people have SLITRK1 mutations without trichotillomania, for example. Is this a "cause" or just a "factor"? They also estimate it accounts for 5% of cases but the press release doesn't explain what gives them that idea. I thought the BBC were dumbing down with talk of "faulty wiring" but that came from the press release! Lots of handwaving going on here :-) Colin°^Talk 17:18, 15 May 2007 (UTC)

Right. The press did the same thing when this gene was implicated in Tourette's, and you had to read the actual journal report to sort it all out. The popular press cannot be counted on to report this sort of thing correctly. I haven't read the full journal report in this case, because I know it's the same kind of deal as in the TS connection. This gene may be a factor for a small subset of patients, which makes it interesting for research, rather than some kind of breakthrough. SandyGeorgia (Talk) 17:26, 15 May 2007 (UTC)

Re the WSJ sometimes being more accurate than the NEJM, I read both regularly and I could find many examples. I already mentioned COX-2 inhibitors. A recent one is Ketek -- the NEJM published an article citing an industry-sponsored study, but the WSJ pointed out that the doctor who recruited the largest number of patients in that study was indicted after an FDA investigation, pleaded guilty to fraud for fabricating patients and falsifying patient records in the study, and is now in jail. If you read the Perspectives section in the NEJM, you'll see news-style articles that report on these problems, though the NEJM is not always forthright on their own role. Nbauman 18:15, 15 May 2007 (UTC)

But, again, this was already covered in our discussion above. This isn't reporting about the "medicine"; it's reporting about a current even surrounding the medical event, in which case citing the popular press makes sense. Maybe we need to reword to clarify the distinction? When we went over this (above), it seemed we agreed. SandyGeorgia (Talk) 19:33, 15 May 2007 (UTC)

The question in that case was whether there were scientifically valid studies to support the safety and efficacy of Ketek. The NEJM claimed, without qualification, that there were. The WSJ reported that the studies did not follow scientific protocol and were not valid. The WSJ reported unpublished FDA reports, which they got through FOI requests and leaks, about liver failure from Ketek. That's medicine.

Michael Fiore was on the Public Health Service panel whose guidelines said that smokers couldn't quit without nicotine patches, gum and other substitutes, and published articles saying that in peer reviewed journals. The WSJ reported that Fiore got $1 million from GlaxoSmithKline which makes Nicorette gum, and manufacturers of other substitutes -- without disclosing his financial ties. The WSJ quoted other experts and cited studies which found that patches, gum, etc. offered no benefit. That's medicine.

Here you have peer-reviewed journals publishing articles by a paid consultant to drug companies, who recommends his company's drug without disclosing his financial interest. A newspaper discloses those interests, reports other studies that say the drug doesn't do any good, and interviews medical experts who confirm that. The question of whether nicotine patches help people stop smoking is a question of medicine. Which source is more reliable?

You cannot make an unqualified statement that newspapers or popular magazines are not as reliable as peer-reviewed journals. Nbauman 20:21, 15 May 2007 (UTC)

Scandal, scandal, scandal. That's all you, and the newspapers, seem to be bothered about. Unless there's a "story" to be told, the press just aren't interested. What about all the drugs that work and were well researched? What about the stories in the UK press about the latest cancer "treatments" being denied to patients on the NHS that when you investigate them, are perhaps for the very good reason that there is no evidence that they work, or work any better than existing, cheaper treatments. The press get these stories wrong at least as often as right. You still haven't proven they are "reliable" at all. Only that, occasionally, they get something right and perhaps prove others wrong. When you say that the "WSJ quoted other experts and cited studies which found that patches, gum, etc. offered no benefit" it would be doing so selectively in order to make its case and emphasise the scandal. The journalist would certainly have ignored any studies that disagreed with their case. I accept and understand that. It is not medicine, it is journalism. It has a useful place in our world, but its place in an encyclopaedia is (as Sandy says) when documenting current/historical events.

Only a very tiny percentage of drugs and medicine interest the press. If it is good news, it is over-sold. If it is bad news it is the worst scandal ever. An encyclopaedia does not need to be at the cutting edge. We can wait for things to be proven, and if some of that "proof" was wrong or fraudulent, we can wait for judges and experts to agree it was wrong. Colin°^Talk 22:12, 15 May 2007 (UTC)

You're simply asserting your own personal opinion, without providing any evidence. I'm giving multiple specific examples, and also citing academic research, such as Health News Review, where journalists and doctors review and evaluate the news coverage -- and come to different conclusions than you do. Why don't you consider the possibility that many doctors who specialize in patient education, and disagree with you, may be right?

You state that "The journalist would certainly have ignored any studies that disagreed with their case" -- but this is not true, and you would have seen that if you had read the article itself. The WSJ always gets both sides of the story, as it did in this case. You're the one who is ignoring facts that disagree with your case. Nbauman 12:41, 16 May 2007 (UTC)

We've already been over examples from all sides, and this is going in circles. You've given examples of current events that some newspapers got right. So that we can stop going in circles, can you propose some wording that 1) gives preference to peer-reviewed sources over the popular press when they are available (see the example I gave), and still allows for popular press reporting of current events, while avoiding "recentism"? We need proposed wording that recognizes the value of peer-review, and the mistakes that can be made in the lay press. SandyGeorgia (Talk) 15:04, 16 May 2007 (UTC)

Yes, that's what I'm trying to do. I only object to the oversimplified, unsupported statement that (1) If it appears in the popular press it's not reliable and (2) if it appears in the peer-reviewed literature it is reliable.

My position is that the popular press varies in reliability. I gave you links to a web site run by doctors and journalists, in which doctors review and evaluate the reliability of articles in the popular press. I also cited reference books that librarians use, such as Magazines for Libraries, that evaluate the reliability of popular magazines. Some popular publications and newspapers are more reliable than others.

Peer-reviewed journals are usually more reliable, but not all of them. I liked to a publication, the Brandon-Hill list, that lists the most reliable peer-reviewed medical publications. Some peer-reviewed publications are financed by drug companies, or medical device companies, and publish "peer-reviewed" articles that support the use of their products. So some peer reviwed journals are more reliable than others.

I would suggest that the guidelines include language like that above.

In general, good peer-reviewed publications are more reliable than the newspapers or popular press. But there are lots of exceptions. Even the peer-reviewed journals, like The Lancet, will publish articles that they know are wrong, because they want to get the argument out for debate, as they did with that article on rats who ate genetically modified potatoes. The BMJ (I think) published an article on mercury preservatives in vaccines which they and every legitimate doctor have repudiated, but people keep quoting it. Nbauman 21:26, 16 May 2007 (UTC)

Classification and grading of clinical evidence

Hello. Are the AHRQ guidelines on grading evidence too technical for inclusion here? I'd really like to see a summary of the grading as outlined in evidence-based medicine included here - I think it's important, no? Just a table like this:

• Ia: evidence from meta-analysis of randomised controlled trials (RCTs).
• Ib: evidence from at least one RCT.
• IIa: evidence from at least one well-designed controlled study without randomisation.
• IIb: evidence from at least one other type of well-designed quasi-experimental study.
• III: evidence from well-designed non-experimental descriptive studies, such as comparative studies, correlation studies, and case control studies.
• IV: evidence from expert committee reports or opinions and/or clinical experience of respected authorities.

Grade	Evidence	Description
A	Ia, Ib	Requires at least one RCT as part of the body of literature of overall good quality and consistency addressing the specific recommendation
B	IIa, IIb, III	Requires availability of well-conducted clinical studies but no RCTs on the topic of recommendation
C	IV	Requires evidence from expert committee reports or opinions and/or clinical experience of respected authorities. Indicates absence of directly applicable studies of good quality.

Would do the trick. Thoughts? Nmg20 17:37, 15 May 2007 (UTC)

I think that would be a good and even necessary addition. You can't write about medicine if you don't understand this. Nbauman 18:17, 15 May 2007 (UTC)

Please add to the page, we can tidy/refine later. There is more than one way of grading sources. The current headings (Periodicals, Books, Online) could perhaps be demoted to 2nd level under a new "Media types" heading (or similar wording). Then we could have another top level heading for "Research types", for example.

We should link to Trish Greenhalgh's "How to Read a Paper" Series. If people read that, they'd have a good idea of different study types. Colin°^Talk 22:20, 15 May 2007 (UTC)

I've added something along these lines, although I haven't added the link to the "How to Read a Paper" series, which should certainly go in. I wasn't feeling very inspired when I made the changes, though, and it certainly needs proofing by a couple of other editors if anyone has time... Nmg20 01:59, 18 May 2007 (UTC)

Question about conference proceedings

I find that it is often easier to access reprints of conference proceedings online than complete journal articles. I'm not sure if that is only true for myself as a veterinarian or if it also applies to M.D.s. Where do these proceedings fall in the matter of reliability, in this project's opinon? Note that these are major conferences, with well-respected lecturers, and I'm only referring to reviews of topics, not new research. Thanks. --Joelmills 03:12, 25 June 2007 (UTC)

Still easier: don't bother citing anything. (That was a joke.) I cite such sources only as a last recourse, and then only if I know the article actually was presented at the conference. Sometimes the proceedings are published before the conference, and then at the conference the article is retracted! This is more common when the proceeding volume consists of abstracts or short articles that are little more than abstracts. Some proceedings volumes are peer reviewed and/or of the highest quality, but they are in the minority, and you really have to know the specific research community to know which proceedings volumes are top-notch and which ones are not. It isn't enough to go by series title, because this can change from year to year, depending on who is the editor. --Una Smith 15:40, 9 July 2007 (UTC)

Proposed changes

In the spirit of being bold, I've made a few changes to the proposal. Here is a summary diff. Feel free to revert them if they seem redundant or inappropriate. In general, I'd like to be a little more explicit on the fact that primary sources (journal articles reporting original findings) are a welcome and even necessary part of medical articles, but that the interpretation of such research must hew carefully to that provided in reliable secondary sources (reviews/textbooks). I've seen significant issues with editors citing a number of basic-science journal articles and then leaping to a totally off-the-wall conclusion, which is then defended as "cited content".

Another issue is articles on supposed medical conditions which have never been reported or recognized by any medical authority (see mucoid plaque). I would favor including something in the guideline along the lines of, "If a purported medical condition, test, or treatment has been described and evaluated by the medical community, then it should be easy to cite reliable sources on the subject. In the absence of such sources, topics should not be presented as if they are accepted by the medical community." But perhaps this is overstepping the bounds of this proposed guideline. MastCell ^Talk 17:04, 25 June 2007 (UTC)

A related example is pyroluria. SandyGeorgia (Talk) 17:07, 25 June 2007 (UTC)

Yes, good example. In both cases I think it's important to place the burden of proof where it belongs (i.e. show us that this is a real, accepted medical condition) rather than being in the position of trying to prove a negative. Whether this guideline is the place to do so, I don't know. MastCell ^Talk 17:10, 25 June 2007 (UTC)

I very much agree with a statement like that being included into the proposal. Also, there are ideas of diseases/conditions made as a statement of fact, even though the person espousing such ideas freely admit that they go against the medical community. In other words, a person states something like, "the medical community states this, but that is wrong or baseless because of this fact." That does not help in making articles reliable. Perhaps the sentence could read something like, ..."In the absence of such sources, topics should not be presented as accepted by the medical community or as a statement of fact." Or some variant of that. Either way, no matter what "facts" someone presents, if it goes against the vast medical consensus, then those views should not be presented as "facts". Of course, truly contested material should be presented as such. Perhaps a sentence should be inserted saying something similar if one is not included already. - Dozenist talk 18:29, 25 June 2007 (UTC)

PubMed vs. secondary sources

Discussion moved from Wikipedia talk:WikiProject Clinical medicine:

WP:MEDRS seems to be at odds with WP:MEDMOS; MEDMOS encourages the use of PubMed references, MEDRS implicitly discourages them.

WP:MEDRS states:

In general, Wikipedia's medical articles should use published reliable secondary sources whenever possible. Reliable primary sources may be used only with great care, because it's easy to misuse them. For that reason, edits that rely on primary sources should only make descriptive claims that can be checked by anyone without specialist knowledge. Any interpretation of primary source material requires a secondary source.

In my opinion:

• The above (in WP:MEDRS) should be further qualified. Primary sources, IMHO, are accessible to an interested layperson, with the vast amount of credible medical information (e.g. Merck Manual, eMedicine, Medlineplus.org, |Canadian Health Network) out there and the strong base of Wikipedia articles that cover topics in medicine and experimental physiology.
• Primary sources should be the key references-- secondary sources should be considered supplemental. Primary sources should be explained -- like any good secondary source for the lay public.
• Good secondary sources base their info from primary sources. I think Wikipedia has enough people with expertise to deliver nuanced interpretations of primary sources that can compete handily with respected secondary sources.
• Use of secondary sources from PubMed (i.e. review articles) should be encouraged.

I look forward to the discussion. My thoughts on this arose from this discussion-- and are related to changes to the McClintock effect article. Nephron  T|C 06:12, 24 June 2007 (UTC)

I wonder whether that's an old policy which is now outdated as the 'pedia continues to grow in depth. I don't worry about it myself and often use primary sources as do many others. Have a look at alot of FA nominees.cheers, Casliber (talk · contribs) 06:17, 24 June 2007 (UTC)

I didn't think scientific journals were necessarily considered primary sources. Mostly, it means that care should be exerted when using recent, sweepingly new results, but I've never seen people objecting to theuse of scientific journals on the base of them being "primary sources." I guess this is because no article can exist that is not based on some amount of existing work. I certainly agree that review articles/material should be encouraged, if only because that material is generally more directly accesible, and can provide of otherwise difficult to locate further material. The couple review sources I used for Verbascum thapsus provided most of the important structure for the article. Circeus 07:18, 24 June 2007 (UTC)

• It seems like a non-sensical sentiment to dismiss pubmed indexed articles as preferred sources. I agree, however, that people who are unfamiliar with some areas will use refs. out of context to prove this or that. The expertise of editors in some of these specialty areas helps to filter the wheat from chafe. Secondary sources like standard medical textbooks, positions of health ministries (eg. FDA, health Canada) etc. can be used to bring contextDroliver 15:37, 24 June 2007 (UTC)

The danger with favoring, or relying on, primary sources is that you're depending on the expertise and insight of the editor to use and interpret them appropriately. For instance, a primary source stating that a compund shrinks human tumors engrafted in NOD/SCID mice may be presented as "XXX is highly effective against many human cancers." (Yes, I have been scarred by the DCA thing). In cases where 1 study has showed an effect, but dozens of others have confirmed that the effect does not exist, selective citation is a danger. I don't think we should deprecate primary sources (since they're so vital to explaining any medical topic), but I do think we should insist on something along the lines of, "The use of primary sources (e.g. journal articles) is encouraged on medical topics, but interpretations of these sources should hew carefully to that presented by the authors or by reliable secondary sources such as review articles and medical textbooks." This might discourage the inevitable idiosyncratic usage of primary sources while still encouraging their general inclusion. Thoughts? MastCell ^Talk 15:54, 24 June 2007 (UTC)

"The us of some types (e.g. scientific journal articles) of primary is encouraged" without reference to scientific topics in particular (why not physics, maths or astronomy??) is a better wording to me. Why don't you bring your suggestion to the talk page? It should be noted that by Wikipedia's own definitions, Medical journals articles are neither "original research" ("unpublished facts, arguments, concepts, statements, or theories" or "any unpublished analysis or synthesis of published material that appears to advance a position"), nor are they "unreliable" (reliable sources: "credible published materials with a reliable publication process"), so for the most part, these concerns are unfounded. I don't think they fit Wikipedia's definition of Primary sources either, which refers mostly to historical documents. Examples cited are: "archeological artifacts; photographs; newspaper accounts which contain first-hand material, rather than analysis or commentary of other material; historical documents such as diaries, census results, video or transcripts of surveillance, public hearings, trials, or interviews; tabulated results of surveys or questionnaires; written or recorded notes of laboratory and field experiments or observations;". Furthermore, an example of a secondary source is "[a]n historian's interpretation of the decline of the Roman Empire, or analysis of the historical Jesus," which can certainly be a "primary source" journal articles. Further removed sources, because of their accessibility, continue to have advantages, but it is a Good Thing to be able to credit the originators of novel ideas. Circeus 17:53, 24 June 2007 (UTC)

• I'm a bit puzzled by the idea that the comments in MEDRS "implicitly discourage" PubMed references since PubMed isn't mentioned at all. PubMed indexes medical articles, which may be primary or secondary sources. They may be research, letters, biographies, book reviews, case notes, etc, etc. MEDRS is based on RS (a while back) and I believe the views on Primary and Secondary sources apply to medicine just as much as history or science, for example. Perhaps there is a middle ground were we encourage editors to cite primary (often seminal) papers inline but back this up with review articles/textbooks listed as References where they have been used by the editor to confirm his interpretation (if any) of the primary material.

I feel we should also say something to discourage citing a primary source to confirm a fact that the editor only knows through reading a secondary source. The citation may include the primary source but should also say "as cited by ..." to give details of the actual source used by the editor.

If you read MEDRS, you should see it (attempts to) define what are primary/secondary sources wrt medicine. There most certainly are primary sources in medicine.

I'm pleased that MEDRS is starting to be discussed. It is not ready to become a guideline without much further work and discussion from project members. Should this discussion be moved to MEDRS's talk page? Colin°^Talk 18:21, 24 June 2007 (UTC)

Good idea... I thought of that after I'd already commented here. I'll move over there. MastCell ^Talk 19:40, 24 June 2007 (UTC)

I have been looking at WP:MEDRS, and I would be relieved to have some guidelines such as those that are listed. This would become very relevant to topics in dentistry, such as "new-and-improved" products but especially on fluoride and amalgam. It seems to me that, by far, the most important item in MEDRS is that an article must "present the prevailing medical or scientific consensus." Anything else placed in an article should be labeled as a minority view or one that is not accepted by the established consensus. As long as this principle is followed, then I do not foresee major reliability nor original research problems arising. Secondary sources can be encouraged in the guideline to make certain that medical/scientific consensus is presented, but I think the most important point to emphasize is that (regardless of the source) the content presented in the article, whether held by consensus or a minority viewpoint, must be presented as such. Saying all this, I hope this proposal can eventually be elevated to a guideline with a little work. - Dozenist talk 14:26, 25 June 2007 (UTC)

When dealing with referencing medical articles, I personally rely on the following hierarchy:

1. High-quality review in a high-impact factor core medical journal (e.g. Lancet, NEJM)
2. Several primary studies corrobating each others' results, or a good meta-analysis
3. Primary studies without evidence of a trend/phenomenon - in these cases good consensus on a talkpage is very useful
4. Websites/popular press articles etc

Notability of primary/original research should really be agreed upon by consensus on the talkpage. JFW | T@lk 18:56, 25 June 2007 (UTC)

I think JFW's hierarchy is sound & reasonable. It's fairly easy to demonstrate a general consensus from reliable sources which should trump out of context single references on things. An important adjunct to this list would be referring to the standard current issue medical or surgical textbooks which tend to caputure a general overview at the time of publication. In most areas of medicine, there is little to cause radical reinvention of a field due to publication lag time.

Wikipedia, being what it is, has served as an attractive source for agenda-driven editors on topics like autism, breast implants, gulf-war syndrome, psychotherapy, homosexuality, and others. Dealing with non-mainstream POV proves hard with many such topicsDroliver 14:14, 1 July 2007 (UTC)

Misuse of primary sources

Have a look at this old version of Green tea. The FDA rejection of health benefits can be found here, which, although in the form of a letter, is the result of a serious review of the available evidence. Look how it is dismissed:

"Contradicting the FDA, A 2006 study ..."

I don't know much about green tea or those studies, but heavyweight studies such as the FDA one should not generally be placed lower in the importance-hierarchy than individual research papers. It is a common misconception on WP that primary is better, probably due to the word's other uses in the English language. MEDRS must not give this impression. Colin°^Talk 18:17, 25 June 2007 (UTC)

I've edited the relevant section to try to emphasize this point, which I completely agree with. I would also plan to add something along the lines of, "Individual primary sources should not be cited or juxtaposed to "debunk" the conclusions of reliable secondary sources, unless the primary source itself makes such a claim. Controversies or areas of uncertainty in medicine should be illustrated with secondary sources describing the varying viewpoints." MastCell ^Talk 19:52, 25 June 2007 (UTC)

Textbooks vs. Monographs

To my mind, "undergraduate medical textbooks" are absolutely appropriate as sources for medicine-related articles, and are likely to be more appropriate for an online encyclopaedia than postgraduate ones. I've reverted this change in the main article for now, but reckon it merits a discussion on here... Nmg20 (talk) 16:01, 17 November 2007 (UTC)

My recent change of text (which you reverted) was actually a self-correction. I had mistakenly used the word "textbook" when expanding this page back in Nov 2006. I had not intended to mean undergraduate, college or school books, which is what a textbook is. Such books are usually extremely general in nature (e.g., a whole book on neurology), usually have few references themselves (which makes them a dead-end as far as research is concerned), contain irrelevant question/answer sections for student study, and are not always written by experts in the field (but rather, by experts in teaching, albeit medically qualified experts). I'm sure there are exceptions. My point is that a book whose purpose is teaching (and, at one extreme, exam cramming) is not usually the best source from which to write an encyclopaedia. Would such a book be referred to in a medical review paper? Colin°^Talk 19:17, 17 November 2007 (UTC)

I agree with almost everything you've written, and no, undergrad textbooks wouldn't normally be referred to in medical papers. However - at the risk of stating the obvious - we're not trying to produce medical review papers here, we're trying to provide a detailed-but-accessible summary of available medical information.

Undergraduate textbooks have several advantages from this point of view. They're written relatively simply, they rarely include controversial information (which is not to say that we shouldn't include such info, merely that it can be sourced elsewhere), and they are generally excellent summaries of the currently accepted medical / scientific understanding of whatever the subject they deal with is.

Yes, true. But whether that makes them good sources is another matter. I fully agree that reading such books when researching a medical WP article could be very helpful as you describe. We need to explain things in language that is simple and such textbooks are a step towards lay language, whereas books written for experts can be quite opaque. In fact, I also recommend reading government/charity-produced patient-info pages too as they often cut to the essential facts (from a patients POV) and are good at explaining things. The US Gov text is often also public-domain too! I'd still try to find a better source, however. Try looking at autism or coeliac disease and see how much of that is covered in your undergrad textbooks. When you base your text on someone else, it is much easier to do so (without blatant plagiarism) if it is more detailed than what you aim to produce. The act of condensing the text is one way of putting it into your own words. On controversial topics (like autism) it is very helpful to know the ultimate source of the information. If someone challenges your prevalence figures, for example, you can know how up-to-date the survey was, how big it was, what country it was done in, what population group they studied and also search to see who cites that research to find out if it is respected or has been superseded or rejected. It is very difficult to do that with teaching material, that might just give a figure (because, for a student, that is all that is important). Colin°^Talk 10:59, 18 November 2007 (UTC)

I'd also take issue with the idea that textbooks are rarely written by a specialist in that field. To take three textbooks I myself have used and know are commonly recommended: Obstetrics and Gynaecology is by Lawrence Impey, MRCOG and consultant obstetrician at the John Radcliffe. Neuroanatomy is by Crossman (prof in Anatomy at Manchester) and Neary (professor of neurology). Even the crap-sounding Cardiovascular system at a glance is by Aaronson (reader in pharmacology at GKT/KCL), Ward (professor of respiratory cell physiology at GKT/KCL) and Wiener (professor of medicine and physiology at Johns Hopkins). I'm not sure these are exceptions - do other contributors have views here?

You've got me there. I'm not medically trained. In other fields, the teachers are good teachers, not necessarily good practitioners or researchers. However, I would argue that a book on a big subject, like neurology, written by one or two authors, is going to have chapters where the author is relatively non-expert. One can't be an expert in all of epilepsy, Parkinson's, autism, brain tumours, strokes, .... Even if the book has several hundred pages, it probably only has about a dozen on epilepsy, which effectively shrinks it to a booklet as as source for that article. Colin°^Talk 10:59, 18 November 2007 (UTC)

I agree, however, that exam-question books are not suitable sources, and nor are cramming books - but I think we would lose out by excluding textbooks. Put it this way - people will still be using newspaper articles in medical articles here, and those are far worse secondary sources than medical textbooks... Nmg20 (talk) 23:36, 17 November 2007 (UTC)

Very much agree. This "guideline" needs more work and expansion. I don't think there is a black and white rule or a cut-off point. There is a spectrum of sources and editors should use material as high up the spectrum as they can. Textbooks are better than newspapers, no doubt. Not everyone has access to a university library, and any source is better than no source. Colin°^Talk 10:59, 18 November 2007 (UTC)

I seriously doubt that *any* undergraduate textbook covers Tourette syndrome accurately; I would not want to weaken this guildeine to allow their inclusion. SandyGeorgia (Talk) 16:29, 18 November 2007 (UTC)

I think Colin's "hierarchy" model is a good one. There may certainly be some usable info on Tourette's in an undergrad-level text that we wouldn't want to definitively outlaw; on the other hand, if the undergrad-level text is contradicted by a more specialized medical text, then it's clear which we should emphasize. MastCell ^Talk 20:55, 18 November 2007 (UTC)

I agree with MastCell here - while undergrad textbooks might not cover Tourette's to your satisfaction, SandyGeorgia, I'm betting they'd cover it a hell of a lot better than The Sun/(insert your national rag of choice here)! I agree with Colin's hierarchy too. Nmg20 (talk) 15:03, 29 April 2008 (UTC)

I think we all agree on what WP:V regards as acceptable wrt medical sourcing. However, the change I tried to make here was to clarify what made "an ideal source" or an "excellent secondary source". I'm afraid that undergraduate textbooks are neither, though they still satisfy WP:V. So I'd like to redo that change, if you don't mind, to emphasise what editors on this project consider the best possible sources. In summary: undergraduate textbooks are allowed by WP:V as sources, but our guidelines prefer peer-reviewed review papers and scholarly monographs. Any objections Colin°^Talk 18:40, 29 April 2008 (UTC)

Is there such a thing as an "undergraduate" textbook? If there is, it should be secondary to "non-undergraduate" textbooks as IMHO content aimed at students is simplified to ease understanding, and often does not mention major controversies. I generally find recent review articles in high-profile sources much more useful and detailed than textbooks (especially in fast-moving fields where the textbook is easily outdated). Textbooks also have a habit of being low on useful references. JFW | T@lk 05:37, 30 April 2008 (UTC)

By "undergraduate textbook", I mean books bought and read while studying to be qualified in a discipline (medicine). But a general definition of any textbook is a book aimed at a student (whether at school or postgraduate). By monograph (which are usually multi-author these days), I mean a book that a consultant would have on his or her shelf (written by experts for experts). Do you have a different definition or usage of those words? Colin°^Talk 07:55, 30 April 2008 (UTC)

No, I think we agree on the definitions. But my point was that undergraduate textbooks still eliminate important content because it has little bearing on the general knowledge required of a medical student, while being highly relevant when discussing a condition in more detail. Monographs can be just as outdated as textbooks, but to me seem to be more detailed. JFW | T@lk 11:36, 30 April 2008 (UTC)

To my mind, an "undergraduate textbook" is one with something in the title which makes clear that it's for undergraduate (pre-clinical or clinical) medical students. It is not any textbook used by undergraduates, some of which are detailed, in-depth discussions of topics which are directly relevant to medical theory but not always practice.

By way of example, I'd be surprised if many consultants have a copy of Ganong (ISBN 978-0071440400) or Kumar & Clark (ISBN 978-0702027635) on their bookshelves, but both to my mind would make first-rate sources, and are liable to be better updated and more widely read than most monographs. That's not to undermine the importance of monographs, merely to say that they tend to be more specific and/or more clinical, and so will not be suitable as references for all parts of an article.

This is probably a long way round of saying "I agree", but if what you're saying is we shouldn't be allowing the Crash Course books (e.g. ISBN 978-0723433507), I agree; if what you're saying is we shouldn't allow Ganong or Kumar & Clark, I don't! Nmg20 (talk) 13:22, 1 May 2008 (UTC)

Ok we agree. But for a given medical article, there's probably only a page or two that is relevant out of the 1000+ pages in those textbooks, and from that you might be able to use it to source a sentence or two on basic body/medicine facts. To write the meat of an article, you need a much more specific reference. Lots of them. That's why I didn't want to recommend them, but neither do I want to exclude them. Colin°^Talk 17:11, 1 May 2008 (UTC)

In the absence of secondary sources...

In some of my recent article work (e.g. subarachnoid hemorrhage, Wilson's disease, ascending cholangitis) I have found that even the best recent clinical reviews are still short of information, especially on the softer areas like quality of life and prognosis. I find myself reaching for primary sources to complement the main reviews, but I remain concerned that we are opening ourselves up to WP:SYNTH. Are there any views on this? JFW | T@lk 15:10, 18 June 2008 (UTC)

I've looked over those articles, and I'm not sure which parts particularly you mean - can you give an example? Nmg20 (talk) 09:18, 19 June 2008 (UTC)

My point is that reviews sometimes don't cover the points that you'd really want to address. For instance, on SAH I wanted to mention the fact that many people with previous SAH have persistent headaches. The only evidence for this could be found in a primary research study that definitely addressed the quesiton, but is of inferior strength on our "hierarchy" of sources. JFW | T@lk 10:13, 19 June 2008 (UTC)

RFC:Are "primary studies" not secondary sources for information on prior studies?

(Please skip down to the next sub-section for another succinct introduction.)

I think there's a ubiquitous misunderstanding of the word secondary reflected on this article. A primary study generally reviews and discusses their findings in light of prior evidence. This makes it a secondary source for information on these articles. I've noted this with a RfC at WT:NOR here, and also over at Talk:Coeliac disease#Misunderstanding of secondary in the context of primary studies and reviews. The question is: is a reviewer necessarily more credible to comment on the prior science than a researcher discussing a primary study, all else equal? I don't think so -- although there may be a small bias, I don't think the reviewer should be considered immune to these biases. Now, systematic reviews help to eliminate bias by forcing the reviewer to be precise and evaluate all studies -- but these are uncommon, and still susceptible to bias. In summary, more importance should probably be placed on the date of the publication and the comprehensiveness with which it approaches a topic. Very broad reviews are likely to miss important details which specialized papers will discuss. ImpIn | (t - c) 06:44, 28 June 2008 (UTC)

Primary studies are often very selective in the sources they cite when approaching their present subject. The simple reason is that primary studies often deal with a specific aspect of a disease or phenomenon. If "important details" are not mentioned in high-quality reviews, are they relevant for a general purpose encyclopedia? JFW | T@lk 07:23, 29 June 2008 (UTC)

Do you admit the difference between a review and a systematic review? How do you judge what's a quality review? This review is published by the BMJ, but it does not seem high-quality. It is so broad that it offers little, if any, analysis or explanation for why it selected the studies it did. You seem to have this obsession with being mentioned in a review -- but if that review does not explain why it selected the studies it did, it should not be taken as bestowing more credibility on that particular study. The selective citation and discussion of primary studies is an asset when you're trying to report on scientific analysis. II | (t - c) 01:43, 30 June 2008 (UTC)

Could you please WP:CIVIL and WP:AGF? I take offense at your words "you seem to be have this obsession", while all I'm actually doing is explaining this guideline to you.

Many reviews are quasi-systematic in the sense that they profess to be based on MEDLINE search on the topic with selection happening only on basis of space considerations. The quality of a review can rapidly be judged in a way you did. If it doesn't go about the subject systematically but instead blasts you with useless information it is a bad review. PMJ (not the BMJ, although they have the same publisher) has a habit of publishing reviews of borderline quality IMHO.

Obviously your question needs more eyes than just mine, so let's see what others say. In my view, reviews are better sources of information than primary studies because they weed out what is not important and place what is important in context. In certain instances, I'm sure it is reasonable to use primary research studies as sources if they are better (in editorial terms) than extant reviews; for instance, on subarachnoid hemorrhage I used the ISAT study not just for its results but also for some general knowledge on the typical approach to intracranial aneurysms; that is only because I thought it adequately reflects current practice and the knowledge could not easily be found in other sources, such as the VanGijn and Suarez reviews that I used. Otherwise, reviews normally do have preference for the reasons I gave. JFW | T@lk 05:37, 30 June 2008 (UTC)

Secondary sources are preferable to primary ones, even if we're talking about the "previous work" area of primary sources. As a practical matter, when a secondary source is reviewed, its reviewers check more carefully that it's comprehensive, neutral, etc. For a primary source, reviewers concentrate on the new results being reported, and tend to treat the previous-work section less carefully. Generally speaking, primary sources try to make a point and to advance research in a particular area, and are more prone to list previous work that agrees with them, and are less prone to list other sources that disagree; whereas secondary sources are trying to cover a topic more generally and fairly, and are a much better way to achieve NPOV. Of course, this is just a tendency, and one can find bad secondary sources and good primary ones; but it is a strong tendency and should not be ignored. Eubulides (talk) 10:36, 30 June 2008 (UTC)

Sorry for the obsession comment, JFW. I really appreciate that you've taken the time to discuss this with me. I don't think the fact that some finding didn't get picked up by a review is evidence that it is not notable or should not be included in an article. Similarly, your "wait till it gets mentioned in a review", which you said about the persistent symptoms study (diff) and the casein study, strikes me as strange. Do you mean wait until it is evaluated in a review? What makes a review better at evaluating than a primary article? It's the extent of the evaluation and similar studies which matters, not the "review", which whether it is mentioned in a review, which is just a label. Note that that small study is mentioned in the excellent systematic review PMID 18315587 (I believe it is free access, but the link doesn't seem to be working). Let's say 5 years go by and no review picks up on the casein study, but a couple other studies are published finding the same result. Does that mean it is not significant, or that it should not be included in the article? I don't think so. But it is entirely plausible that 5 years will go by with no reviews picking up on that study. And even if a review did, if no other studies were done when it did, it would simply be mentioning the study. How does that effect its validity? I don't agree with the general censorship attitude which is spreading across Wikipedia. MEDRS says that primary studies have less weight; quite right. But several primary studies have more weight, whether they're mentioned in reviews or not. And even only 1 study on a particular subject doesn't mean it shouldn't be included in the article -- nor does MEDRS say that, although people persist in trying to say that such a position is necessary. Note: part of this is discussion is over whether primary sources are in fact only primary. Eubulides seems to skirt over this and assume that they are. However, it is clear that a primary source's review of previous literature is secondary, just as a review or a newspaper article is secondary. It is removed from the source itself. As far as pushing a point, most reviews are published by people who have published primary articles, hopefully several. Thus they are not immune from bias. II | (t - c)

I disagree that primary sources can be treated as secondary sources for the purpose of WP:MEDRS. I reverted the change you made along those lines without previous discussion. Such an important change to these guidelines should be discussed in detail before installing, with a reasonable consensus that they're a good idea. Please see the "as a practical matter" comments above for why secondary sources should be preferred to previous-work sections of primary sources. Eubulides (talk) 11:23, 30 June 2008 (UTC)

Please explain to me how the popular press can be considered "secondary" when covering scientific literature, but a scientific article commenting on literature is not. I had a RfC up on this at WT:NOR for over a week. I dropped noted on TimVickers and SandyGeorgia's talk pages, but they declined to comment (WP:SILENCE). Note that my addition did express a preference for reviews first and "if necessary primary articles". Note that the current version states that (by definition secondary) popular press and press releases can be cited in articles. Considering all these facts, I find it pretty surprising that you're opposing this change, and I'd like you to be clearer on why. You've stated your bias position, but that doesn't really make sense; all scientists have biases, reviewers included. We just have to trust them. Do you have any other reasons? Also, your above comment refers to reviews as if they are secondary sources and primary articles are not:

Generally speaking, primary sources try to make a point and to advance research in a particular area, and are more prone to list previous work that agrees with them, and are less prone to list other sources that disagree; whereas secondary sources are trying to cover a topic more generally and fairly, and are a much better way to achieve NPOV.

This makes me wonder if you're following my argument. The definition of a secondary source is simple: the person is discussing a source besides herself. Under this definition, a primary article discussing previous literature is, by logical definition, a secondary source. II | (t - c) 11:56, 30 June 2008 (UTC)

• The statement "all scientists have biases, reviewers included. We just have to trust them" implies that we should trust individual studies as much as we trust reviews. This implication is incorrect. We trust reviews more, and there is good reason for this.
• I understand and disagree with your argument that the previous-work sections of a primary source should be given more weight than the rest of the primary source. Please see my comments below for more on reviews vs. primary studies.

Eubulides (talk) 19:24, 30 June 2008 (UTC)

Eubulides, you've got me confused. I fully support your opinion that the mini review of previous work (which often extends beyond the research focus into other background information about the condition or treatment) can be inferior to a dedicated literature review article. However, I do agree with ImperfectlyInformed that that section of such a paper is a secondary source. Can we find some way of stating this while also informing the reader of its place in the hierarchy of quality. Colin°^Talk 12:54, 30 June 2008 (UTC)

• My main objection to calling the previous-work sections of primary sources "secondary sources" for the purpose of WP:MEDRS is that this would elevate the presumed reliability of previous-work sections to be above that of the primary sources themselves.
• Suppose, for example, we have two competing research groups, a "mainstream" group and a "fringe" group. Each group publishes a primary source; the mainstream source's previous-work section cites only mainstream work, whereas the fringe source's previous-work section cites both mainstream and fringe work, putting fringe work in the best possible light. If these previous-work sections are both considered to be "secondary sources", the fringe source would trump the mainstream source for the purpose of WP:MEDRS, but not vice versa; and this would contradict WP:SOURCES, a core Wikipedia policy.
• I am not opposed to mentioning the previous-work sections, but they should not be put at the same level as reviews, as they are often biased in favor of the primary study in question, and in that sense are closer to the reliability of a primary source than to the reliability of a review. If a review is available it should be used in preference to the previous-work section of a primary source.
• Here is a quick first cut at some wording: "Technically, the previous-work section of a primary source is a secondary source, but it is written from the point of view of the primary source and generally speaking it is not more reliable than the rest of the primary source is; so for the purpose of WP:MEDRS it can be treated as having the reliability of its containing primary source." Maybe we could put it in a footnote, as this is a bit of a distraction from the main text.

Eubulides (talk) 19:24, 30 June 2008 (UTC)

No, this just makes things more confusing. There's nothing "technically" about it. The previous-work section cites primary sources and is one step removed from the original research it reviews. It is a secondary source and I can see no reason to treat it as having the "reliability" of its containing primary source [which vary in quality from case notes through to meta-analysis and large-scale studies] We have no problem labelling press releases and newspaper articles as poor secondary sources, and the previous-work section surely stands well above them, in general.

I don't think the use of previous-work sections is uncommon enough to relegate this to a footnote. I'm also conscious that editors come here nursing battle wounds. There are many areas where utterly uncontroversial information could be sourced to weaker material if that is all an editor has available -- we shouldn't allow some wikilawyer to remove text that more than meets WP:V just because it isn't sourced to a Cochrane review.

I suspect we need guidance how just how much to trust, repeat and generalise material taken from the Conclusion section of a primary source. There are issues with synthesis, undue weight and reliability. Some conclusions are very guarded, others make claims that repeated out-of-context (i.e., to be read by non-specialists unaware of just how limited the study was) would appear to give them much more weight than they deserve. Colin°^Talk 20:33, 30 June 2008 (UTC)

• It is a secondary source and I can see no reason to treat it as having the "reliability" of its containing primary source. Yes, and that was the point I was trying to make, albeit in a hurry and with suboptimal words. (Sorry, I misinterpreted your statement: let me rephrase:) I agree that a source will have more reliability for some points than for others, and that the previous-work section of a primary source is often of higher quality than the rest of the source (obviously there will be some counterexamples). However, generally speaking a previous-work section is less reliable than a review (obviously there will be counterexamples here, too). I agree that the previous-work section of a high-quality primary source would normally stand above a press release, for reliability purposes.
• I also agree that material shouldn't be removed merely because it is supported only by the previous-work section of a primary source.
• What I'm worried about is that a previous-work section of a primary source will be used to dispute a reliable review, under the argument that we have two secondary sources that disagree and that both sides should be covered evenly. That wouldn't be right. All other things being equal, a review should have considerably more weight than the previous-work section of a primary study.
• I agree that advice about conclusions of primary studies would be helpful. I'm afraid that many primary studies are worse than what you've described, in that their conclusions are not justified by their results. Here's an example I discussed recently in Talk:Chiropractic: it was proposed to cite Rubinstein et al. 2007 (PMID 17693331), who state in their conclusion that "the benefits of chiropractic care for neck pain seem to outweigh the potential risks", despite the fact that their primary study presented no risk-benefit model of any kind, and had no control group (which would have been necessary to measure any risk or benefit due to chiropractic care; see this discussion). I'm afraid that conclusions of this quality are all too common; Wikipedia articles should not be relying on them.

Eubulides (talk) 21:20, 30 June 2008 (UTC)

Good, we seem to be in agreement after all. Perhaps we should take the guidance on primary-study-conclusions to a new section. Surely if the journal was prepared to accept a study with a conclusion unjustified by the results or method, then it would also be lax enough to accept a later review that failed to point out the failings in that study? Are we relying on merely the "tendency" for reviews to pick these things up (or ignore the crap)? In the section preceding this, JFW has the problem where his secondary sources just don't cover the material needed to make a rounded encyclopaedic article. Should we, in this case, rely on good editor judgement in the selection of primary material, and if challenged fall back on requiring good secondary sources? What about the combination of sourcing some text to both the primary research and secondary review, where we rely on the primary article for extra detail but ensure any (contentious) claims are backed up by the review? Colin°^Talk 22:18, 30 June 2008 (UTC)

• You're right that that paper does raise some questions about the editing of the Journal of Manipulative and Physiological Therapeutics. The first two authors of that paper are on the JMPT's editorial board, by the way.
• I think we are relying on the tendency of higher-quality science to be more serious about reliable reviews than lower-quality science is.
• If no reliable reviews are available for a topic, that's an indication that the topic isn't that notable.
• If a reliable review ignores a primary source that is on-topic, that's a sign that the primary source is not notable.
• In some cases there may be high-quality primary studies that are too new to be reviewed, and which should be included: but we should be quite careful about citing such studies, for obvious reasons. If published in the Lancet or Science that's one thing; if published in the Journal of the Australasian College of Nutrition & Environmental Medicine (as, for example, this claim that wireless electronics cause autism), that's another matter entirely: such studies should be treated very carefully in Wikipedia, and should not be given much weight.
• If there is a significant dispute among editors about whether primary sources should be used, I'd say the default should be to omit them. There should be a compelling reason to cite a primary source when secondary sources are available.
• If it matters, I'm unfamiliar with the JFW case, and my comments above are not based on any understanding of it other than what has appeared in this thread.

Eubulides (talk) 00:00, 1 July 2008 (UTC)

ImperfectlyInformed, without wanting to distract this discussion into specifics of coeliac disease, the "persistent symptoms study" appears to offer, for consideration by the professional reader of the journal, suggested changes to clinical guidance. Out "Treatment" section should not be based on speculative conclusions by the authors of trials of less than forty people on a significant disease. As with any responsible physician, we should look to professional clinical guidelines, systematic evidence-based reviews or reports of consensus opinion by specialists in field. That's the ideal, of course. If you are writing about an extremely rare disease, then the consensus opinion of experts might well be based on nothing better than a handful of case reports. Coeliac disease doesn't fit that category. Colin°^Talk 12:54, 30 June 2008 (UTC)

Colin, I believe the "if necessary primary articles" conveys the fact that reviews should be preferred over primary article discussion sections. Also, I'm aware that that particular study doesn't have a lot of weight; there's a reason I didn't try to add it back in again. It's an example. When JFW removed it, he said it might be added if it was cited in a review. It was cited in the high-quality 2008 systematic review. Does that mean it can be added? Based on the reasoning I've heard from JFW, and displayed in that edit summary, yes. I don't really see why though. Being mentioned in a review, as I've stated, does not make a study more valid. Being replicated makes a study more valid, whether these replications are picked up in a review or not. By the way, that the casein study is another example of a different food allergy appearing in coeliac patients -- a confirmation of sorts of the speculation in that earlier study. My reservations of adding it to the article would more center around its technical language, however. Also, although I'm no medical researcher, selective citation of studies in reviews (not systematic reviews) appears common; just now over I've noticed that different 2002, 2004 review, and PMID 17916948 (2007) reviews on selenium and mercury all cite different studies. Could use your input on our discussion over at mercury poisoning. They all come to basically the same conclusions, though: that there is a strong relationship between selenium and mercury toxicity which needs to be investigated further. The 2002 review says the epidemiological evidence is weak, and the 2004 review makes some comments on why (whale meat as opposed to fish). I've noticed the same selective citations in the coeliac disease article. I've noticed the same in nutrition reviews ([9]). II | (t - c) 13:35, 30 June 2008 (UTC)

It should be immediately obvious from my above postings that I cannot be holding the position you are suggesting. Being mentioned in a review is of course not a selection criterion; every single study could be included. Again: editorial judgement should be used here. If the review continues to accord to the study some epithets like "landmark trial", "pivotal study" etc then it is more likely to be suitable for inclusion.

I'm really getting the feeling that we are descending into Wikilawyering all because of my (and others') specific attempts to discourage violations of WP:WEIGHT, WP:SYNTH and other content guidelines. I will not easily be persuaded on this matter. That is not "general censorship attitude which is spreading across Wikipedia". That is an attempt to build an encyclopedia on a solid basis that might not be representing the very latest research but will be providing a reliable general picture of the subject.

I'm not expecting to be offering further comments unless new arguments are advanced that have the potential of displacing comments made by Eubulides, Colin and myself. And a quick word of advice: never use the C word unless there is an actual case of censorship: it makes you acutely unpopular. JFW | T@lk 13:55, 30 June 2008 (UTC)

I think you're assuming bad faith. There are scientific facts which are being suppressed for a pedantic reason i.e that they are not cited in a review. The fact is that a review focused upon casein in wheat will probably never happen -- thus the likely sensitivity of coeliac patients to casein will never be mentioned. The strong finding of budenoside is similar; that study may not be replicated for another few years. The longstanding (7 years?) misunderstanding of the word secondary is not a minor issue. Nor is it minor that you seem to place a greater emphasis on a study's mention in "a high-quality review" (based on a wiki editor's opinion) than on replications. You're fine with a lot of behind the scenes editorial work in interpreting reviews which are high-quality, but when citing scientific facts stated in plain language, you seem to think it is verboten. I don't think that makes sense. The former seems more questionable to me than the latter. Interesting findings should be reported, and similar studies can be reported alongside. That's not SYNTH, that's just pointing to studies. For example, at least 4 studies have been done showing Se reducing the toxicity of MeHg (methylmercury), with 1 exception. None of these are all cited in one review; they are cited in different reviews, and the most recent (2007) in perhaps none. I think stating that "Several studies have found that Se reduces the toxicity of MeHg in rats, with an exception"[footnotes] makes sense. You apparently do not.

Also, your wording makes it unclear: when exactly could I cite a study such as the casein one? Should it be replicated once? After there's a systematic review on casein and coeliac patients? After that one study is mentioned in a review? How about the budenoside study? There is nothing in MEDRS which says you cannot cite primary studies, especially remarkable ones like these. You're actually pushing for a policy which does not exist. The current policy even says that popular press articles can sometimes be cited, and here you're fighting tooth and tail against the addition of remarkable primary studies.

Note: I stand by my censorship comment; whether the censorship is intentional or not, it amounts to censorship. In case you haven't noticed, I'm not at Wikipedia to win popularity contests. You're appealing to your own fictitious policy to keep interesting, encyclopedia worthy-content out of the encyclopedia. Further, this impels other people to do the same, and allows people to justify censorship (recent example). Wikipedia is not conservativopedia; if Einstein had published his paper on Relativity today, we would not want to "wait until it is verified" to note it. There's no reason for that position. There's no rule that a study has to be replicated "or noted in a high-quality review" before it gets noted on Wikipedia. Sure, reviews get greater weight, but when they aren't available, individual studies are citable. Somehow MEDRS even allows for popular press and press releases, as well. II | (t - c) 14:26, 30 June 2008 (UTC)

ImperfectlyInformed, you are focussing too much on whether "in a review" is a necessary and sufficient condition for inclusion in a WP article. Also, the use of press releases and newspaper articles "in some contexts" doesn't really go into just how limited those contexts are. They certainly aren't appropriate for medical facts or descriptions of clinical practice. Your opinion that "Interesting findings should be reported" just doesn't work for WP. Every journal article ever published is or was interesting to someone. WP is not a news magazine or a blog of the latest developments in X. Your other suggestion that "similar studies can be reported alongside" sounds dangerously close to you performing your own meta-analysis of primary research. Don't go there.

If researchers and physician's aren't writing about or following up on research that you consider important, then the problem really lies with them, not WP. The policy requirements force us to only document what other people think is significant. Colin°^Talk 16:55, 30 June 2008 (UTC)

Err... I am? I'm arguing that it is not be a necessary and sufficient condition; JFW is arguing that it is. I haven't tried to cite using press releases, I'm just pointing out that they can be allowed. Clearly, then, small primary studies can also be cited. These journal articles are more than interesting -- they are highly valuable to patients. The casein study (2007) found that 50% of the coeliac sample reacted to casein. The budenoside found that 50% of those with refractory coeliac disease recovered. These can have life-changing impacts on people. It's not simple news; it is compelling, valuable breaking scientific research, which should be noted in the article. Also, did you glance at the change which I put in? Do you see anything wrong with it? Also, the idea that reporting similar studies alongside each other is a meta-analysis -- well, I certainly disagree. II | (t - c) 17:36, 30 June 2008 (UTC)

• I disagree that we should be citing small primary studies with a high frequency. That sort of editing would greatly decrease the reliability of the information contained in Wikipedia. Wikipedia should emphasize reliable reviews, and should cite primary studies over reviews only for very good reasons, reasons that I have not seen for any of the changes being proposed here.
• I agree with JFW that accusations of "censorship" are out of line here. As are charges of "assuming bad faith". I see no justification for such heated rhetoric here.

Eubulides (talk) 19:24, 30 June 2008 (UTC)

This issue comes up regularly. I feel very strongly that articles need to be based on secondary sources (high-impact review articles, major textbooks, summary/guideline statements from major respected medical organizations) rather than individual primary studies wherever possible, especially in areas with any sort of controversy. We should absolutely cite relevant primary studies, but they should not fundamentally drive coverage, particularly of controversial issues.

Here's why: it's trivially easy for anyone with slight sophistication to mine the primary "reliable" medical literature to advance whatever editorial point they like. I'm thinking of creating an article claiming that HIV cannot possibly be the cause of AIDS, sourced entirely to "reliable", Pubmed-indexed, peer-reviewed publications. It's easy with selective citation, and the only real defense is common sense - an editor's selection and presentation of primary medical studies should never contradict, supercede, or ignore syntheses by reliable third-party sources. MastCell ^Talk 19:13, 1 July 2008 (UTC)

Sorry, let's start over and take things one thing at a time -- primary articles are secondary sources

Colin has admitted, as is obvious, that the "mini-reviews" in primary articles are secondary sources. To be precise in our language, secondary should not be used as a synonym for reviews. Here is what I attempted to add (diff 1, diff 2). The final text looks as follows: (please read slowly and specifically point towards problems in the addition, and evidence supporting your conclusions if possible)

A secondary source in medicine summarizes one or more primary or secondary sources, usually to give an overview of the current understanding of a medical topic. Review articles and specialist textbooks are examples of secondary sources. A good secondary source from a reputable publisher will be written by an expert in the field and be editorially or peer reviewed. Journalists writing in the popular press, and marketing departments who issue press releases, tend to write poorer secondary source material; however, such material may be appropriate for inclusion in some contexts. (Begin addition) Primary research articles can also be secondary sources of prior literature, and are superior to the popular press in this respect. The best secondary sources are systematic reviews, which look at all available evidence on a particular topic and justify the inclusion and exclusion of evidence. After systematic reviews, preference should be given to the most up-to-date reviews and if necessary primary articles which discuss the largest range of evidence on a particular subject in the most non-technical, analytical manner.

(My addition in red.)

Eubulides reverted this. He stated that he disagreed that primary articles can be secondary sources. However, that primary articles are secondary sources is a fact, just like gravity is a fact. I pointed out that the popular press is a citable secondary source and requested that he explain how the citation and discussion of a primary study is not. He has not answered this question. My edit ranks the sources as follows: 1) systematic reviews, 2) reviews, 3) discussion in primary articles, 4) popular press. We can address the particular issue of citing primary articles separately. Let's discuss the issues with this edit right now. II | (t - c) 23:35, 30 June 2008 (UTC)

• As mentioned above, the main problem with this addition is that the resulting text would suggest that the previous-work parts of primary sources are preferable to primary sources themselves. The result would be a net minus compared to how the page is now.

• I disagree with the blanket contentions that the previous-work section of a research article is superior to the popular press, that systematic reviews are the best secondary sources, that an older systematic review is better than a newer general review, and that primary sources that use "the most non-technical, analytic manner" (what's that?) are best. Some of these contentions are reasonable in many cases but not all cases; others are more-debatable. In short, a good chunk of the advice here is too dogmatic, and in some cases perhaps even wrong.
• The proposed addition also has organizational problems. It would cause the discussion to jump from reviews to popular press to research articles, then back to reviews, then a mention of "up-to-date" that seemingly applies only to reviews, and then back to primary sources.
• The most important problem, though, is the proposed elevation of previous-work sections of primary sources. We are attempting to discuss the issues involved, which are not trivial, in #Are "primary studies" not secondary sources for information on prior studies?. Please be patient as the discussion goes on, and give it time to evolve.

Eubulides (talk) 00:00, 1 July 2008 (UTC)

Secondary sources from the popular press tend to be very poor. I am actually in favour of placing significant restrictions on these. Just look at the fracas we've had over number needed to treat and that Business Week article that criticised studies (e.g. ASCOT) without mentioning their name and journal of publication... JFW | T@lk 06:31, 1 July 2008 (UTC)

I'm not familar with those fracases, but I agree about the popular press. I prefer citing popular-press articles in the laysummary= parameter of Template:Cite journal, so as not to emphasize them unduly over the more-reliable sources. Also, unless I find a really first-class popular press article (such as the New York Times) I tend to prefer press releases put out by the sources' authors, as these tend to reflect the sources better than the popular-press articles do. Something like this, for example:

• Johnson CP, Myers SM, Council on Children with Disabilities (2007). "Identification and evaluation of children with autism spectrum disorders". Pediatrics. 120 (5): 1183–215. doi:10.1542/peds.2007-2361. PMID 17967920. {{cite journal}}: Unknown parameter |laydate= ignored (help); Unknown parameter |laysource= ignored (help); Unknown parameter |laysummary= ignored (help)CS1 maint: multiple names: authors list (link)

Eubulides (talk) 06:43, 1 July 2008 (UTC)

Going down the list (I was going to just interject on each point, but thought you might take offense -- let me know if you want to try that):

• I'm not following your reasoning here. Please elaborate.
• Hmm. Which ones are debatable? I'm opposed to any blanket, categorical statements; however, we have to suggest which things are more reliable or else we've got nothing. Systematic reviews are considered by the medical community to be the highest level. Here at Wikipedia, we should be deferring to the medical community. As far as the most non-technical manner: this applies to primary articles, but I see no reason why it should not apply to all articles. Since Wikipedia is not written by experts, and since we don't know whether people are experts, things have to be written clearly so that we can understand them. Analytical means that, rather than just describing the sources which are being cited, it analyzes their rigor in light of other evidence. This is something that systematic reviews generally do, and general reviews generally do not.
• The section needs to provide an overview of secondary sources. I think it could be reorganized simply by which is generally the most preferable, but I was adapting to what was there. A paragraph break after the introduction of primary articles could be appropriate, since that section is devoted to analyzing what is the most preferred.
• The proposed elevation of the citing from primary articles to a level above popular press is controversial, you say. But JFW seems to disagree with this. I have provided an example below -- in general, I believe that author credentials, readability by a non-expert, and the level of discussion/number of citation in a work are more important than whether it is "primary research" or "a review". Please respond to this point. II | (t - c) 04:54, 2 July 2008 (UTC)

Proposed wording change for primary/secondary

With the above discussion in mind, would the following change make sense? In WP:MEDRS #Article type, change from:

"Research papers are, of course, primary sources."

to:

"Research papers are, of course, primary sources; although they normally contain previous-work sections that are secondary sources, these sections are typically far less reliable than reviews, and should not be used to "debunk" reviews or other primary sources of similar quality."

Eubulides (talk) 20:36, 1 July 2008 (UTC)

Sure. Summaries of prior research presented in the "Introduction" of primary-source papers are not particularly rigorous. MastCell ^Talk 21:08, 1 July 2008 (UTC)

I think the debunk issue is separate, and the problem with using weak sources isn't just about a battle against some other source -- sometimes the weak source material just shouldn't be mentioned. Could we modify the section "Using primary sources to "debunk" the conclusions of secondary sources" to be more generally about using weaker sources or study types to debunk statements drawn from stronger sources and studies. Then give examples such as using a primary study against a review. Colin°^Talk 21:46, 1 July 2008 (UTC)

That sounds like a good idea. I think the most problematic issue is when a weak or primary source is mined to produce undue weight for a specific viewpoint, but a more general phrasing of respecting the hierarchy of sources might be better. MastCell ^Talk 21:48, 1 July 2008 (UTC)

Support for Eubulides' expansion. Agree we should leave debunking aside. JFW | T@lk 22:09, 1 July 2008 (UTC)

OK, then I think we have support for changing "Research papers are, of course, primary sources." to ""Research papers are, of course, primary sources; although they normally contain previous-work sections that are secondary sources, these sections are typically far less reliable than reviews." The debunking rewording will require some more thought, since the article already mentions debunking elsewhere. Eubulides (talk) 05:57, 2 July 2008 (UTC)

I don't understand how/why the introduction/discussion sections of primary articles are categorically "much less reliable" than reviews. In general, what I've read lately suggests the exact opposite. Categorical statements to this effect are misleading. Now, we can all point to examples. I can show you 3-4 poor review articles right now, out of the 5-6 that I've read lately. Many reviews, unfortunately, seem to describe primary articles briefly rather than analyzing them. The reality is that reliability is, as it should be, more connected to the author than the type of publication. This can be assessed by looking at how many papers the author has published on the topic. "Review", "primary article" -- these are simply labels. An actual example: Let's say you've got a 2006 review published by one author with 11 (second author 11) papers which discusses, among other things (as reviews frequently do) Quality of Life (QOL) in coeliac patients. It simply describes a few previous studies, noting generally that studies suggest that women suffer more after diagnosis. In 2007, a primary article on Quality of Life whose main author has published over 200 papers (the second 121, the third 303, the fourth 839), many of them on coeliac disease and several on QOL specifically. These are the premier experts of the field. He discusses the issue in detail in his paper, citing more QOL papers than the review does. He differs (refutes?) with the review, noting that women in western countries "report a lower HRQOL measured by the SF-36 than men". Is he more reliable, as an expert on QOL and coeliac disease? Why would he not be?

In general, you may have a better chance of hearing from the real experts, and hearing their in-depth analysis, in the discussion sections of their papers. The subpar "experts" may be more likely to publish reviews than to do "primary research". And these reviews cover such a wide range that often they just describe them in simple sentences, which offers no value over the article's own abstract. II | (t - c) 04:30, 2 July 2008 (UTC)

At the risk of drawing out this debate: editors are allowed to use editorial judgement in selecting sources, and if you look hard enough you will find primary sources that are better than recent reviews. In this case, there is nothing wrong with discussing this on a case-to-case basis on the article's talkpage and gaining consensus for your actions. Earlier on in this debate I gave a few examples where I personally deviated from this policy, and I wasn't shot down in flames.

I don't think we should be discussing specific sources in specific articles here (the coeliac disease talk page is colossal anyway), but since when do we determine reliability of any source by counting the contributor's number of mentions on PubMed? (I'm sure you will find plenty of casuistry to defend your stance here, so keep it brief if you can.) JFW | T@lk 05:15, 2 July 2008 (UTC)

Well, yes, I'm all for case-by-case studies. Thing is: people look to here, and can stonewall you with a few curt words like "look at MEDRS, primary articles are significantly worse, can't do it". In fact, when I tried to put in the above article, SandyGeorgia objected that we need to use reviews, although I'll admit there was another source she may have been objecting to more the reasoning wasn't laid out as clearly as here. I quoted because this was a direct example of a primary article refuting ("debunking") a review.

You ask when we look at PubMed?? Check out WP:SPS. When using self-published sources from experts, we look at whether the person has published research in the field. For medicine, this usually begins with a glance at their PubMed list. The scientific community measures the number of citations to roughly measure a journals' prestige. Since every article published requires extensive research, and must pass peer review, I think it is a pretty good proxy for how knowledgeable the researcher is. Certainly, as internet wanderers with limited information, we should assume that someone who has published primary research on the subject is better qualified than someone who has not. Do you disagree? Especially if the person has published many articles on a particular subject, as these people have published on coeliac disease. I'd like to hear your reasoning for why we should generally place more weight on the label "review" vrs "primary article" rather than the article's author and content (does it cite lots of sources? does it rigorously analyze those sources? is it understandable by a non-expert?). So far there's been no evidence provided for this general assertion. II | (t - c) 05:28, 2 July 2008 (UTC)

The new wording doesn't say "categorically", it says "typically". I sense that most of the above thread continues a debate that would be better pursued on the talk page of the article in question. It's better to not try to "resolve" such debates by modifying WP:MEDRS to "win" the argument. Eubulides (talk) 05:57, 2 July 2008 (UTC)

Still no reasoning or evidence? Perhaps you should see what the academic community actually says on this issue i.e. cite some sources? The article issues with those sources are done. I brought it in as evidence. Specifically, I have provided a counterexample to the claim that reviews are more reliable (Hauser is not only a published expert on HRQOL, but his primary paper cites more sources). I can produce many, many more examples if necessary, although my library is fairly weak, and it would take time. Further, I have provided reasoning for why primary articles are often superior: the real experts are publishing primary papers rather than reviews, and these primary papers, since they specifically focus on a single issue, often cite more literature on that topic with more rigorous analysis of the validity of the research. You're operating on assumptions. Please provide evidence (or at least reasoning) for your claim that primary articles are less reliable than reviews. II | (t - c) 06:08, 2 July 2008 (UTC)

Again, the new wording says "typically", which allows for counterexamples. My experience is that reviews tend to be much more reliable. See, for example, Abrahams & Geschwind 2008 (PMID 18414403) vs. Cannell 2008 (PMID 17920208). Obviously this is just one example. Overall, though, my experience is quite clear: it's easier to build a bogus case with primary sources, and WP:MEDRS should continue to strongly discourage that sort of thing. I don't sense any consensus to weaken that part of WP:MEDRS; on the contrary, the consensus seems to be to strengthen the cautions against primary sources. Eubulides (talk) 06:48, 2 July 2008 (UTC)

[Outdent]If you cut the "far", then I'd be more inclined to support it. Also, it could be worded better: "Research papers are primary souces, although they are secondary sources in their discussion of prior literature. In this respect they are typically less reliable than reviews because they cover less sources (?)." However, that section is not the right section to be discussing the reliability of different article types. Why don't we have a section focused upon reliability of different article types? Also, that section (/Wikipedia:WikiProject_Medicine/Reliable_sources#Article_type) has a factual error: reviews are more likely to contain original research than systematic reviews. Systematic reviews are highly unlikely to contain original research. Reviews can be variable -- some take a bunch of articles and come to a novel conclusion based upon that literature. Systematic reviews simply analyze the rigorousness overall conclusion of studies on a narrow topic. This error should be fixed.

In fact, this entire article is rather redundant and scattered, and could use some serious copyediting. I'll do some after we resolve this, and we can deal with it per BRD. II | (t - c) 06:59, 2 July 2008 (UTC)

The main goal of this section should be to amplify WP:NOR as it applies to medical topics. It is an obvious violation of the spirit of WP:NOR to mine the medical literature to advance an editorial point out of proportion to its representation among experts in the field. If "primary" sources are being used appropriately, to proportionately illustrate the evidence base for various views, then there's no problem. We shouldn't be excessively prescriptive ("a narrative review always trumps the intro of a research paper, which always trumps the New York Times"), but should instead focus on why it's problematic to mine the primary literature and give general guidance on how to comply with the spirit of WP:NOR, WP:V, and WP:WEIGHT on medical topics.

Using a PubMed count to establish relative reliability is a phenomenally bad idea. It's like using the number of books an author has published to establish how good of a writer they are. David Reardon publishes far more on abortion and mental health than nearly anyone else, but his findings are minoritarian if not discredited in the field. If there is really a head-to-head battle between the findings of two primary articles, then the "referee" should be found in summaries and syntheses of evidence by expert panels, major professional groups, or in review articles published in reputable, high-impact journals. It's not complicated unless we make it so. MastCell ^Talk 19:30, 2 July 2008 (UTC)

Since there is is consensus, I added the proposed text, sans the "debunk" phrase and the "far". I'd rather see the proposed copyediting first on the talk page, before installing it on the project page; what seems like copyediting to one editor can easily look like substantive changes to others. I wholeheartedly agree with MastCell's cautions against primary literature mining. Eubulides (talk) 20:27, 2 July 2008 (UTC)

JAMA

http://jama.ama-assn.org/cgi/content/full/300/1/98 - an excellent set of instructions for people wanting to submit letters to JAMA. I think numerous points in that article are readily applicable to this policy. JFW | T@lk 08:29, 2 July 2008 (UTC)

What academics think about scientific reviews

I suggested to Eubulides that he look at what the academic community says about reviews. I've found some studies. PMID 1834807 (1991) discusses a system used to rate reviews. This could be of significant use for us, as we need to evaluate reviews. It would be interesting to see where this has gone. Related links in PubMed has a vast amount of related articles. PMID 9496383 (1997) finds that most reviews are hardly systematic (this is a bad thing). PMID 10610646 (1999 - free access) finds the same thing. PMID 17606172 (2007) focuses on meta-analysis, but finds improvement. PMID 16277721 (2005) says meta-analyses are generally poor. PMC 1602036 (2006) evaluates Cochrane reviews vrs industry reviews -- obviously, industry reviews are worse. PMID 9092319 (1997) is a guide for finding systematic reviews. PMC 2379630 (1993) specifically compares OR and reviews. It notes that the answers provided by broad reviews should not be accepted uncritically as valid. Conclusion: Certainly, as my original edit to MEDRS reflects, reviews should generally get priority over primary articles -- but people need to recognize the difference in reviews. Most reviews I've seen are not systematic. Here is an example of an overly broad review. These reviews are less reliable than OR in many cases, since they are often both written by an outsider and give cursory attention to many complex issues. Eubulides has argued that systematic reviews should not get priority; this is directly contradicted by the scientists, and does not make good sense. As I stated earlier, it should go: 1) systematic reviews; 2) good, preferably quasi-systematic -- ie a review which states its methods for including literature 3); OR/broad reviews; 4) popular press/press releases. II | (t - c) 10:08, 2 July 2008 (UTC)

I do not support your hierarchy at all. Systematic reviews tend to ask very specific questions (is this drug any good for this indication), often quantitative and often involving meta-analysis. Clinical reviews are qualitative, not quantitative. I don't understand your distinction between (2) and (3): what is OR here - good quality reviews tend not to include much OR unless it reflects clinical experience by aknowledged experts not otherwise documented. (4) is worse than primary sources and should be avoided under all circumstances.

I get a continuing feeling that you are trying to modify the rules because you have very specific ideas about the kind of content you want to include; you are trying to defend your points in various different places (WP:PARENT). Eubulides and MastCells have already explained that the kind of modification you suggest is open to a wide range of abuses. JFW | T@lk 11:19, 2 July 2008 (UTC)

I've not been following all the discussions here, but one obvious way to 'rank' reviews would be by the quality of the journal they are published in, which can be achieved fairly straightforwardly using the impact factor. I see this has been mentioned previously above on this page, and while it's by no means perfect, but it has the advantage of being impartial and relying on the view of the scientific community at large, where deciding whether a review is or is not systematic or "broad" is rather more subjective.Nmg20 (talk) 16:42, 2 July 2008 (UTC)

This is a useful general rule, as much as I hate impact factors. The people asked to write reviews in the New England Journal of Medicine are nearly always widely regarded as at the top of their fields. I wouldn't want to see a battle fought on grounds of "this journal's IF is 7.23534 while that one's is only 5.21234", but as a general rule, the prominence and respectability of the journal carrying the review is a useful metric. MastCell ^Talk 19:35, 2 July 2008 (UTC)

I also dislike impact factors, I suspect a bit more than MastCell does. Although impact factors provide not-too-terrible estimates of journal quality, they are very poor tools for judging quality levels of individual articles; they should at best be just one of the many rules of thumb, and certainly not the most important one, for estimating article quality. In some areas of biology and medicine they are essentially useless. To some extent one needs to know something of the field in order to use them; and if you know that much, you don't really need them. Eubulides (talk) 20:36, 2 July 2008 (UTC)

Instead of impact factors, you could always give precedence to the journal with the shorter name. This is a useful rule of thumb: after all, Cell is great; Journal of Cell Biology is good; American Journal of Cell Biology is OK; and American Journal of the Society for Biological Mechanisms of Cells and Subcellular Organisms is likely not so good. :) MastCell ^Talk 21:21, 2 July 2008 (UTC)

I think the MastCell rule of thumb should be firmly enshrined into this policy. Incidentally, this would pitch Gut before Blood, and these before The Lancet and most certainly N Engl J Med (I mean, FOUR WORDS y'know!) JFW | T@lk 21:49, 2 July 2008 (UTC)

• I like that rule too. It would make RN the most reliable journal in medicine.
• Ooo! Ooo! Let's go the other way. I just now asked Pubmed for actual journals with long names. Any rule that puts the Journal of Clinical and Experimental Psychopathology and Quarterly Review of Psychiatry and Neurology at the bottom of the heap is a good rule in my book.

Eubulides (talk) 22:15, 2 July 2008 (UTC)

Whasswrong with JCEPPQRPN? :) MastCell ^Talk 22:32, 2 July 2008 (UTC)

JFW: you seem to conflate meta-analysis and systematic review. The lead to systematic reviews not misleading, although it is not sourced. Systematic reviews are considered the top in quality in medical science. You're writing contrary to much evidence presented (above). All of them say that "reviews need to be systematic". Otherwise you're at the mercy of prejudices -- you don't know how much they've just grabbed what they want you to hear. Asking specific questions is a good thing in a review, because you can't cover a ton of questions well -- they is just too many studies. As far as your assumptions of bad faith -- well, they are what they are: uncivil assumptions of bad faith. II | (t - c) 23:34, 2 July 2008 (UTC)

I think I was being rather polite while assuming anything about faith. You have succeeded, though, in forging consensus that seems diametrically opposed to your views.

As I said, systematic reviews are good within the scope they address but tend to have a narrow scope. For clinical articles we tend to agree that clinical reviews (even if slightly quasi-systematic in their selection of sources) are better on the whole. It doesn't seem to be broke, and I don't think there are immediate plans for fixing it. JFW | T@lk 00:15, 3 July 2008 (UTC)

The reviews presented above show that, to the contrary, many of the reviews are bad, and the system can be considered somewhat broken. However, these days systematic reviews are being more and more published. They can be found for many of the most important questions, and they should be found. Reviews which do not state their methods for finding research, and justify their exclusion of sources, cannot be considered quasi-systematic or most likely should be not considered high-quality. These reviews are not much better than primary article "discussion" sections. I'm trying to reduce the misunderstanding of what a "high-quality" review is. You're right that I have succeeded in "forging a consensus" against my (and the scientific community's) position. Perhaps it is my tone and approach. It is discouraging, and perhaps Wikipedia is not a good place for me to spend my time.

As far as my wish to include compelling primary articles such as the casein study (which, as I've already pointed out several times, I'm not certain about including because it is technically complex), or the budenoside study (refractory coeliac disease is a debilitating disease; a compelling study like that could offer great value to patients), or the dental amalgam/blood selenium levels (which has been cited in several reviews, and offers a compelling hypothesis for "dental amalgam illness) -- first, despite the small sample sizes all of these present highly stat. significant results far from null. They are strong, if small, studies, and they address highly notable, relevant questions within their respective topics. The issue of including those studies is a somewhat separate issue from the one that I'm working on here, which is to try and reduce the frequent misunderstanding as to what makes a review high-quality. Rather than base my assertions on opinions and assumptions, I've gone to the scientific community, which firmly backs up my position. But alas -- it is no use, apparently. II | (t - c) 00:38, 3 July 2008 (UTC)

• "firmly backs up my position"? I'm afraid that is sheer hyperbole. The blizzard of PMIDs mentioned above doesn't support the argument that primary studies are typically, or even often, more reliable than reviews. What they basically say is that some reviews are good, some not so good. Nobody's disagreeing with that here; but this doesn't contradict what's in WP:MEDRS now.
• One of the sources you mentioned, Hutchison 1993 (PMID 8499790) actually underscores the point that the recent-work sections of primary sources are less reliable. It warns the reader that the results of a sample of primary studies might not be representative, and then going on to say: "Similarly, although the authors of reports of original research often discuss their findings in the light of previous related research, they cannot be counted on to present a comprehensive and balanced review of the relevant literature."
• I haven't looked at the casein stuff, but the "dental amalgam illness" stuff is clearly a minority/fringe viewpoint, and any change to WP:MEDRS would make it easier to emphasize a "compelling hypothesis" like that would encourage violations of WP:REDFLAG and WP:WEIGHT. We already have plenty problems in that area; let's not make things worse.

Eubulides (talk) 01:29, 3 July 2008 (UTC)

The difference might be that the scientific community wants to conduct science while we are just trying to make an encyclopedia. JFW | T@lk 01:15, 3 July 2008 (UTC)

Eubulides: It backs up my assertion that overly broad, non-analytical, and non-systematic (low-quality) reviews are little better than original research. Reviews should be ranked. There are specific, concrete criteria for evaluating reviews, which that paper lists. When a review fails those criteria, it is not much better than an original research article. I suggest that we incorporate the basic review assessments that that article proposes into MEDRS: 1) Is the question clearly defined? 2) Does the review focus on a specific question? 3) Is the author obviously biased? 4) Are the methods used to gather articles described? 5) Are references scanty? 6) Are the primary studies critically appraised? 7) Are there research design and population described? These are a good start in telling people how to analyze reviews. It's not enough to simply say "use high-quality reviews". Distinguishing between high-quality and low-quality reviews is possible, and should be done. I believe low-quality reviews are typically little better than OR, but obviously there's wide variations. II | (t - c) 02:08, 3 July 2008 (UTC)

No, no, absolutely not. We are not going to prioritize reviews on the basis of whether specific Wikipedia editors think the author is "obviously biased". This is starting to strike me as a very elaborate attempt to rewrite general guidelines to win a specific content dispute. Where on Wikipedia is there a problem with "low-quality" reviews inappropriately trumping "high-quality reviews"? MastCell ^Talk 04:32, 3 July 2008 (UTC)

That criteria would be a good one to exclude, as it is too subjective. The more concrete ones are not subjective. Does the review specify its search methods? Does it obviously exclude relevant papers without justification (it is not difficult to find this out)? Does it have an overly broad question? As far as low-quality reviews trumping high-quality reviews -- perhaps not, although there is clear evidence that people do not understand how important and reliable systematic reviews over "reviews"; I have recently added a systematic review to coeliac disease, which was passed over despite being highlighted on the Talk page months ago, and the most reliable systematic review on multiple chemical sensitivity is buried rather than highlighted as it should be. The fact that people are objecting that systematic reviews are not more reliable than "clinical reviews" is evidence as well. Your bad faith assumptions are, again, just that. I will readily admit that these guidelines would not help me add the primary articles that I'd like to add, because those primary papers are not better than the reviews. I'm actually trying to reduce confusion and increase understanding of what makes a review reliable. II | (t - c) 23:27, 3 July 2008 (UTC)

Selection of sources is a matter of editorial judgement. I think most of us are bright enough to spot a rubbish review when we see one. What you are demanding is practically impossible. When there are doubts about the quality of reviews, these can be expressed on the relevant article's talkpage and resolved by consensus.

If you were to apply your seven criteria to the "recent research" sections of primary research articles, how do you think they would perform? JFW | T@lk 08:22, 3 July 2008 (UTC)

I continue to be deeply skeptical of any changes to the guidelines that would encourage editors to use the contents of primary-study articles to debunk reviews. And I continue to disagree with ImperfectlyInformed's characterization of Hutchison. Eubulides (talk) 08:39, 3 July 2008 (UTC)

For info: I am deeply, deeply suspicious of editors who come to Wikipedia, fly in the face of consensus, claim in their posts to be representative of an (unreferenced) "scientific community", and mutter darkly that because the consensus view is not their own perhaps Wikipedia is not a productive place for them to spend time.

The key problem with User:ImperfectlyInformed's claims about review papers is that he proposes no method for assessing what is a good and what is a bad review. While I heartily agree with the criticisms of impact factor above, it has the advantage of being objective. Doing what User:ImperfectlyInformed appears to be suggesting - allowing individual editors to use the review assessment criteria from PMID 8499790 to rank reviews - is something which we as editors should be relying on the scientific press to do. If they get it wrong, so be it, but that's far better than the alternative, which as User:MastCell has pointed out is subjective, editor-by-editor assessment of articles which is fundamentally original research.

Regarding the inclusion of "compelling primary articles", I think the best response is the old adage, 'let repetition make reputation' - these studies might squeak through as suitably disclaimed one-line mentions, but only merit stronger inclusion once they have been repeated a number of times; until that point they can and must be regarded as experimental at best and statistical fluke at worst.Nmg20 (talk) 09:51, 3 July 2008 (UTC)

I think Nick hits the nail on the head here. The more points ImperfectlyInformed makes, the more consensus solidifies to the exact opposite. It is now time for II to either conform to consensus or cease editing articles to which this policy applies. JFW | T@lk 11:04, 3 July 2008 (UTC)

Is consensus now solidifying that systematic reviews are inferior to broad reviews, JFW? Nmg20 -- I just proposed concrete methods, which were actually introduced as a quick way to analyze a review in a paper (PMC 2379630 1993). There is a more sophisticated method in place, although I don't have access to the paper (PMID 1834807). As far as relying on the scientific press -- are there regular reports "ranking" scientific revews? If so, please tell me abuot them. JFW has already endorsed an editorial assessment of reviews, and it really is inevitable -- we have to assess reviews, although preferably we use sources to do it. Here is JFW's quote: "The quality of a review can rapidly be judged in a way you did. If it doesn't go about the subject systematically but instead blasts you with useless information it is a bad review." That's not much different from what I'm saying, except that I'd rather we defined what constitutes a systematic approach. As far as repetition: I heartily agree. I wouldn't want to give these more than a curt mention anyway. And if people oppose, sure, I'll bow to consensus. But I don't think 1 person opposing is consensus. There are other cases where repetition has happened, and a review has not picked up on it, but the most recent primary paper has. For example, 4 papers have found that selenium has reduced mercury toxicity in rats. Is noting that "Several studies have found that selenium reduces the symptoms of mercury toxicity in rats", while linking to the most recent primary paper, a bad approach? II | (t - c) 23:27, 3 July 2008 (UTC)

Yes, it's a bad approach. Editors should not go rooting through the primary literature to cite support for a non-mainstream hypothesis when there is a perfectly reasonable mainstream review that covers the subject in question. Eubulides (talk) 06:26, 4 July 2008 (UTC)

All of the reviews basically support the view that selenium supplementation in rats appears to reduce the symptoms of mercury toxicity. So your assertion that this is a non-mainstream observation seems strange. If it was a non-mainstream observation, then you would have a good point. The lack of epidemiologic evidence can be cited to Watanabe (2002), although it is critically analyzed in the Seychelles reviews (2004) -- this is not the place for the in-depth discussion, of course. II 07:06, 4 July 2008 (UTC)

I agree that this is not the place for in-depth discussion of the topic of "dental amalgam illness"; a better place is Talk:Dental amalgam controversy. Eubulides (talk) 08:54, 4 July 2008 (UTC)

Argh - obviously a general review based on a systematic literature search is better than one based on haphazardly citing all papers you might have in your reference manager. That is plain and simple, and is usually obviously about 20 seconds after you start reading the review.

As for your second point - just citing swathes of recent research papers tends to flood an article with crud.

Where are the other people who oppose consensus? Could you approach them and bring them to this discussion, and otherwise admit an an argumentum ad populum? JFW | T@lk 06:01, 4 July 2008 (UTC)

JFW, it may be obvious to you, but you went to medical school. It may not be obvious to many others. Why not add some hints as to what makes a review high-quality or not to the page? Currently there is confusion about the benefits of systematic reviews, as evidenced by, for example, Eubulides objection that systematic reviews are not better than general reviews. Sure, not all systematic reviews are better than clinical reviews, but there is a reason that systematic reviews are the gold standard in evidence-based medicine. As far as the other people who oppose consensus: you may have misinterpreted me. What consensus are you referring to? I'm referring to consensus on adding certain primary articles. The primary articles which I've proposed including (casein, budenoside, selenium) have each been explicitly opposed by 1 person. With casein/budenoside, you. With selenium, Eubulides. Each of you are free to back up each other, but you haven't so far. Still, I haven't added them. We'll see if other people come along and add their input -- currently there is no consensus to include, but certainly no consensus not to include, either. II 07:06, 4 July 2008 (UTC)

This page is about policy. We want a policy that gives primacy to reviews. You were suggesting that there are people disagreeing with this. JFW | T@lk 08:25, 4 July 2008 (UTC)

I've had a look through PMID 1834807; it seems entirely reasonable although worth pointing out that it's getting on for being two decades old, and so its comments on the state of the medical literature may not be as valid now as they were in '91. I am entirely in favour of these scoring systems being used, just not on wikipedia - that article opens by saying, "We have previously described the development of our criteria, and demonstrated that they can be reliably applied by clinical epidemiologists, clinicians with some methodological training, and by research assistants." The problem with including that in this guidance is that not all editors here fit one of those categories, and as the article's conclusion points out, "Respondents to the sensibility survey felt that the criteria demanded an excessive degree of subjective judgement. We believe the respondents are accurate in identifying this limitation of the criteria. Although we strove to minimize the need for judgement, some judgement is inevitable."

So while this appears to be an excellent article, I'd suggest that it's somewhat dated - several of its criticisms have been taken on board by authors since it was published - and it's intended for use within the research community, not within encyclopaedia editors who may or may not be equipped to score reviews using it.

Unlike Eubulides and Jfdwolff, I would support inclusion of a note like the one you mention about selenium if the studies are from different centres, published in decent journals, and have some sort of scientific rationale behind the findings. Nmg20 (talk) 06:38, 4 July 2008 (UTC)

The problem is that we currently tell Wikipedia editors to use high-quality reviews. We would prefer to have things cited to high-quality reviews, wouldn't we? If we don't give people some guidelines on how to identify high-quality reviews, how will they know? Not everyone who edits medical articles is a medical professional, and apparently even medical professionals need guidelines. As long as the guidelines are not overly complex, I don't see why we can't incorporate them to some degree. II 07:06, 4 July 2008 (UTC)

I don't actually object to giving a short list of examples that would distinguish a good review from a bad one.

Nmg20, I don't think we should be forming consensus here on sources for article content elsewhere. We are discussing general principles. The examples ImperfectlyInformed gives are a typical example of recent work that is yet to be discussed in reviews. These days, all major topics of medicine are reviewed constantly - just look at the number of reviews about a rare condition like pulmonary hypertension that appear even in core journals. It should therefore not take long for a major development to reach the reviews. If it does not, then was it actually a major development? I find this an excellent way of separating chaff from corn, and is the basis of my opinion in this entire discussion. JFW | T@lk 08:17, 4 July 2008 (UTC)

OK, then I nominate the following four citations as examples of good and bad reviews. Just for fun, I won't label which is which:

• Abrahams BS, Geschwind DH (2008). "Advances in autism genetics: on the threshold of a new neurobiology". Nat Rev Genet. 9 (5): 341–55. doi:10.1038/nrg2346. PMID 18414403.
• Ernst E, Canter PH (2006). "A systematic review of systematic reviews of spinal manipulation". J R Soc Med. 99 (4): 192–6. doi:10.1258/jrsm.99.4.192. PMID 16574972.
• Mutter J, Naumann J, Schneider R, Walach H, Haley B (2005). "Mercury and autism: accelerating evidence?" (PDF). Neuro Endocrinol Lett. 26 (5): 439–46. PMID 16264412. Retrieved 2008-07-04.{{cite journal}}: CS1 maint: multiple names: authors list (link)
• Raymond JL, Ralston NVC (2004). "Mercury: selenium interations and health implications" (PDF). Seychelles Med Dent J. 7 (1): 72–6. Retrieved 2008-07-04.

Eubulides (talk) 08:54, 4 July 2008 (UTC)

Oh, Boyd Haley definitely! And do let me know the actual impact factor of the Seychelles Medical and Dental Journal. Does it have one? JFW | T@lk 13:31, 4 July 2008 (UTC)

No fair, you know too much! The SMDJ's impact factor is undefined, since it's not in the impact-factor database. It's not in Pubmed either. Eubulides (talk) 17:11, 4 July 2008 (UTC)

The Cochrane Collaboration has various set of criteria for evaluating studies. Those criteria may be useful models here. Those criteria do not include impact factor of the journal in which the study appears. Setting criteria is a very active area of work, as the best choice of criteria is an open problem. Try a Google search for "site:cochrane.org criteria" to find a slew of conference abstracts. --Una Smith (talk) 15:19, 4 July 2008 (UTC)

Every systematic review and every evidence-based guideline should have a way of ranking the quality of studies (from "high quality meta-analysis" to "consensus amongst experts without support from trials"). The problem is that these criteria invariably apply to original studies, and cannot readily be used to grade reviews.

What ImperfectlyInformed has been suggesting, quite reasonably, is that we look closely at the quality of reviews before using them as references. But there is no way of grading reviews in a way quite like grading primary studies. Until there is a reliable way of doing so, I think editorial judgement is needed, and the normal process of consensus forming applies. JFW | T@lk 22:29, 5 July 2008 (UTC)

I'd like to venture a comment here, even though I'm coming in late on the conversation. it seems to me that the point behind this 3rd-hand, secondary source prescription is that we want to make sure that the views we use reflect a general consensus within a significant portion of a field. reviews as a rule are neither brilliant nor innovative, and it's precisely those lacks that make them useful in WP - they usually reflect a nice run-of-the-mill consensus in the discipline. the 'literature' sections of primary research, by contrast, may (and often do) include recent, innovative, primary research material by other authors doing similar work. there's no question that primary research authors cherry-pick their sources for the purposes of support, criticism, or relevance to their own work. primary researchers are trying to manufacture or influence the current understanding in their field - that's what research is for - and so it's not at all clear to me that primary research will make clear distinctions between the actual current understandings in the field and the author's personal perceptions of what the field should understand.

if Wikipedia has to wait seven years for a result to become fully accepted by the medical community, then Wikipedia should wait seven years. best not to get ahead of scientific consensus... --Ludwigs2 09:39, 7 July 2008 (UTC)

Very well put, thank you Ludwigs2. There will be situations where people will be unable to wait seven years. As long as there are secondary sources supporting the relevance of a new development, we may be able to include this. But those secondary sources need to be blooming good (e.g. editorials or commentaries in core clinical journals). JFW | T@lk 10:15, 7 July 2008 (UTC)

I could go either way on this. Just because something is labeled a "review" doesn't mean that it's really a review, much less than it's a good one. I've seen case studies with a sample size of one labeled as reviews. (I know: several of you are horrified. But it does happen.) I'm also not convinced that when Joe Smith, PhD, writes his "review", that it's materially different from the summary that Joe Smith, PhD, put in his ground-breaking, world-changing paper last month. I think that a substantial level of editor judgment is required here. WhatamIdoing (talk) 06:23, 10 July 2008 (UTC)

Age of sources?

Is there any way to give a rule of thumb for how recent sources should be? How old is too old? Are sources from the mid or early '90s OK? I guess this would vary based on the subject and how fast it's developing. Any advice about how to gauge this? delldot talk 15:29, 10 July 2008 (UTC)

Yes, I guess it would depend on the subject. Generally, the amount of articles published since then give an impression of how fast a field is moving. If there have been hundred or more reviews since an article was published, it is probably obsolete. For rare or controversial subjects with only a few articles available, mid 90s might still be OK. In general, the newer the article is, the better, but you have to take into account other factors as well, such as which journal, seminal article, free full text available, etc. --Steven Fruitsmaak (Reply) 16:29, 10 July 2008 (UTC)

This subject comes up often enough (at least for me, when I am editing articles) that I propose that a section be added for it, after Wikipedia:WikiProject Medicine/Reliable sources #Assess the quality of evidence available. We could call the new subsection Use up-to-date evidence. Below is a draft (which is too long, but I figure it could be trimmed down). Eubulides (talk) 17:08, 10 July 2008 (UTC)

Use up-to-date evidence (draft)

Here are some rules of thumb for keeping an article up-to-date while maintaining the more-important goal of reliability. These guidelines are appropriate for actively-researched areas with many primary sources and several reviews, and may need to be relaxed in areas where little progress is being made and few reviews are being published.

• Look for reviews published in the last five years or so, preferably in the last two or three years. The range of reviews examined should be wide enough to catch at least one full review cycle, containing newer reviews written and published in the light of older ones and of more-recent primary studies.

• Within this range, things can be tricky. Although the most-recent reviews include later research results, do not automatically give more weight to the review that happens to have been published most recently, as this is recentism.

• Prefer recent reviews to older primary sources on the same topic. If recent reviews don't mention an older primary source, the older source is dubious. Conversely, an older primary source that is seminal, replicated, and often-cited in reviews is notable in its own right and can be mentioned in the main text in a context established by reviews. For example, Genetics might mention Darwin's 1859 book On the Origin of Species as part of a discussion supported by recent reviews.

These are just rules of thumb. There are exceptions:

• History sections often cite older work, for obvious reasons.

• Cochrane Library reviews are generally of high quality and are routinely maintained even if their initial publication dates fall outside the above window. For example, this citation:

Proctor ML, Roberts H, Farquhar CM (2001). "Combined oral contraceptive pill (OCP) as treatment for primary dysmenorrhoea". Cochrane Database Syst Rev (4): CD002120. doi:10.1002/14651858.CD002120. PMID 11687142.{{cite journal}}: CS1 maint: multiple names: authors list (link)

is originally dated 2001 but was published in Cochrane Database of Systematic Reviews 2008 Issue 2 with a marker "Status: Unchanged", meaning that as of 2008 they still stood behind the review as not being unduly dated.

(end of draft) Eubulides (talk) 17:08, 10 July 2008 (UTC)

Use up-to-date evidence (comments on draft)

This is great Eubulides! Very helpful. What made you come up with the five year mark? Just curious. Maybe wording should be added in that sentence to emphasize that this is a rule of thumb, e.g. 'about five years'. delldot talk 20:03, 10 July 2008 (UTC)

Generally good (groaning because I got the opposite advice years ago and replaced a lot of review sources with primary sources on Tourette syndrome, and now I'm going to have to spend hours checking and undoing some sources, and often, the review papers are inferior). But ... when a primary research paper is replicated and cited over and over again in every review and is seminal research in the field, and we don't want to lose that source, is it ok to occasionally cite both? In the case of TS, for example, PMID 9651407 is primary research, we have free, full text linked, and it is seminal in the field and widely cited in every important review (there are other examples). I hate to lose the free, full text link in the article. SandyGeorgia (Talk) 20:20, 10 July 2008 (UTC)

Good comments, thanks.

• The five year mark is my own approximation to conservatively catching a full range of review cycles, no matter how dilatory the referees. If something isn't mentioned in a full review cycle, and it's not recent, it's not notable.
• I quite agree that older primary sources are worth citing if sufficiently notable; my personal preference in those cases is to mention the sources directly in the text rather than just in citations, under the theory that if they're so notable that one can violate the usual guideline about preferring reviews to primary sources, then they are notable enough to mention in the main text.
• I tweaked the above draft to try to capture both of these points.

Eubulides (talk) 20:50, 10 July 2008 (UTC)

Well, I've got my work cut out for me; thanks a lot :-))) SandyGeorgia (Talk) 21:00, 10 July 2008 (UTC)

Endorse Eubulides' excellent guidance. JFW | T@lk 22:14, 10 July 2008 (UTC)

Endorse what is a very elegant condensation of the consensus from this page. I would say about primary sources that where, as User:SandyGeorgia points out, you have a primary source which has been replicated (cf above, 'let replications make reputations') and where said primary source has featured in subsequent reviews, it is certainly worth citing. Nmg20 (talk) 22:41, 10 July 2008 (UTC)

Comment Besides wordiness, there are a couple questionable statements. The presumption that if a review does not pick up a primary study, that study is not significant is plainly mistaken. Unless the review justifies the exclusion of the study, it should be assumed that the research done for the review was not as comprehensive as it should have been. Assuming otherwise is original research, and just a bad type of assumption to make. The best reviews look for evidence systematically, and provide information justifying why they excluded the articles they did. The following review provides a table, even noting that many of the articles were reviews -- which is unnecessary, but appreciated.[10] If an article does not do this, it is evidence that the review is just not that great. Also, if a review only mentions or repeats the claims of a study, which are being repeated in the article, without providing any appraisal, criticism, ect., then citing the review as the source of the assertion appears questionable. It might be best to cite the paper and the review, or something, but it is misleading to cite a review for a statement which is actually just being repeated. If a review is making a statement based on several primary studies, that is a different story. I agree with the last two points. II | (t - c) 01:44, 11 July 2008 (UTC)

I can't find much to agree with in your post, but if primary studies are replicated, there should be little problem using them. SandyGeorgia (Talk) 01:56, 11 July 2008 (UTC)

Most of my post is in reaction to this sentence: "And if recent reviews don't mention the older primary source, the older source is dubious." Also, you don't agree that good reviews justify their exclusion of studies? That seems curious. II | (t - c) 02:00, 11 July 2008 (UTC)

We should not encourage the use of primary studies to dispute reviews. The use of an older primary study to dispute a newer review on the same topic is particularly troublesome. There should be no requirement that the review explicitly explain why the primary study was not included, as that'd be a recipe for letting low-quality older primary studies slip in. The proposal is merely a rule of thumb, and obviously there will be exceptions; but exceptions should be few and well-justified. Eubulides (talk) 06:31, 11 July 2008 (UTC)

This guideline is about subjects where there is a large body of research with frequent reviews. If such a situation, omission of a primary paper from reviews is a reasonable indication that is has not made it onto the radar. That sounds like a perfectly reasonable way of doing things. JFW | T@lk 02:26, 11 July 2008 (UTC)

Coeliac disease is an area where there is a (relatively) large body of research. It is one of the more studied areas in gastroenterology at the moment. Nevertheless, the evidence even in coeliac disease is not all that large, but that's just because the evidence in few areas is truly vast. There is no good reason not to create a table justifying your exclusion of studies, or to at least justify the exclusion of studies. There's also no mention in Eubulides' draft of the systematic aspect of reviewing. This review excludes many prominent studies, but does that mean these studies are not a big deal? Of course not. The review is biased. As are other reviews which exclude relevant primary articles without justification. II | (t - c) 02:32, 11 July 2008 (UTC)

• I don't follow the part about "creating a table" and so forth; this isn't mentioned in the draft. Perhaps this is referring to some other discussion about coeliac disease, a discussion that had a table in it?
• The draft is about the tension between reliability (which favors reviews) and keeping up-to-date (which favors primary studies). It is not about reviews vs. systematic reviews vs. meta-analyses vs. etc., which is a different topic.

Eubulides (talk) 06:31, 11 July 2008 (UTC)

I have some significant concerns about this, perhaps mostly because of the way "rules of thumb" turn into sweeping, iron-clad requirements after a few months, and perhaps because I get the feeling that none of you have any connections to people who write reviews and therefore put too much faith in them.

Sure, if you're only working on articles about congestive heart failure and colon cancer and other common conditions, then the concepts here make a great starting point. However, this isn't going to work at all for very rare diseases, where a well-written case study from twenty years ago may actually be your most reliable source. Consider ODDD. I know: you've never heard of it. But go search for oculodentodigital at pubmed.gov, and limit your search to the last five years. You'll get thirty-five (35) papers. The only "review" on the disease in the last five years (as opposed to the genetics and physiology that underlie the disease) is actually a case study involving three patients. It's dated 2004. I don't expect a better review to appear in 2009, or even by 2014. My expectation is based on the fact that there have apparently never been any proper reviews published for this condition. And what is the first thing the editor reads here? "Do not cite primary sources" -- the only sources that exist for this disease.

This also isn't going to work well for many aspects of uncommon diseases. For example: consider some third-string treatment for an uncommon cancer. You've got a twenty-year old paper that gives you a success rate. It's the only randomized controlled study ever done using the specific treatment in this specific cancer. The recent review cites this paper and summarizes the conclusions in two words: "poor prognosis." According to this, the actual survival rate is suddenly not important, because the study was done before the review, and the review doesn't re-report the actual numbers. Is that what you really want? To put an expiration date on data?

I also think that citing any study that is mentioned favorably in recent reviews should be acceptable. For one thing, we get more detailed articles that way. For another, if the original article is retracted, then we know what we need to change. A review that cites Hwang Woo-Suk favorably is not going to be retracted just because the world later discovered that this Korean scientist fabricated much of his stem cell research.

As ImpIn points out, this scheme works poorly in cases where the recent reviews only cover certain aspects of a disease. I frequently see very good reviews in terms of treatment, and that also completely neglect epidemiology. It's hard to find epidemiological information for less developed countries. Sometimes the best we can do is a rather old paper. The fact that an American or European author skips over the prevalence of a disease in Africa or South Asia doesn't mean that this kind of information unimportant for our worldwide encyclopedia: it means that the review is incomplete. In very common diseases, nearly all of the reviews are deliberately incomplete: you'd write a review on a specific aspect or sub-type of hypertension, because otherwise your review would be the length of a book. I won't say that the authors are necessarily biased because of this -- but reviews cannot be assumed to be complete.

Finally, this advice is completely wrong for history sections, for what ought to be perfectly obvious reasons.

Yes, I know: you only meant this to apply to certain "actively-researched areas with hundreds of primary sources and dozens of reviews". But it's not actually that obvious to those who don't already know what you intended to accomplish. The first thing the editor reads is "Do not cite primary sources." As written, I don't think that this communicates what I think you want to say.

I don't mind stating a general preference for recent reviews, although I still prize editor judgement and a good final product over mindless compliance with rules. I could probably support a system of rules like this if it were clearly stated that this guidance only applies to the sections of an article that deal with current practice in diseases where proper reviews are readily available. I might also add that primary papers aren't bad in themselves, so long as they don't actively contradict all of the recent reviews. Fundamentally, I think that if we're going to publish this, then the caveats and restrictions need to go first, not last, and they need to be stated more strongly than the guidance. For example, "Do not cite primary sources..." should be "Consider citing a recent, comprehensive review in a reputable journal instead of older primary sources." The section might begin with the sentence about this advice only applying to articles on actively-researched areas with hundreds of primary sources and dozens of reviews, although the general principles might be applicable in some less common diseases. WhatamIdoing (talk) 02:52, 11 July 2008 (UTC)

Editorial judgement is always required, and as I stated Eubulides' rules of thumb become irrelevant if the most recent review is a case series of three patients. Common sense applies. JFW | T@lk 02:56, 11 July 2008 (UTC)

• I redid the draft to emphasize that the rules are designed for active areas and need to be relaxed in areas where few reviews are available; and they obviously don't apply to history sections. Thanks for mentioning that.
• I agree that lots of reviews are bad. Still, guidelines like these can be helpful as rules of thumb. Novice editors too often ascribe too much weight to sources that are too old, and they too often suffer from WP:RECENTISM.
• The draft doesn't say "Don't cite primary sources" or even "Don't cite primary sources older than reviews." It says "Do not cite primary sources older than recent reviews on the same topic." (emphasis added). This should address some of the concerns raised.
• This is off-topic, but wouldn't Joss et al. 2008 (PMID 17476528) count at least as a literature review for ODDD?

Eubulides (talk) 06:31, 11 July 2008 (UTC)

I'm glad of WhatamIdoing's comments and the changes made. I think we can sometimes concentrate too much on the big diseases that attract controversy and edits from POV pushers. Wrt citing primary sources for studies you wish to comment on, I have found it useful to use the following style:

A study in 1999[citation to study] found that blah blah blah.[citation to review mentioning the study]

In effect, the primary source is being used purely to show the study took place and to act as a footnote for the reader should they wish to read the primary material about the study. The secondary source is used to back up the conclusions of the study. My preference is to restrict the explicit mention of studies (the History section is one obvious example) since if the results of the study are now accepted widely, then they can just be stated as facts. Colin°^Talk 11:22, 11 July 2008 (UTC)

I totally agree with Colin's point about citing both the primary study and a review demonstrating its relevance. This is even more relevant if the study in question was of earthshattering relevance (cue to the ISAT study in subarachnoid hemorrhage, the 4S study in secondary prevention of myocardial infarction, etc). When trials assume that degree of importance in daily practice, it would be ridiculous to omit them from articles.

Endorse new version with the modifications suggested by WhatamIdoing. JFW | T@lk 11:40, 11 July 2008 (UTC)

Colin: I have the unpopular habit of citing many primary sources, and I really don't see why you would cite a review if you mention a study from 1999. That's very confusing for readers.

The reason I mix reviews with a lot of primary sources is because is usually start writing with 2 to 3 review articles or books, UpToDate etc, and for most specific statements (especially high quality evidence, e.g. RCT or meta-analysis in high-ranking journal) I cite the same primary sources as the review cites. This might be "wrong" but I myself would prefer to read an article which cites a lot of primary high-quality evidence rather than a handful of reviews. --Steven Fruitsmaak (Reply) 12:31, 11 July 2008 (UTC)

That approach is often fine, but becomes much harder to sustain in areas of controversy. Often, the controversy is not spelled out in the primary studies, but rather they only cite evidence to support their perspective and ignore the rest. Only a review can reliably enunciate that there is a controversy. Once a primary study has been lent the relevant weight, I can't imagine that there is a problem citing further information from the same study.

If the text is phrased properly, the reader will understand that a primary study is only being quoted because it is independently notable, and the review to support the general picture (including lending notability to the primary source). I think there is much to say for Colin's approach, as simply relying on primary source material opens your contributions to notability questions. JFW | T@lk 13:18, 11 July 2008 (UTC)

I also agree with Colin's point; that style is a perfectly reasonable one, even if it's not my own. I reworded the draft to try to make this more explicit, using the admittedly extreme example of Genetics citing Darwin. Eubulides (talk) 17:49, 11 July 2008 (UTC)

I agree with citing both in many cases, and I said exactly this above: "It might be best to cite the paper and the review, or something, but it is misleading to cite a review for a statement which is actually just being repeated [from a primary study]." I know that the APA parenethical citation style encourages you to cite the original source being cited whenever possible, and I imagine a similar practice is at least somewhat encouraged in footnote referencing, because it's much better for the readers to know the original source of an assertion. As my quote shows, I also agree with Steve -- in less controversial articles, citing key primary articles is the appropriate way to go unless the review is doing some critical appraisal, synthesizing several studies -- and often reviews are not doing critical appraisal, but rather just listing reviews. II | (t - c) 23:33, 11 July 2008 (UTC)

I'm leery of relaxing this requirement to say that it's OK to cite primary studies "in many cases". It should not be a regular thing, done casually; that's too prone to abuse in this context. If a Wikipedia article repeats a key point from a primary study that has not been reviewed or is not particularly notable, that is a danger sign, and the guideline should not be encouraging that sort of sourcing. This is as opposed to primary studies that are notable (influential, replicated, etc.) which are worth mentioning in the main text and then of course cited. Direct citations to less-notable studies is certainly appropriate for bylined articles published in peer-reviewed journals (which is what the APA parenthetical style is designed for), but it is not appropriate for routine use in the (mostly) anonymous articles published in Wikipedia. Eubulides (talk) 01:30, 12 July 2008 (UTC)

I agree; the example I raised (above) has been replicated (at least twice, I think), is mentioned in every current review of TS, and changed the conventional wisdom and thinking about Tourette syndrome. Articles that rely on primary sources an easily be taken over by synthesis, undue weight and recentism. We should be using them in rare, notable instances. SandyGeorgia (Talk) 01:39, 12 July 2008 (UTC)

I like the current version much better. I think it much less likely to be "over-applied" while still clearly communicating ideal practice. A few somewhat random comments:

• ImpIn, the primary reason that reviewers don't list every single study, with a rationale for including or excluding it, is space constraint. You would spend pages and pages just listing articles for celiac disease.
• Steven, I seriously doubt that anyone will be confused by having a statement of fact followed by references to both the primary study and the review that agrees with it. From the reader's perspective, ^[1][2] is no more confusing than just ^[1].
• Eubulides, I haven't read the Joss paper, but from the title and the journal's classification of it as a "short report", it looks like exactly what was being deprecated at the beginning of this conversation: general disease information being summarized in the context of a primary study. In this case, it may well be the best that exists, but it's not the dedicated review that you were advocating for.

From here, there are only two issues left (IMO) that we might want to address: The utility of free/full text versions to our readers [as in, do not delete the ref to a six-year-old free review in favor of a four-year-old review that >95% of readers won't be able to access], and the make-work aspect of keeping things up to date [as in, if the specific information is still entirely accurate, it's not necessary to delete a somewhat older review just to keep the refs within the ideal timeline]. WhatamIdoing (talk) 18:26, 12 July 2008 (UTC)

Thanks, I hope we're converging.

• I agree that freely-readable versions are preferable (all other things being equal) but would prefer putting that in a separate section, as it cuts across the other issues in WP:MEDRS.
• What should be said about make-work? I tried formulating some sentences but came up empty. For example, I wasn't happy with "Don't bother updating an article every time a new review comes out, as this can lead to needless article churn." I'm a fan of keeping articles up-to-date and would not like to discourage editors who work hard at this. Perhaps it's better to say nothing on this topic, and rely on common sense?
• I followed up re Joss et al. by creating Talk:Oculodentodigital syndrome #Joss et al. 2008.

Eubulides (talk) 20:07, 12 July 2008 (UTC)

Your solution for the "freely readable" issue is good. Perhaps the "make-work" issue will wait until we have an actual problem. We can always point editors at the archives if they ask. WhatamIdoing (talk) 06:36, 13 July 2008 (UTC)

WhatamIdoing, "the primary reason that reviewers don't list every single study, with a rationale for including or excluding it, is space constraint. You would spend pages and pages just listing articles for celiac disease." Couple points: good reviews should be focused for this reason, and it is not that hard to cover all the articles once you've made your question specific, because one can group them like "Several studies found such and such"(1-6). The primary reason that most studies do not list all articles is 1) poor research, 2) overly broad focus (look for a more specific reiew), or 3) bias. If you want examples, I've seen plenty. Look at my section above. There are also academic articles which state that reviews with a specific questions are preferred, and the Cochrane reviews follow this guideline as well (browse through them). II | (t - c) 01:20, 13 July 2008 (UTC)

I think we're going to have to agree to disagree here. There are many legitimate reasons to write a general review. A general review cannot reasonably meet your demands about explaining why every single study is or isn't mentioned in it. WhatamIdoing (talk) 06:36, 13 July 2008 (UTC)

I think ImpInf meant that a good review states clearly which kind of study has been excluded. In my experience these exclusion criteria are either because the studies are small or not replicated or because there are space constraints. I'm sure some will disagree, but much of the coeliac disease literature is of rather poor quality, and a good review would do well to separate the chaff from the corn. The same applies to many other conditions, especially those that are easily diagnosed but have protean symptoms. JFW | T@lk 06:42, 13 July 2008 (UTC)

This systematic review of gluten intake for coeliac patients does just that, in table 1.[11] It seems much better to explicitly (and briefly) identify the problems in the studies which are being excluded. Otherwise I'm left with suspicions of bias. Some reviews do explicitly mention their exclusion of studies, and this seems to be a "best practice". I don't think we should automatically conclude that, if a study is not cited in a review, it is dubious. We can't be making these assumptions unless they are stated explicitly; it is original research. Earlier I pointed out that this review [12] was biased. It leaves out a couple critical studies on lithium orotate. Specifically, they cite two studies which I haven't been able to get a hold of in favor of lithium orotate, (note the reliance upon primary articles in that article -- which is necessary). I doubt that this biased exclusion of contradicting papers is all that uncommon, both on the alt. med. side (the latter review is more alt. med) and the mainstream med. side; although perhaps more on the alt. med side. Biased reviews prefer to just pass over contradicting studies without mention. If this is done, it seems to me a "redflag" for bias -- even if a non-confirming study is small, it should be mentioned. One should not simply pretend that contradicting, or non-confirming, studies do not exist, and we should not assume that the researcher who excludes these studies is doing so for a good reason. As far as the problems with overly broad reviews, there is an academic paper which states the same thing as me (PMID 8499790). Broad reviews are good for getting the feel of the literature, but for a good analysis of particular areas, you should be using specific reviews, which can afford to take the time and analyze the literature in-depth. II | (t - c) 20:09, 13 July 2008 (UTC)

I think the debate over review quality has been done to death and there is very little us, as Wikipedian's, can do about improving the literature. We use the best sources we can. Discussions over whether this or that review is biased should be taken to the relevant article's talk page. II, I would take your lecture on what makes a good review, and how we can identify bias more seriously were it not for this diff proudly displayed on your user page. I particularly enjoyed the "Other studies have found that coconut oil can help in weight loss and poison recovery." statement and sourcing. Colin°^Talk 21:01, 13 July 2008 (UTC)

Yes, ImpIn, systematic reviews are supposed to explain their methodology. However, a review doesn't not have to be systematic for it to be good. WhatamIdoing (talk) 17:46, 14 July 2008 (UTC)

Discussion on the draft itself seems to have died down, so I added it, except that I omitted the detailed example of citing a Cochrane review, which on rereading didn't seem to be worth all that space on the project page. If someone else thinks that example is worth while please feel free to add it of course. Eubulides (talk) 18:03, 14 July 2008 (UTC)