Wikipedia talk:Identifying reliable sources (medicine)

Intent?

I'm curious as to the intent of this page; do you seek to create an exhaustive list of magazines and books to use as resources? Or is this page a group of examples, in which case I wonder why it is needed in addition to the normal pages like WP:RS? (Radiant) 13:39, 14 November 2006 (UTC)

This page started as a place to put the "reliable sources" portion of the WP:MEDMOS, which I felt were not style issues and should go elsewhere. It is not intended to have exhaustive list and certainy not for all books/journals as there are simply too many that are of excellent quality. The only reason to list a handful of top journals is to highight those that are regarded as the cream.

Recent discussions on talk WP Medicine have asked:

• Are primary sources better than secondary sources? In particular, editors who cite papers covering basic science possibly being interpreted for its affects in the human population.
• Is XYZ a reliable web site that we should consider either using as a reference or providing external links?

This page could be used to capture consensus on those issues. I could also serve as a resource of sites useful to WP Medicine editors. It is currently very much a work-in-progress. I hoped that others might contribute. Perhaps the links to it are too subtle at present. Colin°^Talk 14:17, 14 November 2006 (UTC)

• I'm no expert on the matter of medicine, but if you want consensus on this page you're definitely going to need more input. While you're technically correct about source matters not belonging in the MOS, if that page is more active than this one it's not a big problem - it is common on Wikipedia for pages to have a somewhat broader scope than their title suggests. (Radiant) 15:10, 15 November 2006 (UTC)

Previous version at RS

WP:RS was substantially rewritten on 1st December 2006 (diff). The following text, which had existed for a while and which we had borrowed for these guidelines, was lost:

Physical sciences, mathematics and medicine

Cite peer-reviewed scientific publications and check community consensus

Scientific journals are the best place to find primary source articles about experiments, including medical studies. Any serious scientific journal is peer-reviewed. Many articles are excluded from peer-reviewed journals because they report what is in the opinion of the editors unimportant or questionable research. In particular be careful of material in a journal that is not peer-reviewed reporting material in a different field. (See the Marty Rimm and Sokal affairs.)

The fact that a statement is published in a refereed journal does not make it true. Even a well-designed experiment or study can produce flawed results or fall victim to deliberate fraud. (See the Retracted article on neurotoxicity of ecstasy and the Schön affair.)

Honesty and the policies of neutrality and No original research demand that we present the prevailing "scientific consensus". Polling a group of experts in the field wouldn't be practical for many editors but fortunately there is an easier way. The scientific consensus can be found in recent, authoritative review articles or textbooks and some forms of monographs.

There is sometimes no single prevailing view because the available evidence does not yet point to a single answer. Because Wikipedia not only aims to be accurate, but also useful, it tries to explain the theories and empirical justification for each school of thought, with reference to published sources. Editors must not, however, create arguments themselves in favor of, or against, any particular theory or position. See Wikipedia:No original research, which is policy. Although significant-minority views are welcome in Wikipedia, the views of tiny minorities need not be reported. (See Wikipedia:Neutral Point of View.)

Make readers aware of any uncertainty or controversy. A well-referenced article will point to specific journal articles or specific theories proposed by specific researchers.

In science, avoid citing the popular press

The popular press generally does not cover science well. Articles in newspapers and popular magazines generally lack the context to judge experimental results. They tend to overemphasize the certainty of any result, for instance presenting a new experimental medicine as the "discovery of the cure" of a disease. Also, newspapers and magazines frequently publish articles about scientific results before those results have been peer-reviewed or reproduced by other experimenters. They also tend not to report adequately on the methodology of scientific work, or the degree of experimental error. Thus, popular newspaper and magazine sources are generally not reliable sources for science and medicine articles.

What can a popular-press article on scientific research provide? Often, the most useful thing is the name of the head researcher involved in a project, and the name of his or her institution. For instance, a newspaper article quoting Joe Smith of the Woods Hole Oceanographic Institution regarding whales' response to sonar gives you a strong suggestion of where to go to find more: look up his work on the subject. Rather than citing the newspaper article, cite his published papers.

Which science journals are reputable?

One method to determine which journals are held in high esteem by scientists is to look at impact factor ratings, which track how many times a given journal is cited by articles in other publications. Be aware, however, that these impact factors are not necessarily valid for all academic fields and specialties.

In general, journals published by prominent scientific societies are of better quality than those produced by commercial publishers. The American Association for the Advancement of Science's journal Science is among the most highly regarded; the journals Nature and Cell are notable non-society publications.

Keep in mind that even a reputable journal may occasionally post a retraction of an experimental result. Articles may be selected on the grounds that they are interesting or highly promising, not merely because they seem reliable.

arXiv preprints and conference abstracts

There are a growing number of sources on the web that publish preprints of articles and conference abstracts, the most popular of these being arXiv. Such websites exercise no editorial control over papers published there. For this reason, arXiv (or similar) preprints and conference abstracts should be considered to be self-published, as they have not been published by a third-party source, and should be treated in the same way as other self-published material. See the section above on self-published sources. Most of them are also primary sources, to be treated with the caution as described in various sections of this guideline.

Researchers may publish on arXiv for different reasons: to establish priority in a competitive field, to make available newly developed methods to the scientific community while the publication is undergoing peer-review (a specially lengthy process in mathematics), and sometimes to publish a paper that has been rejected from several journals or to bypass peer-review for publications of dubious quality. Editors should be aware that preprints in such collections, like those in the arXiv collection, may or may not be accepted by the journal for which they were written — in some cases they are written solely for the arXiv and are never submitted for publication. Similarly, material presented at a conference may not merit publication in a scientific journal.

Evaluating experiments and studies

There are techniques that scientists use to prevent common errors, and to help others replicate results. Some characteristics to look for are experimental control (such as placebo controls), and double-blind methods for medical studies. Detail about the design and implementation of the experiment should be available, as well as raw data. Reliable studies don't just present conclusions.

Popular science and medicine books

Responding to "How do you know what the point is? Discuss this in Talk before deleting."

I wish to remove the line that says "Some well known and respected popular science authors include Richard Dawkins and Stephen Jay Gould." Can the author of this sentence please explain the point of including just a couple of names out of potentially hundreds of worthy authors? Colin°^Talk 14:27, 13 April 2007 (UTC)

Well I originally gave three names in this edit but Sandy removed Oliver Sacks. Colin°^Talk 14:27, 13 April 2007 (UTC)

I don't care much for Richard Dawkins myself and am now questioning the value of providing a short list. This page needs general guidelines instead Colin°^Talk 14:27, 13 April 2007 (UTC)

Yeah. I see what you mean. Just remove it. Colin°^Talk 14:27, 13 April 2007 (UTC)

Done. Colin°^Talk 14:27, 13 April 2007 (UTC)

(Apologies for the sarcasm – just a bit of fun).

Nice debate, sound conclusion :-) As an example, Sacks is infamous for his non-standard and sensationalist views on TS (I've heard other physicians opining on his writing). While we're on the topic: I don't understand either of these edits. [1] We need to get back to, Avoid citing the popular press, as they usually get it wrong. [2] The Merck Manual is an utter and total inaccurate wreck when it comes to TS; I hope it's better in other areas. SandyGeorgia (Talk) 14:33, 13 April 2007 (UTC)

The reason for including a few names is that the text gives the reader no way to distinguish between reliable books and unreliable books, or primary sources and tertiary sources. Giving a few examples gives them an idea of what the text is referring to. The reason for choosing 2 or 3 examples out of potentially hundreds of worthy authors is that you have to stop somewhere. Nbauman 15:17, 13 April 2007 (UTC)

OK, maybe we can approach that from a different angle, then. Instead of giving examples of book authors who may be reliable (and I certainly disagree on using Sacks except in very limited contexts, as I did on TS)—what if we instead give an example of a popular or vanity press book that shouldn't be used ? There are legions, for example, in the realm of ADHD. Patty Duke's book is often used to cite bipolar articles—is that reasonable? In other words, maybe we can say what we "shouldn't" use instead of what we should ??? SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)

That path may lead to conflict. Nbauman is right that it would be helpful to clarify primary from secondary and tertiary. I recently read a good book on Aspirin (ISBN 0747570833) that is thoroughly sourced. It certainly isn't a primary source but is a bit of a mix between secondary and tertiary. It is probably a reasonable source for historical material on the aspirin page but since the author isn't a scientist or doctor, would be a poor choice for the pharmacological material. Very few books are primary sources. Those that are tend to be quoted as a source of original ideas, which is why Sacks and Dawkins were on the list, though the latter hasn't much to say that would concern this wiki project. Colin°^Talk 15:38, 13 April 2007 (UTC)

Not sure how to fix this, so will offer an example (not one I'd want to highlight, though). Comings, Tourette Syndrome and Human Behavior. He owns the vanity press, Hope Press. He couldn't get his work accepted by peers, so he self-published. (For a scathing review of his methodological flaws, see this article.) He's a recognized physician in the field (more infamous than famous),so someone could argue that he's a reliable source. I suppose he would be covered under WP:RS self-published sources, but that's the kind of example I'm concerned about. In terms of wacky theories, anyone can get a book published, so maybe we can find an example that's not self-published as Comings is. And, the problem with Sacks is his focus on sensationalized aspects and his personal views. SandyGeorgia (Talk) 17:01, 13 April 2007 (UTC)

Scientists aren't always the best people for describing the scientific literature and qualities like reliability. The people with the expertise to do that are science librarians. Science librarians work with scientists, and many science librarians have had experience as bench scientists and then went on to be trained as librarians. There are many published reference books on library science that cover the subject of this entry much better than this entry does, and have solved many of the problems that these editors are struggling with. I wish some medical librarians would work on this entry. Nbauman 15:17, 13 April 2007 (UTC)

I don't think we have a medical librarian on board, but I do know someone who may be able to look at your concerns. SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)

I read the book reviews in Science, New England Journal of Medicine, and BMJ. They often review popular books and recommend them strongly. For example, the NEJM last month reviewed "Skin: A Natural History" and a book about Jack Kevorkian. BMJ has been particularly supportive of books that give the patient's perspective. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book, who are we to tell people not to read them? Nbauman

Essay or guideline?

This page is currently a bit too much like an essay than a guideline. That might be fine if we want to keep it an essay, and we may. If not, then the text needs to be condensed and keep to the point. There's a lot more that can be said on this topic and I really welcome other contributors. It might be best to let the text expand a bit before we start refining it back down. That said, if something important got deleted or watered down, then we should bring it back. Colin°^Talk 14:44, 13 April 2007 (UTC)

Newspapers

I deleted the claim that broadsheets can be reliable sources of medical information, while tabloids are not. Some tabloids are excellent sources of medical information, and conversely.

Here's a good example from today's New York Daily News, about the automobile accident of New Jersey Governor Jon Corzine, who was not wearing a seat belt.

[3] A difficult recovery that could take months, by Christina Boyle, New York Daily News, April 14th 2007.

This is what teachers call a "teachable moment," an opportunity to teach an important message because the subject has everyone's attention. This story explains exactly how someone is hurt in an auto accident if they're not wearing a seat belt, and it explains Corzine's injuries in meaningful detail. I've read hundreds of accident reports in the engineering literature, and this tabloid news story covers all the essential points. Nbauman 14:12, 14 April 2007 (UTC)

Looks reasonably well reported - but that is just the issue. All the journalist did was parrot what she was told by the doctor, while probably simplifying the language and possibly making mistakes along the way. I'm not sure how such a story could form the basis of a source on a medical article - just on a bio of the poor man himself. I appreciate that this is just an example where you considered the reporting to be of a good quality.

Perhaps US tabloids are different to UK ones. Some quality broadsheets employ journalists that write original material that may be worth citing on Wikipedia. Though, like the article above, I can't think of a circumstance where I'd do so on a medical article other than in a history/bio section. The journalist Ben Goldacre has a column Bad Science that frequently points out the shortcomings in the UK newspapers, broadsheet and tabloid.

So while I'd support your deletion of broadsheets, I wouldn't support the claim that "Some tabloids are excellent sources of medical information". Not from my experience, anyway. Colin°^Talk 15:01, 14 April 2007 (UTC)

I have to agree. Reporting on a current event is different than sourcing a medical article. A report like that might be useful somewhere in an article about seat belt safety, but shouldn't be the sole basis for medical statements. SandyGeorgia (Talk) 15:18, 14 April 2007 (UTC)

You say, "All the journalist did was parrot what she was told by the doctor, while probably simplifying the language," as if that were a trivial task.

That's not all she did. First, she had to identify one of the important scientific issues, which in this case was, "How do people get injured in automobile accidents, and how do seat belts prevent those injuries?"

Second, she had to find a doctor who was authoritative and knowledgeable, just as a journal editor would have to find an authoritative author for a review article. She didn't get any old doctor, she interviewed an academic doctor at a major critical care center.

Third, she had to get the doctor's intended meaning and quotes right. That's not easy. If she didn't do that right, the doctor would write a letter of correction. I've read a lot of the peer-reviewed auto safety engineering literature, and in my reading of this article, she seems to have gotten it right.

You say, "and possibly making mistakes along the way." You don't know that. That's speculation based on no evidence. The question at issue is whether she made mistakes. In my reading, she didn't.

The reason this news story is so important is that it performs a function of public health education. Most of the medical professional societies in the specialties that deal with automobile trauma have argued for educating the public about the dangers of automobile accidents and the protective effects of seat belts. This article is directly fulfilling that purpose. She quoted a doctor saying so directly:

Had Corzine been belted in, his injuries would have been much less severe. "A seat belt keeps your body fixed to the seat so you wouldn't move forward so much," Shou said. "A seat belt can significantly decrease the chances of these type of injuries."

This is exactly the kind of statements doctors say they want newspapers to print.

But go back to the original question, which is, are tabloid newspapers accurate and reliable? This example from the New York Daily News is clearly accurate and reliable, like all the other medical news I've read in the Daily News. According to the BMJ, and my own sampling, the U.K. broadsheets are not always that accurate. Neither is the New York Times. So any generalized conclusion about the accuracy of tabloids vs. broadside is contrary to fact. Nbauman 17:04, 14 April 2007 (UTC)

I don't think we disagree that much. We just have different experiences. It is great that your New York Daily News is so good. Our tabloids are truly awful. I agree many of the broadsheets are awful too - even the journalists that are supposed to be the house science or medicine reporter. Colin°^Talk 17:30, 14 April 2007 (UTC)

So, let's find a way to make our statements as generally correct as possible; this "tabloid" may have gotten it right, but unfortunately, most don't. There are situations in articles where quoting the popular press is fine, but to the extent possible, peer reviewed literature would be preferred. In my experience, *most* of the time, the popular press gets it wrong in subtle or blatant ways. Can we find some wording around the notion that there are acceptable uses, but peer-reviewed medical literature should be consulted ? SandyGeorgia (Talk) 18:00, 14 April 2007 (UTC)

I've removed the line:

Even peer-reviewed journals like the New England Journal of Medicine cite articles in newspapers like the New York Times and Wall Street Journal.

This may be true but I've yet to see an example. If they do, surely it is more for certain social, historical or biographical information rather than for medical facts? Some examples would help. I've also removed the line:

Newspapers should be judged on the facts, not on prejudices.

Which IMO is not written in a suitable tone.

Finally, I've added a line to clarify where I think there is consensus for using and not using newspapers. Colin°^Talk 11:37, 16 April 2007 (UTC)

Colin wrote: This may be true but I've yet to see an example.

See the example that I cited below of the NYT article on Guidant being quoted in the NEJM article on Guidant. Nbauman 13:27, 16 April 2007 (UTC)

Yes but that's current-affairs. It's a story about a certain individual and the citation backs up the story. And, as you note later, the "Perspective" section of NEJM might not be peer reviewed (and hence the term "peer reviewed journals" is a simplification). The current text, that you've restored, gives the impression that medical journals routinely cite newspapers. They may very occasionally cite them for news but not for medical facts. Please reconsider your restoration of that text. I'm not going to edit-war over this. Can anyone else help us reach consensus? Colin°^Talk 13:54, 16 April 2007 (UTC)

I can't work on it today (or maybe even tomorrow) because I'm flooded from the Nor'easter, but one way to address Nbauman's concerns may be to go back to the top of the article and make it all more about what the highest-quality and medical sources do right, and less about what the others may do wrong (taking care not to overgeneralize). Then we can revisit the newspapers and other sections in that context. If we approach it that way, we may also reduce some redundancy. For example, "The quality of newspaper coverage of medicine ranges from excellent to irresponsible, and they should be verified like any other sources" can be said of most of the other sources (not just newspapers), so we should rephrase in a way that we're not just repeating WP:RS and/or singling out newspapers, rather emphasizing what are the best sources and why. I hope this makes sense; I haven't had much sleep :-) SandyGeorgia (Talk) 15:48, 16 April 2007 (UTC)

I'm sure this guideline will be rewritten more than once before we get it right. There are so many poor sources, that I can't see how the guideline would be useful without mentioning them. After all, people are going to quote this guideline when disputing a source, not when congratulating someone for using a top quality one. One thing that might help is to clarify that this guideline is for the medical facts in medical articles. The social, biographical, current-affairs, etc information can come from sources that meet the general WP:RS. Colin°^Talk 17:11, 16 April 2007 (UTC)

Nbauman, earlier you said "who are we to tell people not to read them". I think this may be one source of our disagreement. This is not an article to advise people what to read (whether for pleasure or for research for a WP article). It is solely concerned with what we should cite. Certain books, newspapers, magazines and blogs may be a reliable source of medical information. They are not suitable for citation in an encyclopaedia (wrt to medical facts). This page is not read by Wikipedia's readers - it isn't for them. It is for editors. Are you trying to help improve the quality of our sources, or defending popular journalism? Colin°^Talk 17:11, 16 April 2007 (UTC)

Colin, I'll restate that. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book as a reliable source, I would accept that book as a reliable source for Wikipedia. Why not? Nbauman 18:38, 17 April 2007 (UTC)

They aren't "recommend[ing] a popular book as a reliable source" for medical facts in an encyclopaedia. They are saying that a book is to be recommended for reading. Such books might be suitable sources for Wikipedia (particularly those that have footnotes, citations and a scholarly comprehensive style). There are usually sources that would be a better choice. However, I return to newspapers, which is our source of dispute. Newspapers are good for news; old newspapers are good for old news. If my doctor offered advice based on what he read in the Daily Mail that day, I'd report him to the authorities. Wikipedia should be no different. Colin°^Talk 22:01, 17 April 2007 (UTC)

OK, here's a newspaper article that got the clinical facts more accurately than the peer-reviewed medical journals did. If your doctor had depended on the Wall Street Journal rather than the New England Journal of Medicine in considering whether to prescribe Ketek, he might have saved your life. Saving patients' lives is after all one of the purposes of medicine. You might rather die than have your doctor use information from newspapers rather than peer-reviewed journals, and I admire you for sticking to principles, but it seems a bit pedantic.

[4]Infected Data: Fraud, Errors Taint Key Study Of Widely Used Sanofi Drug; Despite Some Faked Results, FDA Approves Antibiotic; One Doctor's Cocaine Use; Company Defends Safety, By ANNA WILDE MATHEWS, Wall Street Journal, May 1, 2006.

... Now documents including internal Aventis emails reviewed by The Wall Street Journal are raising questions about a key clinical trial -- called study 3014 -- of more than 24,000 people that the company submitted to the FDA seeking approval for the drug.

The doctor who treated the most patients in the study, Maria "Anne" Kirkman Campbell, is in federal prison after pleading guilty to defrauding Aventis and others. An indictment says Dr. Campbell fabricated data she sent to the company. The documents show that Aventis was worried about Dr. Campbell early in study 3014 but didn't tell the FDA until the agency's own inspectors discovered the problem independently....

The full extent of the study's problems has never been made public. Its results were cited last month in an article in the New England Journal of Medicine that suggested Ketek is as safe as other antibiotics. Five of the six authors of that article disclosed that they received consulting fees from Sanofi-Aventis, and the sixth was an Aventis employee at the time of the study. Nbauman 02:15, 20 April 2007 (UTC)

Avoid citing the popular press

The section, In science, avoid citing the popular press is complete personal opinion, completely unsourced, completely overgeneralized, and completely wrong. For example:

[5]Annals of Medicine: The Bell Curve; What happens when patients find out how good their doctors really are? by Atul Gawande, The New Yorker, Dec. 6, 2004

Gawande is an MD, and in addition to the New Yorker he writes for the New England Journal of Medicine. Does the author of this section believe that Gawande does not cover science well when he writes for the New Yorker, but does cover science well when he writes for the NEJM? Similarly, Gina Kolata, a PhD, used to write for Science magazine before she moved to the New York Times. I could give many similar examples.

Many articles in the popular press have identified problems in medicine that have been ignored by the peer-reviewed literature, for example defective heart defibrillators, or financial conflicts of interest in the committees that set guidelines and recommend drugs. Peer-reviewed journals often cite newspapers in the footnotes.

The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them.

I believe that every article in the popular press should be evaluated on its own merits. You can't replace critical evaluation with a rule of thumb like 'The popular press generally does not cover science well."

This section should be completely rewritten. Nbauman 17:24, 14 April 2007 (UTC)

OK, as an example, if that author also writes for peer-reviewed journals, we should 1) be able to quote him from peer-reviewed work, and 2) be able to justify that the particular author is a reliable source, even if the popular press normally isn't. Remember, reliability of sources, like everything else on Wiki, is subject to consensus. Again, I think we can find compromise wording to the effect that there may be times when a reliable source is quoted in the popular press, but we should consult medical consensus and make sure they got it right. SandyGeorgia (Talk) 18:02, 14 April 2007 (UTC)

I think the consensus statement should start out by saying that the accuracy and reliability of the popular press varies greatly, and the accuracy and reliability of articles may vary within one publication. The popular press can be helpful in finding and organizing information, but everything in the popular press is subject to verification from the peer-reviewed literature. Still, the popular press sometimes reports important information that the peer-reviewed literature does not.

There are library science publications, such as Magazines for Libraries, that evaluate the popular press.

The popular press exposed the dangers of Vioxx before the NEJM did. The Wall Street Journal recently exposed a conflict of interest at the NEJM in which the editor who wrote an editorial supporting a policy of the National Kidney Foundation was on the board of the Boston chapter of the NKF, as I recall.

You just have to examine the facts on an case-by-case basis, and sometimes all you can do is give the opposing viewpoints. Nbauman 18:39, 14 April 2007 (UTC)

Well, you got me on the Vioxx. hits much too close to home :'-( Colin is a better writer than I am, so I'll stay out of tweaking the wording, and leave that to the two of you, but I agree with some of your points now. We just need to word it in a way that doesn't open the door too wide. SandyGeorgia (Talk) 18:56, 14 April 2007 (UTC)

I'm going to think a bit more about this one before changing any text. I'll read the Bell Curve thing later too. Hope someone else joins the debate. The current Newspaper's section reads more like a polemic than a guideline. Remember the context: medical facts. This isn't a guideline for current-affairs, history, biography, sports, etc. I'd love to see an example where citing a newspaper on a medical issue was better than citing an alternative source.

Your arguments that some articles in newspapers are OK can be extended to blogs, personal web sites, drug-company adverts, press releases, vanity press and all the other things that we guide against. The "is an MD" attribute doesn't make someone the holder of the truth. There are plenty nutty MDs out there.

You are in the privileged position of being able to judge whether an individual source (on certain topics) is good or not. Many of our editors, and almost all of our readers are not. This is why we have WP:V and WP:RS. The "truth" is secondary on WP.

WP should never use investigative reporting as a source. Newspapers get it wrong at least as often as right. They like to boast when right. Such reporting has only two editorial controls: "are our lawyers happy" and "will it sell papers". We all know of medical campaigns by newspapers that make the professionals despair. Wikipedia is in no rush. Better to report well established facts than up-to-date nonsense.

Many editors come to WP in order to evangelise the world about their treatment or their conspiracy theories as to the cause of illness. You only have to look at the more controversial sections of the autism topics on Wikipedia to find that such editors love quoting the tabloids.

You mention (in the guideline and above) that peer-reviewed journals "often" cite the popular press. Really? What percentage makes "often"? Are we talking about medical journals?

The following gets me worried:

"The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them."

This has been considered. The section on secondary sources says "Journalists writing in the popular press, and marketing departments who issue press releases tend to write poor secondary source material."

The sort of press-releases that hospitals/labs do, which are written to stimulate funding and press interest rather than advance medical science, are the last thing WP should be using as a source. And when newspapers get hold of these, they either regurgitate verbatim and uncritically, or else they mess with it. I'll try to find an example if you like. The latest cutting edge unproven research is really not what an "encyclopaedia" is about IMO.

Sorry about the rant. Your userpage says your are a professional writer. Then I'd love to know your opinion and get your support for WP:MEDMOS, which is being discussed nearby! Colin°^Talk 20:06, 14 April 2007 (UTC)

Librarians have lists of magazines that they have determined to be reliable, with the caveat that the ultimate responsibility for making a decision rests with the reader. One of the standard reference works, for example, is Magazines for Libraries. Some magazines are more reliable than others. The New Yorker is consistently reliable. So is the news section of the Wall Street Journal, though other sections, like the editorial and personal health columns, are not. There's no simple guideline that anyone could apply to determine whether a source is reliable. The only answer is that popular magazines are diverse, their quality varies, and they are subject to verification like anything else.

As for opening the door to blogs -- that's a problem, but some blogs are reliable sources. There should be a burden of proof that they're reliable, but if they meet that burden, OK. Best example I can think of is CancerGuide: Steve Dunn's Cancer Information Page (which BTW has a good discussion of how to evaluate reliable sources of information, from bench to clinical trial).

You wanted an example of a newspaper that cited the popular press. Here's one -- footnote 2. The academic publications didn't even cover this until the New York Times broke the story. The issue here was that Guidant knew their ICDs were short-circuiting, but kept selling the existing stock even though they had solved the problem with newer models. (I still don't understand why.) The Guidant entry in Wikipedia doesn't seem to have cited them. I'm excerpting these articles since you may not have subcriptions.

[6]The Controversy over Guidant's Implantable Defibrillators,

Steinbrook R,

N Engl J Med 2005; 353:221-224, Jul 21, 2005. Perspective

On October 4, 2001, Joshua Oukrop, a Minnesota teenager with hypertrophic cardiomyopathy and a high risk of sudden death from ventricular fibrillation, received an implantable cardioverter–defibrillator (ICD). The device was a Ventak Prizm 2 DR Model 1861 manufactured by Guidant (Indianapolis). After it was implanted, Oukrop's physicians at the Minneapolis Heart Institute Foundation checked it every three months (most recently on January 31, 2005) and found no problems.

On March 14, 2005, Oukrop, then a 21-year-old college student, collapsed and died in a remote area of southeastern Utah during a spring-break bicycling trip with his girlfriend.1,2 An autopsy revealed no clinically significant pathology beyond his massive left ventricular hypertrophy. His physicians were stunned by his death. ICDs have been shown to be almost invariably successful in preventing sudden death in young patients with hypertrophic cardiomyopathy, as long as they do not have end-stage heart failure — which Oukrop did not.3 When the manufacturer analyzed his ICD, it determined that the device had short-circuited internally while trying to deliver high-voltage therapy and had been permanently disabled (see diagram). Moreover, its memory had been destroyed, making the time of failure impossible to pinpoint....

2. Meier B. Maker of heart device kept flaw from doctors. New York Times. May 24,2005:A1.

[7]Maker of Heart Device Kept Flaw From Doctors

By BARRY MEIER, New York Times May 24, 2005

A medical device maker, the Guidant Corporation, did not tell doctors or patients for three years that a unit implanted in an estimated 24,000 people that is designed to shock a faltering heart contains a flaw that has caused a small number of those units to short-circuit and malfunction.

The Potential for Defibrillator Failure

The matter has come to light after the death of a 21-year-old college student from Minnesota, Joshua Oukrop, with a genetic heart disease. Guidant acknowledges that his device, known as a defibrillator, short-circuited. The young man was in Moab, Utah, on a spring break bicycling trip in March with his girlfriend when he complained of fatigue. He then fell to the ground and died of cardiac arrest.

Guidant subsequently told his doctors that it was aware of 25 other cases in which the defibrillator, a Ventak Prizm 2 Model 1861, had been affected by the same flaw. Guidant said it had changed its manufacturing processes three years ago to fix the problem. The physicians say that had they known earlier, they would have replaced the unit in their patient because he was at high risk of sudden death. His death is the only one known....

Nbauman 21:16, 14 April 2007 (UTC)

But there's a theme to your examples, Nbauman, which may help us solve this dilemma. They are tied to current events. The news media stories came about because of current events, as opposed to being reports per se about the conditions or studies about the conditions. I may not trust most newspapers to accurately tell me about the most recent studies on hypertrophic cardiomyopathy, but maybe I can trust some newspapers to report a story about a current event involving that condition correctly. SandyGeorgia (Talk) 21:30, 14 April 2007 (UTC)

The Bell Curve

I've not finished reading this but already spotted something that highlights why newspapers make poor secondary sources. Page two discusses various operations and the variable success rate amongst surgeons. One example given:

A Scottish study of patients with treatable colon cancer found that the ten-year survival rate ranged from a high of sixty-three per cent to a low of twenty per cent, depending on the surgeon.

The article doesn't fully cite the study, which is typical and understandable (though perhaps not for the online edition, which has no space concerns). So we don't know:

1. Why, on an article in an American magazine, with mostly American examples and statistics, does the author pick a Scottish study? Are they desperate to find an example to back their case? Or perhaps this study is internationally famous?
2. How big was the study? How many patients? How many hospitals? Did it just cover NHS hospitals or private ones too?
3. How was the study conducted?
4. What threshold did they use for "treatable"? Did all the hospitals use the same threshold when choosing patients?
5. What other factors influenced patient outcome? Perhaps patients living in poor areas did worse than richer ones and this affected the local hospitals they went to?
6. Perhaps there was a hospital variation, not just the surgeon. A patient's recovery depends on many professionals' input, not just the surgeon.
7. Did they have other non-surgical therapies too?
8. The article makes a case that the range of success follows a bell curve. But this example only quotes two extremes. We have no idea what shape graph this study produced.

And so on. Most "for-professionals" articles would give more information than this, plus a citation. Without this traceability from Wikipedia to secondary source to primary source, our readers are limited in how much they can learn should they ask questions concerning the reliability of the data and how the author chooses to use it to make their case. The best popular science/medical books provide citations. So should online newspapers. Colin°^Talk 23:09, 14 April 2007 (UTC)

Gawande was citing a classic, frequently-cited study that every oncologist would know. [8]McArdle et al, BMJ 1991;302:1501-5, Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival. University Department of Surgery, Royal Infirmary, Glasgow.Nbauman 06:03, 15 April 2007 (UTC)

Every oncologist might think "Ah, she's citing McArdle and Hole 1991" but almost none of our readers would. I was going to try to look up the paper myself but you've saved me the trouble. Unfortunately, it only goes to strengthen the argument that the primary source should be cited by WP rather than the newspaper (which may be less than helpful in finding it). For example, I can now cite:

• McArdle C, Hole D (1991). "Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival". BMJ. 302 (6791): 1501–5. PMID 1713087.

which fortunately has the full text available free online. Even from the abstract, our reader can tell more than the newspaper provided. We can also see there were two follow-up letters, one critical and one supportive. PubMed also tells me that McArdle and Hole continued their research. Which is just as well: these were "patients with colorectal cancer presenting over the six years from 1974 to 1979". The quality of an operation performed 30 years ago is of diminishing interest to those going under (or holding) the knife today. They published a follow-up paper in 2002 (with a much larger number of patients, operated over the years 1991 and 1994) that confirmed that variability amongst surgeons was still a problem:

• McArdle C, Hole D (2002). "Outcome following surgery for colorectal cancer: analysis by hospital after adjustment for case-mix and deprivation". Br J Cancer. 86 (3): 331–5. PMID 11875693.

So it could be argued that the first might be a historical classic, but the second is of more relevance today. They have also answered some of my questions about poor patients doing less well:

• Hole D, McArdle C (2002). "Impact of socioeconomic deprivation on outcome after surgery for colorectal cancer". Br J Surg. 89 (5): 586–90. PMID 11972548.

I see they have refined their conclusions about surgeon variability, by showing that surgeon speciality is a better guide than just volume of work:

• McArdle C, Hole D (2004). "Influence of volume and specialization on survival following surgery for colorectal cancer". Br J Surg. 91 (5): 610–7. PMID 15122614.

I could go on (for example, to check that those who cite this paper do so favourably) but in just a couple of minutes, I've found so much more high quality source material to help improve a WP article on either colorectal cancer or issues of surgeon competence. Colin°^Talk 07:00, 15 April 2007 (UTC)

---

Let's return to the original question. Is the New Yorker a reliable source of medical information or not? Nbauman 13:47, 15 April 2007 (UTC)

Back to what I said above; there are some things that can possibly be cited to some of the news media (I've done it in Sociological and cultural aspects of Tourette syndrome), but a medical, peer-reviewed source is always preferred, as demonstrated in Colin's example. The news media rarely gives us the full story. Let's make sure our wording gives a strong preference to peer-reviewed sources. I believe (haven't checked) we don't say never cite the popular press; we urge caution. Perhaps we can expand/contract that to say they shouldn't be preferred over the highest-quality, peer-reviewed medical sources, but do have some usefulness. I think (?) your concern is that we may be "bashing" the popular press; perhaps we can better flesh that out. SandyGeorgia (Talk)

Right, I don't want to bash the popular press. Most users of Wikipedia use the popular press (many academic doctors read the popular press, as demonstrated from their footnotes), so we should help them place the popular press in its proper context. I think the context is that their accuracy and reiability varies greatly. Some magazines are accurate enough that we can presume them to be accurate, but they're always open to challenge, and peer reviewed literature usually (but not always) trumps the popular press. There are evaluations of the popular press, in the links I added under Newspapers. Nbauman 18:21, 15 April 2007 (UTC)

Gawande's work for NEJM is peer-reviewed. His journalism is not. In general, peer-reviewed science is the preferred source for any medical content, with all other sources inferior to it. Within peer-reviewed science, I think we should adopt EBM grading. A meta-analysis or systematic review is much more powerful as a source than individual trials, case-control studies, case series or case reports. JFW | T@lk 20:14, 15 April 2007 (UTC)

JFW, can you help work on the wording with Colin here? SandyGeorgia (Talk) 20:24, 15 April 2007 (UTC)

Some of those ideas are already very briefly mentioned but not in such an explicit order. JFW is much, much more knowledgeable on this front than me so I'd rather defer to him. I can tidy the English if required :-). I'd really like JFW to contribute to this, since he's often the one defending these things on the project or article talk pages. Colin°^Talk 20:47, 15 Apri

I don't know. Is the Perspectives section of the NEJM peer-reviewed? Nbauman 00:05, 16 April 2007 (UTC)

That's a good point. Not all forms of writing in these journals get the same treatment. For example, an obituary may be no better checked than one in a newspaper. There are several aspects to judging the quality of a source, not just the journal title or type. Colin°^Talk 07:38, 16 April 2007 (UTC)

Proposed Guideline?

This page was set up as a Proposed Guideline. It would appear that WP only accepts such a proposal for a finite time before marking it Historical or Rejected. Can we decide what we are going to do with it? My gut feeling is that there currently isn't enough traffic or discussion on this for it to move forward quickly enough to become a formal guideline before someone retires it again. If someone else wants to beat a drum to round up some contributors, then great. I was hoping that we might get contributions from editors with experience writing or training reading medical articles. I do believe the project needs these guidelines

1. To have somewhere to point at when the issue arises.
2. To have a central place to discuss what kind of sources are good for medical info and how they should properly be used.

But it may be that those needs can be met by taking the banner off and leaving it as an informal guideline in Project space. Thoughts? Colin°^Talk 11:35, 15 May 2007 (UTC)

The editor who tagged it rejected tagged dozens at the same time; I doubt he read the ongoing discussion at WP:MEDMOS. I think we can push for guideline status, with just a bit more review. Since we were busy on MEDMOS, we haven't really pushed. SandyGeorgia (Talk) 11:45, 15 May 2007 (UTC)

If I may chime in... proposals tend to be marked "historical" if there's no active debate going on. This is a matter of convenience, not a fixed difference. In particular, if active debate starts once more (e.g. through advertising the page) then it's active and no longer historical. I see no substantial dissent on this talk page, although I lack the expertise in medicine to vouch for the quality of the page. If a few medically-inclined people can confirm that it's (1) accurate and (2) useful, then it would make a worthwhile guideline. Guidelines are not a big deal and are certainly allowed to have exceptions. >Radiant< 11:55, 15 May 2007 (UTC)

(A) If there is discussion regarding development of this guideline it should be reflected here

(B) There is insufficient breadth of contribution here to demonstrate a consensus for acceptance

(C) The onus to demonstrate consensus is on the proponents and in default a proposal is rejected without demonstrable consensus

(D) Yes, this was included in a cleanout of dead proposals, and for now I remove my advocacy of rejection. --Kevin Murray 12:07, 15 May 2007 (UTC)

Our efforts (and discussion) had been focused first at WP:MEDMOS; we'll get this moving again. SandyGeorgia (Talk) 12:10, 15 May 2007 (UTC)

Not ready for guidelines

I don't think this guideline is ready to be adopted. Much of it is unsourced, unsupported personal opinion. For example, "The popular press generally does not cover science well." Who says so? What is their evidence? Isn't this an overgeneralization? In some cases, the Wall Street Journal has turned out to be more reliable than the New England Journal of Medicine.

Why doesn't the entry cite the extensive literature on how the popular press covers science (which finds that the quality of coverage varies from very good to very bad)? What about the library literature, such as Magazines for Libraries?

A fundamental problem is that this is just a list of sources and judgments about them. It wouldn't be much help in resolving real disputes that go on in real articles. Define the problem. I would suggest that you look at some actual controversial medical entries, and examine the disputes that come up over reliable sources, etc. Look at Dichloroacetic acid, or Diabetes.

People do cite the popular press all the time. What should we do about it? Most of the peer-reviewed literature isn't available free on the Internet, so it isn't verifiable to someone who doesn't have a subscription. What do we do about that?

What are the disputes and how should they be resolved, in terms of reliable sources? Nbauman 14:26, 15 May 2007 (UTC)

Maybe the proponents could provide some examples of articles which would be removed if this guideline was adopted and why they are inappropriate for inclusion at WP. --Kevin Murray 14:32, 15 May 2007 (UTC)