Wikipedia talk:Identifying reliable sources (medicine)

WP:DUE

Should we add some text about WP:DUE when we address primary sources? Currently we have issues like this [1]

Reliable primary sources may occasionally be used with care as an adjunct to the secondary literature, but there remains potential for misuse. For that reason, edits that rely on primary sources should only describe the conclusions of the source, and should describe these findings clearly so the edit can be checked by editors with no specialist knowledge. In particular, this description should follow closely to the interpretation of the data given by the authors or by other reliable secondary sources. Primary sources should not be cited in support of a conclusion that is not clearly made by the authors or by reliable secondary sources, as defined above (see: Wikipedia:No original research). WP:DUE must also be keep in mind as many comments made may be of undue weight when added to a general medical article.

Comments?Doc James (talk · contribs · email) 07:09, 20 January 2012 (UTC)

Comment. Whatever else, I'd say that many comments made may be of undue weight is not clear enough to be really helpful here. - Smerdis of Tlön - killing the human spirit since 2003! 05:22, 22 January 2012 (UTC)

How about "nearly all comments made" --Doc James (talk · contribs · email) 18:26, 28 January 2012 (UTC)

We do need some sort of clarification of the UNDUE issue with respect to primary sources-- you can find a study to say just about anything, and adding such to articles gives undue weight to obscure topics. On matters that are well covered in secondary reviews, if something is left out, there's usually a reason. I hope we can find some wording to address this problem. SandyGeorgia (Talk) 19:05, 28 January 2012 (UTC)

If I might make a suggestion, how about something like: "When citing primary sources, particular care must be taken to adhere to Wikipedia's undue weight policy. Secondary sources should be used to determine due weight." Jakew (talk) 19:30, 28 January 2012 (UTC)

Sure that sounds good. Doc James (talk · contribs · email) 19:57, 28 January 2012 (UTC)

That sounds good to me, so I have added it. WhatamIdoing (talk) 20:09, 28 January 2012 (UTC)

I had a query about this particular change: what about tertiary sources? These often make a direct effort to provide a fair and balanced approach to a discipline whereas secondary sources make no effort to provide fair weighting when considered amonst others!

The last comment here [2] seems wise. Mindjuicer (talk) 00:08, 18 February 2012 (UTC)

The "last comment" says "See what the reference texts and textbooks say about the two views. Balance them accordingly". For most of MEDRS's purposes, reference texts and textbooks are secondary sources. WhatamIdoing (talk) 19:24, 20 February 2012 (UTC)

Sorry, missed your reply until now.

What about other tertiary sources, such as guidelines from national health boards and the WHO?

My concern is that WP guidelines such as this are used as weapons on many pages. Prominence of secondary sources is easily contested. What is the reason for excluding tertiary sources which, by definition, should provide a balanced overview? --Mindjuicer (talk) 18:42, 23 February 2012 (UTC)

should articles about medical conditions only deal with medical information? what about social practices?

Should articles about medical conditions only deal with medical information? Or can they also report about what people actually do with their disease, (as far as this practice meets WP:notability), even if the medical community has not written anything about it? The current guidelines WP:MEDRS do not deal with this topic: they deal with non-academic sources http://en.wikipedia.org/wiki/Wikipedia:Reliable_sources_%28medicine-related_articles%29#Other_sources as sources of medical information, but they do not deal with them as sources of information about social practices related with a medical condition. This is different: I think that medical information should clearly be separated from social practices, but I also think that both can be part of the same article. For example, evolution article has a section "Social and cultural responses".Mokotillon (talk) 14:47, 24 January 2012 (UTC)

Mokotillon, to avoid fragmenting the discussion, can we post a request once and merely link to the discussion in other places if required. Let's discuss this at WT:MED where there are likely to be more watchers. Colin°^Talk 16:26, 24 January 2012 (UTC)

Open Access Journals: Should I be asking this here instead?

I posted this inquiry about open access pay-to-play journals at the Project Medicine talkpage Wikipedia_talk:WikiProject_Medicine#Open_Access_Journals_and_related_issues, but perhaps it belongs here instead. If so, let me know and I'll move it here instead. Thanks Fladrif (talk) 16:28, 24 January 2012 (UTC)

Listing and reliability

Following a recent discussion at RSN[3], there now seems to be an assumption that a research review appearing in a new or open access journal must be listed in specific indexes (MEDLINE, ICMJE, CSE) in order to qualify as a reliable source. Is this a course of action that should be incorporated in policy? Can we determine that a research review is not high quality if not on these lists? Spicemix (talk) 20:55, 29 January 2012 (UTC)

I wouldn't call this an assumption; it looks more like a consensus. And, it would probably be a good idea to formally incorporate it into policy. Commercial publishers are churning out new medical journals literally by the dozen every single day, most on the author-pays open access model. Most will probably fail either for lack of interest or lack of scholarship, some will continue to be published but remain obscure, and a few will eventually come to be regarded as reputable and reliable. The requirement of MEDRS is that a journal must be "reputable". By definition, that means that it has gained acceptance in the medical community as reliable. If it is not indexed in the core indexes for medical publications like MEDLINE, PUBMED, ICMJE and CSE it can hardly be considered to have that acceptance - inclusion in those indexes is a very low bar indeed. Being listed is not an automatic qualification; not being listed should, however, be an automatic disqualification. If it is not widely cited by other researchers, it cannot be considered to be "reputable". If a journal is brand new and so obscure that it can't even get in those indexes, it definitely should not be used as a source on medical articles. After it is indexed, and after it is widely cited, it will then, and only then, meet the threshold of consideration as being "reputable".Fladrif (talk) 18:55, 31 January 2012 (UTC)

It could be added to the guidelines, but reading the summary by the editor whose closed the discussion, it seems that the reason the journal is not listed in the indices is that it is not a reliable source. TFD (talk) 23:58, 31 January 2012 (UTC)

A couple of thoughts:

• It's not just open access journals or disreputable groups that charge for publishing. Some highly reputable journals have page fees. PNAS charges all authors $70 a page, plus other assorted fees. Circulation has a similar list of charges. Stroke charges for all publications, including letters to the editor. Blood charges $50 to even submit a paper for review. Charging for publication is not an indication of poor quality.
• In terms of the "vanity press" argument, I don't see any particular difference between paying the publisher hundreds (or thousands) of dollars to publish your paper and letting them charge readers $30 a copy to read it vs paying an open access journal the same amount of money to publish your paper, in return for a promise that they won't charge the readers.
• I understand that index listing takes a little while (as does acquiring a reputation), so new journals should probably not be evaluated strictly on that basis.
• Some really lousy journals are listed on those indices. See, e.g., Medical Hypotheses#Abstracting_and_indexing: they don't even pretend to do peer-review.

So I don't think that we want to treat all "paid" journals the same, and I don't think that "open access" or "being indexed" is a good marker of quality.

Finally, it's worth remembering what WP:RS's FAQ page says:

"Are there sources that are "always reliable" or sources that are "always unreliable"?

No. The reliability of a source is entirely dependent on the context of the situation, and the statement it is being used to support. Some sources are generally better than others, but reliability is always contextual."

Even the journals that we say are generally "unreliable" are reliable for certain (limited) purposes, and those that we say are generally "reliable" are completely unreliable for other purposes. The issue is complicated, and simplistic rules aren't going to be helpful. WhatamIdoing (talk) 01:05, 1 February 2012 (UTC)

I agree wholeheartedly that being listed on these indices is no guarantee that a journal is a reliable source or accepted; there are indeed many journals with no peer review on these lists. But getting listed is such a low bar that not being listed on any of them has got to be a red flag that it is not yet accepted at a level that we would regard as reliable. Can anyone identify clearly reliable peer-reviewed medical journals that aren't listed on these indices? Just like being peer reviewed or not is a bright-line distinction, not being indexed on any of the leading indexes should be a bright line disqualification of a source.

Also, it is clear that the consensus at RSN and contentious complaints from advocates on the RSN talkpage and even at ANI was that Nutrition and Metabolism was not a reliable source for medical claims being made by proponents in the Transcendental Meditation research article, and not being listed on these indices was only one of many factors. Nor was being an open access publication the only or deciding factor. What User:Spicemix is not disclosing is that the same factor of appearing on none of these leading indices is one of several factors under discussion at another RSN thread involving Health Science Journal [4] One uninvolved editor has opined that it is not a reliable source for similar claims at the same article; no-one has attempted to defend the source as reliable nor to defend the use to which it is being put in the article. It seems that there is a pattern of TM research being reported in obscure and recent journals, and this question seems directed to the possibility that unindexed obscure journals will fail as sources for those seeking to push that research on Wikipedia. Fladrif (talk) 01:53, 1 February 2012 (UTC)01:48, 1 February 2012 (UTC)

You're missing the point of the FAQ: There are no bright lines. What makes a source be reliable or not depends on how you use it, not just on whether it's peer-reviewed. A non-peer-reviewed source can be perfectly reliable. This guideline recommends multiple types of non-peer-reviewed sources: practice guidelines, position statements, medical textbooks, and more. None of those typically undergo peer review.

Additionally, it's not enough to say "This is a poor publication, so everything in it is unquestionably rubbish." We sometimes accept self-published personal blogs as reliable sources. We accept business' websites as reliable sources. Surely even the worst-run academic journal isn't as completely lacking in editorial oversight as someone's personal blog. You've got to consider all the facts and circumstances. If Einstein published an article in Rubbish Journal, we'd still accept the article, because the creator of the work is sufficiently strong to outweigh our concerns about the publisher. (We'd probably wonder why he was publishing in that journal, but we'd accept it.) "Source" has three meanings on Wikipedia. "Publisher" is only one of the three. You only need one of them to be good enough.

I'm not defending the particular instances at hand; they look pretty dubious to me. But it's a bit more complicated than that. We can't just say "journals must be indexed" and expect that to weed out garbage. We can, however, reasonably predict that any statement like that will result in people saying "See, it's indexed, and therefore obviously reliable". WhatamIdoing (talk) 04:23, 1 February 2012 (UTC)

WaId makes some good points here, and that last is very persuasive. Still, it seems that absent indexing, absent a record of citations, and absent the publishing history that might eventually establish a new journal's impact, we don't have much grounds for saying that journal has a track record of responsible publication practices. A presumption against such journals should simply say the journal doesn't add credence to the authors. Effectively we would treat its contents as self-published until such time as it should establish a good reputation. LeadSongDog come howl! 06:27, 1 February 2012 (UTC)

I think that applies to any new publication, whether it claims to be an academic journal or a local newspaper. You can't have "a reputation for fact-checking and accuracy", to use the wording from WP:RS, until you have a reputation in the first place. WhatamIdoing (talk) 00:32, 2 February 2012 (UTC)

I'd like to mention that there are at least a few truly excellent journals that have embraced the open-access model. The flagship example would have to be the Public Library of Science (PLoS) family of journals. Even the lowest-impact member of this open-access, Creative-Commons-licensed seven-journal family, PLoS ONE, still pulls a respectable impact factor of 4.4. The oldest journals in the group (PLoS Biology and PLoS Medicine, started in 2003 and 2004, respectively) have impact factors of around 13. Interestingly, a few high-impact 'traditional' journals have moved to open-access; BMJ, for instance, has made all of its research articles open-access since 2008.

All of these journals are, of course, indexed in MEDLINE, and I doubt that anyone would have a problem considering them to be reliable sources by any reasonable interpretation. Nevertheless, I feel it's worthwhile to bear in mind that "open-access" doesn't have to be a dirty word, and that there are some remarkably good scientific periodicals that are published that way. TenOfAllTrades(talk) 14:53, 1 February 2012 (UTC)

Others release all articles after a set time period, making them sort of open access and sort of not. I believe that Blood makes all articles free after a year, and I've seen others that do this after five or ten years. (Actually, I don't understand why they don't all make articles free after five or ten years.) WhatamIdoing (talk) 00:32, 2 February 2012 (UTC)

Many thanks to all for these comments. They make clear just how many factors need to be considered when evaluating reliability.

RSN has indicated that Journal of Nutrition and Metabolism can't yet be considered a reliable source, so we'll have to wait and see there, but there are some interesting issues that arise from that decision.

The primary source, Paul-Labrador, evaluated in the review looks very strong. It was conducted at the Cedars-Sinai Medical Center, one of the top ten medical research centers in the US, and is published in a journal of the AMA. It's included in many research reviews.

Also the review itself, despite being published in an unestablished journal, has some strong points. It is systematic, and was funded by NIH. It's peer reviewed, and the authors, Anderson and Taylor, are well credentialed. It comes out of the University of Virginia School of Medicine.

Is there any guidance on what weight these factors should have? Spicemix (talk) 13:43, 3 February 2012 (UTC)

There is strong consensus, developed over multiple threads on talk pages and noticeboards, including RSN, MEDRS, Project Medicine, and Fringe Theories, over many years, and involving many involved and uninvolved editors, that primary sources should not be used at all for medical claims in the TM articles. Fladrif (talk) 19:18, 3 February 2012 (UTC)

Psychology

Based on comments at Talk:Dissociative identity disorder today, it appears that we may need to specifically mention "psychology" as a topic area that is covered by this guideline. It's being asserted that since talk therapy involves no medications, then MEDRS doesn't apply.

I don't want to deal with this now, but perhaps in a few weeks someone will remind me about this. I think we'll want to solicit opinions from the folks at WP:PSY when deciding whether and how to address this perceived loophole. WhatamIdoing (talk) 04:23, 2 February 2012 (UTC)

Meta-analysis

The claim that meta-analyses of independent RCTs provide the best evidence needs a citation (to a high-quality most reliable source).

Thanks!  Kiefer.Wolfowitz 12:36, 5 February 2012 (UTC)

[5], considered by some to be the "bible" of EBM, states meta-analyses of RCTs to be Level 1a evidence, or highest level. Yobol (talk) 03:00, 7 February 2012 (UTC)

Additionally, per Wikipedia:Policies and guidelines#Not_part_of_the_encyclopedia, no such citation is needed. There are no reliable sources on the subject of what the Wikipedia community believes is the best evidence for supporting such claims in a Wikipedia article. WhatamIdoing (talk) 04:57, 7 February 2012 (UTC)

@Yobol,

Thanks for the clarification and citation.

FYI, "meta-analysis" graces the (long) list of (non–Harry-Potter) philosopher's stones in David A. Freedman's Statistical models. I've not read Ingram Olkin making such claims..., so the citation is very helpful.

@WhatamIdoing?

Please read "helpful" or "desirable" for my flawed "necessary".

The guidelines already cite 10 or so references, which provide further reading or substantiate a point that might be questioned by a sophisticated editor. Yobol's reference is similarly useful to the guidelines.

Thanks to you both! Cheers,  Kiefer.Wolfowitz 06:23, 7 February 2012 (UTC)

I updated the article using this source. Page 99 states that systematic reviews of non-randomized studies are worse than meta-analyses of RCTs.  Kiefer.Wolfowitz 06:39, 7 February 2012 (UTC)

Research reviews

MEDRS strongly recommends the use of research reviews, but gives little guidance as to how to determine whether a particular research review is reliable. Currently the only criteria are peer-reviewed secondary sources and publications on Brandon/Hill. Would it be useful to identify other considerations? For example impact factor seems already to be widely referred to. Listing on major indexes has recently been suggested at RSN[6], as have citation of the review and citation of other work by the authors. Other possibilities, raised here[7], are the standing of the authors of the review (well credentialed, widely published?) and the status of their research center; also a consideration of the quality of the research included in the review (publication of the primary sources in top medical journals? the primary research conducted at top medical centers?). Support from the NIH or other respected organization is another suggestion.

A further consideration could be whether the journal carrying the review is put out by a prominent association, organization, or university.

Would the inclusion of factors such as these be helpful? Spicemix (talk) 13:20, 7 February 2012 (UTC)

I'm not sure that the benefits would outweigh the costs. We have a fairly serious problem with POV pushers who want to discredit apparently good sources, and this would pretty much turn into a checklist of reasons to reject any publication that I personally disagree with. That's why the last paragraph of WP:MEDASSESS tells editors not to personally evaluate studies.

Impact factor and citations, in particular, are poor markers for quality. They tell you something about the publication's popularity or prominence. A review that gets cited ten times as an example of flawed thinking has the same citation count as a review that gets cited ten times for its brilliance. WhatamIdoing (talk) 19:30, 20 February 2012 (UTC)