Wikipedia talk:Identifying reliable sources (medicine)

Clarification

"All Wikipedia articles should be based on reliable, published secondary sources," seems to contradict "Therefore, it is vital that the biomedical information in all types of articles be based on reliable, third-party, published sources and accurately reflect current medical knowledge." so it should be changed to "All Wikipedia article topics should be based on reliable, published secondary sources." Since this is a general guideline and not a medrs guideline. The general guideline is to use secondary sources to establish notability for the article topic.

"In particular, this description should follow closely the interpretation of the data given by the authors or by other reliable secondary sources." The word "other" only muddies the meaning. if you read it slowly, and if you read it fast, the word only changes the meaning. It should be removed. The word other implies a meaning that is not there if you take the time to read it slowly.

I moved this down to the advice section from the definition section, because these aren't definitions, they are guidelines.

I added a comma and that also got reverted. Its supposed to have a comma to be grammatically correct, and it tells the reader to slow down. The meaning of it seems to change if you read it fast, as opposed to reading it slowly with or without the comma.

My edits shouldn't have been reverted, it didn't change anything, only clarified this. Leaving these edits in is promoting the mistake of misreading it. - - Sidelight12 ^Talk 03:48, 17 June 2013 (UTC) / 06:45, 17 June 2013 (UTC)

I do not understand how the first two statements are supposedly conflicting. "All Wikipedia articles should be based on reliable, published secondary sources" refers to the actual content. The use of secondary sources is not merely limited to deciding whether to have an article about it ("notability"). Most of that content should come from secondary sources, such as review articles, university-level textbooks, and high-quality reference works. WhatamIdoing (talk) 06:17, 17 June 2013 (UTC)

I mixed up third-party with tertiary sources, so I crossed that part out. - Sidelight12 ^Talk 06:41, 17 June 2013 (UTC)

I disagree that your edits should not have been reverted - it is generally unwise to change a guideline or policy without discussing it first, as they are closely watched. That aside, I don't understand what you are getting at by adding the word "topics" - can you clarify? Do you maybe mean "content" (i.e. specific passages)? Moving the passage to Basic Advice as you propose makes sense to me. Jytdog (talk) 13:09, 17 June 2013 (UTC)

It says "All Wikipedia articles," which makes it a wikipedia wide policy and not a specific medrs policy. The Wikipedia wide policy is that secondary sources be used to establish the notability of an article topic for whether the article should exist, and not the content within the article. By article topic I mean the article namespace, which is already established, not the content within the article. I don't think there is a guideline anywhere that says, all content within all wikipedia articles should be solely based on reliable, published secondary sources. Rather, all content within all Wikipedia articles should be based on reliable published sources. - Sidelight12 ^Talk 04:27, 18 June 2013 (UTC)

You are completely wrong: While notability is indeed established by secondary sorces that is not in contradiction with the fact that wikipedia articles should be based in secondary sources. MEDRS is only the application to medicine of WP:NOR, which in a section (See: WP:PSTS) says

Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources. Secondary or tertiary sources are needed to establish the topic's notability and to avoid novel interpretations of primary sources, though primary sources are permitted if used carefully. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than to an original analysis of the primary-source material by Wikipedia editors

and

Policy: Wikipedia articles usually rely on material from reliable secondary sources. Articles may make an analytic or evaluative claim only if that has been published by a reliable secondary source..

Similarly when choosing sources (See WP:RS) says

Articles should rely on secondary sources whenever possible. For example, a review article, monograph, or textbook is better than a primary research paper. When relying on primary sources, extreme caution is advised: Wikipedians should never interpret the content of primary sources for themselves.

and

Isolated studies are usually considered tentative and may change in the light of further academic research. The reliability of a single study depends on the field. Studies relating to complex and abstruse fields, such as medicine, are less definitive. Avoid undue weight when using single studies in such fields. Meta-analyses, textbooks, and scholarly review articles are preferred when available, so as to provide proper context

and

Primary sources are often difficult to use appropriately. While they can be both reliable and useful in certain situations, they must be used with caution in order to avoid original research. While specific facts may be taken from primary sources, secondary sources that present the same material are preferred. Large blocks of material based purely on primary sources should be avoided. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than original analysis of the primary-source material by Wikipedia editors

and

Ideal sources for biomedical assertions include general or systematic reviews in reliable, third-party, published sources, such as reputable medical journals, widely recognised standard textbooks written by experts in a field, or medical guidelines and position statements from nationally or internationally reputable expert bodies..

In summary: Content in Wikipedia articles should be based in secondary sources to avoid original research. In the future please discuss changes to policies and guidelines first. --Garrondo (talk) 07:12, 18 June 2013 (UTC)

Vote for this change only, removal of the word "other" from the following sentence to:

"In particular, this description should follow closely the interpretation of the data given by the authors or by other reliable secondary sources."

*Support - nominator of proposal - Sidelight12 ^Talk 11:07, 19 June 2013 (UTC)

Proposed edit to guideline

Sorry gang I am back again with the toxicity issue, this time with a specific proposal.

Briefly, IMO there is a widespread problem on Wikipedia with regard to articles that discuss toxicology -- namely, one finds content on toxicity sourced from primary studies reporting results of in vitro and animal studies of a product or active ingredient, conducted by basic researchers with no training in toxicology. Too often articles are filled with laundry lists of such content. These sections are often under section headers like "Health Effects" and in my mind, their appearance in Wikipedia violates MEDRS and PSTS and -- and they serve just to scare people. Examples are Bisphenol A, Glyphosate, Endosulfan in subsections under "Health Effects" on endocrine disruption and developmental effects, The DDT article generally handles this very well but this problem has crept into discussion of "links" to developmental problems and cancer.

Toxicology is a discipline in which scientists are trained to consider typical routes and dose of exposure to a substance, design in vitro and animal experiments to obtain useful data, and then to extrapolate those results to humans. It is special training that a typical doctor or basic biomedical researcher does not have; regulators rely on toxicologists to critically review marketing applications (and to review subsequent studies to decide whether a product should be taken off the market) and companies rely on them to decide whether to invest money in further development. Wikipedia should also rely on them for sources about toxicity, in order to ensure that we provide reliable, consensus content.

To help manage this better, I'm proposing that the following be added to MEDRS, perhaps in the "Assess evidence quality" section, as a new last paragraph. WIth respect to toxicity, that section is especially problematic because it was not written with toxicity in mind (e.g. the paragraph that starts with "The best evidence comes primarily from meta-analyses of randomized controlled trials (RCTs)..." doesn't provide guidance for dealing with toxicity, as it directed to interventions like drugs or medical devices)

I am very open to suggested changes - here it is:

"Content discussing the toxicity of substances falls within this guideline. The toxicity of drugs is studied in cells, animal models, and in RCTs during drug development, and may also emerge in post-marketing surveillance. Substances that are not intended to benefit health ("non-drug substances") will never be the subject of an RCT, since such trials would be unethical. Observational studies of the effects of non-drug substances on humans, or the presence of non-drug substances in humans, may be conducted and published after a product comes to market; these are primary studies. Toxicity studies are generally carried out in animal models or cells, and the results are extrapolated to humans; these too are primary studies. Content that discusses the toxicity of non-drug substances should be sourced from secondary or tertiary sources written by toxicologists, in which primary sources are critically reviewed for their relevance to humans under typical exposures and doses. Primary studies should not be used to support content that discusses toxicity." (note, went through and deleted all old versions Jytdog (talk) 14:20, 22 June 2013 (UTC))

Current proposal as of June 22: Include: "Content discussing the toxicity of substances falls within this guideline. Primary studies should not be used to support content that discusses toxicity." and I will generate a separate essay to Tox for everybody to consider.

Note - original post of this current proposal is at the bottom of this thread; repeated here so it easy to find. Jytdog (talk) 14:20, 22 June 2013 (UTC)

Thanks for considering this.Jytdog (talk) 12:50, 17 June 2013 (UTC)

• Support - It's not really changing the guideline at all, just specifically mentioning that there is a need for high quality secondary sources in toxicology and primary sources should be avoided. I think this is a beneficial clarification to the page per the reasons you describe above. Lesion (talk) 13:03, 17 June 2013 (UTC)

Consider "non-drug substances" instead of "not-drug substances"Lesion (talk) 13:03, 17 June 2013 (UTC)

argh, typo. I fixed it. Thanks! Jytdog (talk) 13:11, 17 June 2013 (UTC)

• Oppose per WP:CREEP. It's not clear how this would work with respect to a substance like asbestos, say. There's lots of reputable, secondary material out there and I'm not sure how you'd determine whether it passes this toxicology test or not. Different countries may have different professional standards and bodies covering such matters. Warden (talk) 14:31, 17 June 2013 (UTC)

Colonel Warden, I would love to hear your thoughts on how to manage toxicity content without more explicit language in MEDRS dealing with it.. happy to hear from you on my Talk page or yours, if you like. Or maybe you don't agree that there is a problem (if you have not considered this before, please see the Bisphenol A page and its Talk page to see what I mean) 14:58, 17 June 2013 (UTC)

• Oppose This is a major change. Jytdog proposes that he be able to reject systematic reviews that are directly about toxicity unless he agrees that the person performing the study should be described as "a toxicologist". The meat of the proposal is "should be sourced from secondary or tertiary sources written by toxicologists". This invites us to produce articles with undue attention to the views of one specialty, to ignore toxicity entirely if no True Toxicologist™ happens to have written a review about it, and to have disputes over whether each author really "counts" as a toxicologist. WhatamIdoing (talk) 14:36, 17 June 2013 (UTC)

Good point. Consider rm "written by toxicologists". Lesion (talk) 14:42, 17 June 2013 (UTC)

Done. thank you both! I am willing to strike "written by toxicologists" as long as the "critical review" language remains. See for example this source that was recently presented (right hand column, about halfway down, paragraph starting "The herbicide glyphosate..." - just cites some primary studies and doesn't comment on their design or relevance - this is the disastrous kind of secondary source that should be excluded) WAID, I am laughing about the TM, but this is not intended just for me (although I want to be able to reach for it) - it is intended for everybody. Hopefully you can accept this with the "written by toxicologists" struck out? Jytdog (talk) 14:58, 17 June 2013 (UTC)

What counts as "critically reviewed"? For example, do all systematic reviews count, or only those systematic reviews that choose primary sources according to some factor like the exposure level? WhatamIdoing (talk) 22:19, 17 June 2013 (UTC)

That is a gorgeous question. While there are advantages in policy writing to being ambiguous, my intention is that the amendment call for the critique of each primary source to be based on toxicological principles - hence, "critically reviewed for their relevance to humans under typical exposures and doses". I would primarily like the amendment (if it gets in) to disallow 2ndary or 3rdary sources that just uncritically cite primary sources; ideally the primary sources would be critiqued under tox principles. Would be happy to discuss changes to accommodate concerns that raises...Jytdog (talk) 22:44, 17 June 2013 (UTC)

follow up, WAID - if what you are looking for is "systematic review" as opposed to "critically reviewed for their relevance to humans under typical exposures and doses" I could live with that as it would exclude what I want to exclude...Jytdog (talk) 22:56, 17 June 2013 (UTC)

I don't really think you'll be happy with that outcome. Consider two papers: a mindlessly performed systematic review of garbage primaries, and a thoughtful literature review (perhaps by a team of respected toxicologists) who tell you that the primaries are all garbage and therefore nobody actually knows whether there are any significant toxicity issues. Which one do you want to be using? WhatamIdoing (talk) 11:10, 20 June 2013 (UTC)

Can a systematic review be "mindlessly performed"?  :) Anyway... based on Colin's feedback I had to move away from "systematic review" ... please see new proposal below... can you help me navigate between you and Colin? Thank you again for commenting. Jytdog (talk) 13:01, 20 June 2013 (UTC)

• Oppose The entire basis of this proposal rests on Jytog's claim that Wikipedia has a "widespread problem" with content stemming from research that has been "conducted by basic researchers with no training in toxicology." The editor claims to abhor "cognitive bias" yet believes a presumptive statement such as this justifies amending the guideline? Are we now going to willfully discriminate against interdisciplinary science? Semitransgenic _talk. 15:39, 17 June 2013 (UTC)

Hi semi, yes I do abhor cognitive bias; that same page notes that I also abhor BS. The way that the results of some experiments are blown up - sometimes by their authors, sometimes by the press, sometime by advocacy groups - to make huge claims about toxicity is sometimes BS. Sometimes attacks on those studies are BS. Sometimes not, all around. The point of the proposed language, is that it takes scientific expertise to interpret and evaluate these studies to make useful, reliable claims about toxicity. I have already accepted the striking of "by toxicologists" so critical reviews by anybody (as long as they review critically for dose/exposure) would be OK. I want to help lift Wikipedia out of the BS-y garbage heap of alarmist claims about in vitro studies and alarmist attacks on those studies - Wikipedia should provide reliable, consensus-driven content about health, not driven by RECENTISM or the most recent controversy about some chemical.Jytdog (talk) 16:13, 17 June 2013 (UTC)

Do you see what I mean, semi? Can you support this amendment, or are there any changes that would make it more acceptable to you? Thank you. Jytdog (talk) 20:10, 17 June 2013 (UTC)

unfortunately i find your additional comments above (I would primarily like the amendment (if it gets in) to disallow 2ndary or 3rdary sources that just uncritically cite primary sources) problematic, this is turning in to an agenda based creepfest. Semitransgenic _talk. 23:35, 17 June 2013 (UTC)

Thanks for replying - it is unclear to me why having only systematic reviews count as useful, is problematic to you. But if there is no way we can work to find acceptable language, then I'll just say that I appreciate your time and your responses. Thanks again.Jytdog (talk) 23:50, 17 June 2013 (UTC)

• Support the revised version (without the demand that a toxicologist review the data). Misinterpretation and panic based on toxicity data can lead to extreme things, such as the recent RCOG report in the UK essentially telling pregnant women not to buy new cars or furniture. Common sense really needs to prevail in this area, and the guideline should be seen to support common sense. JFW | T@lk 15:57, 17 June 2013 (UTC)
• Support Toxicological info as with most medical content should be based on secondary sources. Doc James (talk · contribs · email) (if I write on your page reply on mine) 16:14, 17 June 2013 (UTC)
• Support updated wording. I don't see this as a fundamental change, it simply makes more clear what the guideline already says. I've come across enough questions in this area to agree that the guideline should be clarified. Zad68 03:52, 18 June 2013 (UTC)
• Oppose As the above editor Zad68 mentions, this is not a fundamental change. It doesn't make it more clear. It is quite wordy and just adds more burden to the poor readers trying to get a grip of the sprawling landscape of Wikipedia rules and regulations. II | (t - c) 04:08, 18 June 2013 (UTC)

• Oppose most of it could be shortened to "Primary studies should not be used to support content that discusses toxicity." What if there is a primary study that says something is toxic, and secondary sources haven't caught up to it yet. It needs to be documented on Wikipedia. I think you are referring to the ethics of primary studies to imply its use for humans. Which I see no harm of ethics there. There is already a section that says preclinical tests results shouldn't be assumed for human implications.

Separately, as for ethics of a study, it should be inserted separately from this. in terms of ethics for animals, primary studies that are in situ and in vitro studies should be encouraged. I would need advice on studies that are in vivo, since the tradeoff for knowledge and for what may be unethical animal tests. Having a guideline that discourages entry from in vivo sources would encourage tests that are in situ or in vitro. I see nothing wrong with including in situ and in vitro studies that are primary. Requiring secondary sources only, does not stop this. - Sidelight12 ^Talk 04:47, 18 June 2013 (UTC)

Sidelight, if you see nothing wrong with including in vitro studies that are primary, it is no wonder you oppose this edit. Your views do not reflect the MEDRS guideline at all. Wikipedia should not have a mandate to document information in primary sources that the secondary sources have not caught up with. This is what MEDRS is all about. Lesion (talk) 08:49, 19 June 2013 (UTC)

My view is medrs related. See the section: "Avoid over-emphasizing single studies, particularly in vitro or animal studies" it says these sources are invaluable, and allows their use if used with care. medrs is made up of a collection of contributions by different people's thoughts, and it is a living document, so to say that makes no sense. As much bias as there is against primary sources, medrs doesn't prohibit them. I believe there may be a guideline or reason for a guideline to dictate primary sources, but I don't believe they should be dismissed. One is in place already, and I may understand there being something like this in place for toxicology. - Sidelight12 ^Talk 10:40, 19 June 2013 (UTC)

• Comment I think the current text (without the "written by toxicologists" part) is in alignment with MEDRS and helpful explanation. But I'm concerned that MEDRS is already too long. There's no reason we can't have a separate (short) page giving advice on toxicology. It would be classed as an Essay rather than a Guideline, but plenty good advice pages on WP are so-classed. This could be linked to from MEDRS. WP:TOXIC anyone? -- Colin°^Talk 07:39, 18 June 2013 (UTC)

Thank you for commenting. I would greatly prefer to include this in MEDRS. Would you accept a much shortened version, something like this? Jytdog (talk) 03:59, 19 June 2013 (UTC)

"Content discussing the toxicity of substances falls within this guideline, and should be sourced from secondary or tertiary systematic reviews. Primary studies should not be used to support content that discusses toxicity."

The first clause is fine and may be helpful to include. I oppose requiring systematic reviews. Or any one kind of source like "reviews". The vast majority of issues to toxicity are uncontroversial and can be sourced to any number of reliable documents. We must be careful not to write policy/guideline on the basis of solving only a few specific hard problems because then we raise the bar so high that all the easier articles can't be written at all. Colin°^Talk 09:24, 19 June 2013 (UTC)

I hear you Colin... kind of. But I believe you have stated here many times, that we need to ensure that health related content in Wikipedia reflects the actual consensus. With toxicology content, especially with the rising concern about the risks of "chemical exposure" since the 1970's, there is more and more pressure to include content that is scary and does not reflect the consensus, but instead reflects the most alarmist sides of topics where there is no consensus. Jfdwolff mentioned the recent RCOG advisory, which is here: http://www.rcog.org.uk/files/rcog-corp/5.6.13ChemicalExposures.pdf - if you look at that, it warns pregnant women away from an enormous number of things, based on pretty vaguely defined risk. At Wikipedia, we really need a higher standard for tox to avoid this kind of alarmist discussion. It is a problem and it is only getting worse. I am willing to back off systematic reviews, but would like to keep the qualifier, but only for products that are currently marketed. That way, the standard for content on those, is spilt off from tox content about products that have been withdrawn b/c their tox is well established (e.g. PCBs, asbestos). Also am willing to dump the last sentence. Current version with those modifications:

"Content discussing the toxicity of substances falls within this guideline. Content on the toxicity of substances that are still marketed should be sourced from secondary or tertiary sources in which primary sources are critically reviewed for their relevance to humans under typical routes of exposure and doses."

I disagree with dismissing primary sources altogether, but I might agree to there being a certain guideline for their use in toxicology. I think your concern is promotion of, or conflict of interest for substances. Something like primary sources must be peer reviewed, and if there is a reason for conflict of interest for this source consensus can argue against its use. Another guideline to prevent this use may be needed. I don't want that card used it used if this is not the case, just because someone doesn't like it. - Sidelight12 ^Talk 10:50, 19 June 2013 (UTC)

Using primary sources should be done rarely if ever, for health information. There is almost never a justifiable reason to use one. Jytdog (talk) 02:28, 20 June 2013 (UTC)

• Comment - The only potentially controversial aspect left here is that the edit specifies systematic reviews, whereas MEDRS for other topics requires only "reliable" secondary and tertiary sources. This wording you have compromised with after an editor above challenged the use of the phrase "critically reviewed". Also what is a tertiary systematic review? A systematic review of other systematic reviews? Not heard that term before. I think your argument is, that merely saying secondary sources is not a tight enough restriction on sources for toxicity, because the primary in vitro studies need to be interpreted by experts in the field to be of any relevance, and secondary sources which just cite these primary sources are not suitable sources? Maybe instead of using the term "systematic reviews" just use "reliable", and if need be qualify this afterwards with a rough description of what reliable means in toxicology? Lesion (talk) 09:07, 19 June 2013 (UTC)

Thanks Lesion. That is a great point, "tertiary systematic review" is goofy. I am getting stuck between Colin and WAID here, which is what I suspected might happen. Gonna try another modification above... Jytdog (talk) 02:28, 20 June 2013 (UTC)

• Work needed: I agree with your goals, but the proposal is much too wordy, containing (still) quite a few clauses which are redundant and already part of MEDRS. WP:CREEP-- please try to substantially reword so that the new text doesn't repeat what the guideline already says. Of course toxicology is part of MEDRS. Of course we should rely primarily on secondary sources. We don't need to repeat all of this. I support the proposal intent if the wording is shortened; I oppose the wording as written. SandyGeorgia (Talk) 03:06, 20 June 2013 (UTC)
  • It might be useful to write up an essay talking about how to evaluate a tox paper. It might be useful to editors who are trying to do the right thing (i.e., those not trying to win a dispute) as well as helping crystallize a few points that would be useful here. WhatamIdoing (talk) 11:10, 20 June 2013 (UTC)
• Oppose I'm concerned about the consensus which seems to be developing here, which I believe threatens to become a classic case of WP:LOCALCONSENSUS. I know it's only on the talk page, but when an editor says that "at Wikipedia, we really need a higher standard for tox to avoid this kind of alarmist discussion" and the "alarmist discussion" is a paper from the Royal College of Obstetricians and Gynaecologists, something is wrong. Wikipedia should, of course, summarize published material from such a source in an entirely neutral way regardless of what editors think about it. The purpose of MEDRS is in danger of being lost in Wikilawyering about primary/secondary/review sources. The purpose is to prevent Wikipedia being used to peddle quack remedies and make claims which may damage the health of readers if they act on them. Claims of medical efficacy and warnings of possible toxicity are not the same; much higher standards of evidence should be required for the first than the second, where the precautionary principle applies – although in all cases the strength of the evidence should be reflected in the writing. If a herbal product meets the standards of notability through its widespread use, as documented in reliable sources, and there are reviews which meet the standards of MEDRS to show some degree of efficacy plus some recent primary reports of toxicity not yet reviewed, it would be irresponsible censorship not to mention (in a properly qualified way) the possible toxicity in the article. (It might even expose Wikipedia to legal action in some jurisdictions should the toxicity be confirmed and the product cause harm to someone and it could be shown that Wikipedia editors knew of the reports of toxicity and suppressed them.) Peter coxhead (talk) 09:21, 20 June 2013 (UTC)

Peter, I strongly disagree that we should worry about "legal action". Please read the terms of use, disclaimers etc. before you make such suggestions. Irresponsible? - possibly this could be argued, fine. Illegal? - this is suggestion is a joke and does not qualify as legitimate point in this argument. No-one is going to sue me, or you any other Wikipedia editor (anonymous volunteers) if they fail to add a scientific paper about the toxicity of a substance to Wikipedia, with these terms of use etc in place. Lesion (talk) 09:33, 20 June 2013 (UTC)

Legal action or not, that is a valid ethical reason. If knowledge is available I dislike discriminating against it, when nothing is wrong with it. A vast amount of information is in primary sources that aren't in secondary sources. With good reason some primary sources may be invalid to the context of an article. What about a guideline that says sources must be peer reviewed or reliably published. Also, the source must be third party to affiliations of a substance. Third-party sources can be primary, secondary, or tertiary. - Sidelight12 ^Talk 11:17, 20 June 2013 (UTC)

Are you talking about changing the wording of this proposed edit about toxicology sources, or about changing MEDRS generally? If latter, new section with specific proposed changes delineated please. Lesion (talk) 11:57, 20 June 2013 (UTC)

It covers toxicology, but for instances I can't see now, it could cover more than that. but ok, I might propose this soon to deal with this issue. One discussion at a time seems proper. - Sidelight12 ^Talk 12:09, 20 June 2013 (UTC)

Sidelight above you write " If knowledge is available" - and this is exactly the point. MEDRS exists because we only want to state what is known (or what the medical community believes is true as stated in 2ndary and tertiary sources, which is the closest we can come) - and there is too much content in WIkipedia making claims that X or Y is toxic in A or B ways based on sources that don't take toxicology into account - heck there is too much such content based on primary studies. In o other words, too much content stated something as "known" when it is not known.Jytdog (talk) 12:58, 20 June 2013 (UTC)

Ok, forget the legal aspect (although in many countries attempts to disclaim responsibility through things like terms of use have no legal force). Please answer my main point that efficacy and toxicity aren't the same. The latter is subject to the precautionary principle. We shouldn't state on the basis of primary studies that a particular treatment is efficacious. We should state on the basis of reliable primary studies (preferably plural here) that there are preliminary reports of toxicity. Peter coxhead (talk) 19:15, 20 June 2013 (UTC)

I disagree strongly with you Peter. It is exactly because of the philosophy you are stating here that I wish to strengthen MEDRS specifically with respect to toxcity. MEDRS states "Therefore, it is vital that the biomedical information in all types of articles be based on reliable, third-party, published sources and accurately reflect current medical knowledge." This sentence (and the guideline) makes it clear Wikipedia should not include content that is "preliminary" about any health related matters, and especially not about emotionally laden preliminary conclusions that "X might be killing your baby". The goal is not to be "cutting edge" but instead to be the fat old cow walking at the end of the wagon train - we provide content on what is known - not what is speculated or preliminary. Wikipedia is not the place to go for the latest medical news. It is a place to find reliable knowledge as reflected in secondary sources. Jytdog (talk) 19:45, 20 June 2013 (UTC)

Strongly agree with Jytdog, and s/he has said it better than I could. MEDRS is not just about efficacy of treatments, but about any matter pertaining to human health. People going on about using primary sources here need to actually read MEDRS and WP:RS and realize there is encyclopedia wide policy relating to not using primary sources. Lesion (talk) 20:00, 20 June 2013 (UTC)

I have read (and regularly use) MEDRS, and I'm also very familiar with WP:RS. But editors here need to stop Wikipedia:Wikilawyering by endlessly referring to WP guideline documents, and deal with the issues: "Wikipedia policies and procedures should be interpreted with common sense to achieve the purpose of the policy..." Discuss the issues. Peter coxhead (talk) 20:20, 20 June 2013 (UTC)

Hi Peter, I tried to discuss both the letter and the spirit of policy and guideline, specifically with respect to toxicity. And I tried to directly speak to what you said about warning people. Please tell me what part of the issue you feel I have not addressed. Thanks! Jytdog (talk) 20:24, 20 June 2013 (UTC)

My comment was directed at the one above, by User:Lesion, rather than yours. But the point remains that it's not helpful to keep re-referring to MEDRS when we are discussing what should be in MEDRS. The discussion needs to focus on the purpose of the policy. So I look forward to your reply to my question below. Peter coxhead (talk) 20:34, 20 June 2013 (UTC)

• Weak support. While I agree with the implication of the current revision, it seems to be unnecessarily long-winded. How about something like "Information about the toxicity of substances (both drugs and non-drugs) should be supported by secondary sources." Axl ¤ [Talk] 10:52, 20 June 2013 (UTC)

Thank you for the support, weak as it may be! The wordiness is unfortunate - I have been trying to carve this down to the minimum that (I think) is necessary. The pressure that some editors feel to provide "fair warning" to readers about possible toxicity of currently marketed products (e.g comments by Sidelight and Peter Coxhead above) drives them to want to generate content based on conclusions drawn from primary sources (already pushing the envelope of both PSTS and MEDRS); for them (I believe), tox content is different from efficacy content and there should actually be a lower bar for adding content -- and this is exactly what has been concerning me. I don't think it is enough to just say "content about toxicity falls within MEDRS" - I think we need, within the higher standard that is MEDRS, a slightly higher standard again for content about toxicity for currently marketed products. And that takes more words. I hope that makes sense and that this was not too wordy. Jytdog (talk) 20:02, 20 June 2013 (UTC)

What is the rationale for requiring a higher standard for toxicity than efficacy? Please explain. Peter coxhead (talk) 20:20, 20 June 2013 (UTC)

Thanks for your note above. Happy to answer. I understood you to say above, that you approach tox content differently from efficacy content - that due to the precautionary principle you think that it is Wikipedia's responsibility to warn people about possible toxicities of currently marketed products based on preliminary findings described in primary sources. That is the first time I have heard the idea articulated so clearly (which I appreciate!), but in many Talk pages I have encountered similar motivation, not so clearly articulated. This approach is well-intentioned but the intention violates MEDRS and NOT/RECENTISM, and enacting it violates PSTS and MEDRS in the reliance on primary sources. I made my proposal to prevent this well-intentioned but misguided approach to tox content about currently marketed products; the higher standard serves as a counter-weight against the moral urgency some editors feel to (mis)use Wikipedia to warn the public about these possible dangers. I don't know if you agree but I hope you see the reasoning. Jytdog (talk) 20:53, 20 June 2013 (UTC)

Just re-read this and I think it is kind of dickish. Sorry about that. My intention wasn't to be so harsh, but it was to be clear. Hopefully it is clear. oy. Jytdog (talk) 21:28, 20 June 2013 (UTC)

Don't worry, I wasn't offended. Your argument for requiring higher standards for toxicity is basically to forestall what editors who disagree with you might do. Firstly, this is a poor reason: the reason should be based on a rational appraisal of the issues, not the hypothetical actions of other editors. Secondly, it's pointless, since those who disagree with you in this respect either haven't read MEDRS or aren't going to be influenced by the requirement for the same standard, let alone a higher standard.

A key issue for me is the requirement for editors to reach decisions by consensus on specific content in specific articles taking into account WP's guidelines. Trying to produce over-rigid guidelines to cover all cases (including hypothetical ones) is counter-productive (as we've seen in recent actions at the MOS generally).

My concern is not primarily with currently marketed "orthodox" medical products, and Wikipedia must never be responsible for the kind of scare which stopped parents in the UK giving their children the MMR vacine based on one individual's flawed "research". But many articles on plants which have been used in traditional herbal medicine do pose problems. Where reliable secondary sources attest the traditional and/or ethnobotanical uses of the plant, this information meets Wikipedia's requirements for notability and verifiability – verifiability in terms of the plant's cultural history and use, of course, not in terms of its efficacy. We can – and I regularly do – remove attempts to claim efficacy not supported to the standards of MEDRS. But for many readers, the mere fact that the plant has been used as a "herbal remedy" implies, even though we carefully do not say this, that it may be efficacious. Reviews of quality studies of the toxicity of many such herbal products do not exist; you surely know as well as I do that what gets reported in journals is what scientists get funded to research. For example, for plants used in ayurvedic "medicine" there are large numbers of papers and reviews in Indian journals, but very few reports of toxicity issues, even though these are known to be a problem. It cannot be acceptable to be able to say that X is used in ayurveda, and is believed by ayurvedic practitioners to be effective in the treatment of Y, but not be able to say that a primary study by Z suggests caution in the use of X as toxicity has been reported.

In summary, if it were possible to completely remove the many, many descriptions of traditional herbal "remedies" in Wikipedia, and insist that no report of traditional use can be made without evidence of efficacy or non-efficacy to MEDRS standards, then I might agree with you. But it isn't possible. So I don't agree with you. Peter coxhead (talk) 14:24, 21 June 2013 (UTC)

Hi thanks for not being offended, and for the discussion. It is a little unfair to suggest that I want to amend the guideline "basically to forestall what editors who disagree with you might do." The desire to amend definitely comes from several actual difficult conversations; what that revealed to me was that the guideline was not providing guidance sufficient to keep alarmist, and toxicologically poorly done, material about toxicity out of wikipedia. I imagine this is the same impulse that produced this guideline in the first place -- a desire to have something to help resolve disputes about what is proper to include in Wikipedia. I hear you - a lot - on the issue with traditional medicines. I have done work on articles like that too. It is a struggle to create and keep MEDRS content on them in place. However, while I share your concern, it is off topic for this request for amendment... Jytdog (talk) 22:59, 21 June 2013 (UTC)

If it were agreed that higher standards were required for toxicology reports than efficacy reports, this would (of course) also apply to articles concerned with traditional medicines. Since there are many such articles, it would potentially have a significant numerical impact. So it's not "off topic", it's very much "on topic". The way to keep alarmist, poor material out of Wikipedia is to deal with it article by article. If it's truly alarmist and methodologically poor, it can be removed for that reason, and doesn't need changes to MEDRS which may have unforeseen consequences ("you can write that herbal product X is efficacious because the sources just meet the MEDRS standards but not that it might be toxic because this needs a higher standard"). Peter coxhead (talk) 08:24, 22 June 2013 (UTC)

NOW I understand, thank you for explaining. My amendment was intended for "non-drug substances" (which has now fallen out of the amendment)... and I see what you mean that even if the "non-drug substances" were back in, traditional medicines (which are essentially dietary supplements) are "non-drug substances" too! Had not considered that and it is a great point. Especially with the language as it currently stands. Colin's suggestions about a separate essay is starting to look more sensible, to deal with all this.

Current proposal: Include: "Content discussing the toxicity of substances falls within this guideline. Primary studies should not be used to support content that discusses toxicity." and I will generate a separate essay to Tox for everybody to consider.

Note - I am going back through and deleting previous proposals so this is easier to track. Jytdog (talk) 14:20, 22 June 2013 (UTC)

I think it would be desirable for you to start with the essay. Perhaps you could use diverse classes of examples, like pesticides, prescription drugs (vaccines?), herbal treatments, and vitamins, to explain the typical concerns that editors have. Also, I hope that it will emphasize the one important area of agreement, which is that if excellent secondary sources exist, then they ought to be relied upon. WhatamIdoing (talk) 14:33, 27 June 2013 (UTC)

• Toxicity is already covered by MEDRS, and much of the material is redundant since it is already in the guideline. Maybe have a short sentence listing some of the topics covered by MEDRS instead? IRWolfie- (talk) 09:54, 22 June 2013 (UTC)
• Colin's suggestion makes eminent sense. The use of "non-drug substances" phrasing is potentially problematic in the context of CAM, at least in the US. Most herbals are normally considered as food, but garlic "for hypertension" or even "relaxing" chamomile must properly be considered a drug. Almost anything found in homeopathic materia medica is inert as used (or at most has a tiny dose of ethanol as dilutant). These things still must be considered as drugs because of associated indications, irrespective of whether there is any sign of either specific efficacy or toxicity. As Mr Bumble says, "The law is a ass", but it still is the law. Until there's an outbreak of sanity among legislators we're stuck with it, so let's ignore the drug/non-drug distinction and focus on the toxicity question. How about wp:TOXRS, aka Wikipedia:Identifying reliable sources (toxicity) as a working title for the essay? I'm thinking that we'd generally treat things such as ClinicalTrials.gov or statistical summaries from poison control centres as (low grade) secondary sources, just barely good enough to use with cautious wording. I'd expect some tension at play between avoiding alarmism on one hand and applying the precautionary principle which is so dear to environmentalists on the other. LeadSongDog come howl! 15:18, 27 June 2013 (UTC)
• The problem you appear to be having is reviews you don't like rather than editors citing primary sources directly. Also, people aren't going to change their votes based on the revised proposal, so this is dead unless you start a new section. As a general comment, as WhatamIdoing has articulated quite clearly above, I don't think you'll have much luck accomplishing what you want through editing policy pages. Personally, I don't think policies should be edited to fit specific disputes (see #8 in the cynic's guide). Your introduction and later comments revealed that you believe "we need, within the higher standard that is MEDRS, a slightly higher standard again for content about toxicity for currently marketed products". Your proposal included restrictions upon using reviews based upon the author (a "true" toxicologist), the type of review ("systematic"?) and how much analysis the review gives to the sources it cites. These look very susceptible to gaming. To get into the weeds of one example you cited (Bisphenol A), you're probably not going to be able to remove the perspective of the Chapel Hill scientists such as vom Saal just because the reviews discuss a lot of animal studies and regulators from the FDA (altho not necessarily their scientific advisors) and EU don't really agree. II | (t - c) 08:45, 29 June 2013 (UTC)

Hi ImperfectlyInformed - you are pretty much missing my point. Chapel Hill is not part of my problem with the BPA article and other articles. The main problem is the train wreck of primary sources which under PSTS and MEDRS should not be there at all. There are also some really terrible 2ndary sources cited for tox content that don't review the primary sources they cite using basic tox principles (I linked to some above relevant to BPA and glyphosate), but I have already given up the idea that I can change this guideline to exclude them, as you can clearly see in the current proposal - I have accepted that making that change would affect too many established articles. My issue with some reviews is not that I "don't like" them - by which I think you mean their conclusions - but rather that some secondary sources that make tox claims, fail to use toxicological standards and methods when citing primary sources, making their claims unreliable. The issue is quality of the review through which conclusions are arrived at, not the content of the conclusions themselves. Reliable toxicology says what it says. Which is sometimes grey, not black and white, which is fine, as long as it is actual toxicology and not just hand waving. As to your broader comments - as I have written here a bunch of times; toxicology is a discipline (an important one) within the biomedical sciences and there are way too many "armchair toxicologists" out there. I already have adapted the proposal to the several criticisms so I am not sure why you are rehashing old stuff. In any case, am looking to finish the draft essay this weekend and hope to post it Sunday evening. Will be fun. Jytdog (talk) 09:24, 29 June 2013 (UTC)

The topic of low-quality reviews has come up before - see for example a discussion from 2008 called Dated and low-quality reviews. Back then I was (not very coherently) raising the question of low-quality reviews, and I was also concerned about obscuring primary sources behind the uncritical parroting of a narrative review. There's no simple solution to that problem. Reviewing Bisphenol A, it looks to me like your approach was more deletion rather rewriting, and you also lacked persistence. You have to keep hammering away and incorporating suggestions to produce a consensus. Ultimately I think most people would be happy to see that particular article trimmed down to summary style. By the way, as you probably know there are plenty of scientists who think the Chapel Hill "consensus" is alarmist, so it's a little hard for me to understand where you would draw the line. I mean, if you've got the FDA and EU regulators disagreeing with a source, how is that source OK but the source which you personally (but perhaps no sources) disagree with as "fail[ing] to use toxicological standards and methods" not OK? (I'll admit the disagreeing sides here make it complicated.) This gets into the thorny issue of editors making authoritative judgments on which reviews are valid. I am of course not opposed to editorial research, and I've been involved in cases where, based on editorial consensus, reviews were not used because they were low-quality. However, rewriting the guideline ultimately doesn't change anything: you have no way to prove that you're right, so it comes down to editorial consensus. II | (t - c) 10:09, 29 June 2013 (UTC)

Interesting, nice to see that you struggled with this too. MEDRS itself puts boundaries on the kinds of sources that can be used, and in the Assess section, calls for editors to look at the source and consider what it is before using it. For interventions, meta reviews of RCTs are best, "Systematic reviews of bodies of literature of overall good quality and consistency" are next, etc etc. I was looking to add something to MEDRS along those lines for tox, where it would be, for tox, "reviews of bodies of literature of overall good quality and consistency that take into account route of exposure and typical doses to which humans are exposed". It is not a big leap. Your line of critique about what I "personally" view as as X or Y when no other source calls it that, a) runs up against MEDRS's call for editors to assess, and b) is a red herring - as you well know it is very rare for one review to critique another directly. And again I am not out to drive content in one direction or another. I am trying to get rid of tox content based on sources that don't address basic tox principles. Toxicologists can disagree with one another; it is not for me or Wikipedia to sort that out but rather just to describe the mainstream viewpoints. On endocrine disruptors, my understanding is that mainstream toxicology has consensus that it is a good thing that DES was taken off the market and that there is a risk that other chemicals on the market (and chemicals endogenous to various foods) ~may~ be doing similar harmful things in a more subtle way, but I think that is as far as the broad consensus goes. From there you go to individual chemicals and you find ranges of tox opinion about various risks of harm. The science appears to me, to be somewhat at the end of its rope right now, until more, and carefully done, research is completed. I am very heartened that there is a concerted effort going to coordinate research among academics, the NIH, and regulators to get definitive answers to questions raised about BPA by the Chapel Hill Group, the big NTP review, and the decisions/report by the FDA based on both reports http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3548284/ see especially this ("The consortium represents an unprecedented approach to conducting GLP-compliant research by bringing researchers and regulators together during the planning stage to ensure that results will be maximally useful for risk assessment and regulatory decision making. The grantees and FDA representatives, along with coordinators from the NIEHS/NTP, held their first in-person meeting in March 2012. This collaboration is expected to produce a robust and valuable body of work on the effects of BPA in rats, a key animal model in toxicity testing.") So no, I don't think the Chapel Hill report is outside the consensus and I have never stated a desire to remove it. Jytdog (talk) 15:21, 29 June 2013 (UTC)

DARE guidance?

I'm wondering whether the DARE Database of Abstracts of Reviews of Effects [1][2] could somehow fit into MEDRS as a useful tool to gain unbiased guidance on the reliability of a given systematic review. For many recent reviews on their database [3] DARE provides brief critical appraisals (box here [4]) of the source, with nuanced guidance on the reliability of the authors' conclusions (also accessible via TRIP). For example:

• Questions about the quality of included data and methods of synthesis mean that the authors' conclusions should be treated with caution. [5]
• In general, the authors’ conclusions appear to be appropriate, although they depended on the model assumptions. [6]

As an individual contributor, I find guidance such as this useful for nuancing the weight to give to the findings of particular systematic reviews. 86.161.251.139 (talk) 10:13, 11 July 2013 (UTC)

Agree - less than half (and reducing) have those appraisals, but they're great when they're there. There are several things to know about it - not least of which is to be careful if not going to DARE itself, how often it is updated where you're using it, because there are big differences, and how to request an appraisal. I'm building up a file with quite a few things like this missing from the systematic reviews page.Hildabast (talk) 02:16, 12 July 2013 (UTC)

Reverted edit

I have concerns about a statement in the "Biomedical Journals" section:

• It is usually best to use reviews and meta-analyses where possible, as these give a balanced and general perspective of a topic—and are usually easier to understand.

While agreeing that that narrative reviews tend to be relatively reader friendly, the same can scarcely be said of meta-analyses. For example, here is a recent open-access report I randomly selected on PubMed: [7] [8]. I doubt many general users of Wikipedia would find that easy reading. Also, I don't think that meta-analyses (a particular type of systematic review) can be said to provide a "general perspective of a topic". For methodological reasons, they have to address a rather specific research question. As noted elsewhere in the section, "whereas a narrative review may give a panorama of current knowledge on a particular topic, a systematic review tends to have a narrower focus."

I don't see the sentence adds anything useful to the paragraph and I suggest removing it. 86.161.251.139 (talk) 15:06, 11 July 2013 (UTC)

Disagree... the particular meta-analysis you selected would be readable to an editor familiar with the work in the field, I'd think. And it's a particularly "deep" example, in my opinion. Here's one that I think is more typical: [9], conclusion is "On the basis of the current research, any clinical benefit of oral glucosamine for patients with chronic LBP and radiographic changes of spinal OA can neither be demonstrated nor excluded based on insufficient data and the low quality of existing studies." - that's very accessible language to a typical editor. Don't forget that we're talking about recommended sources for editors to use. The job of the editor is to take the source and represent it accurately and accessibly to the end-reader. I find meta-analyses are useful, they're usually a dozen pages of methodology followed by one paragraph of conclusion, and it's just that conclusion we're looking for, as editors. Zad68 15:28, 11 July 2013 (UTC)

Maybe you've misunderstood my point... I've absolutely nothing against encouraging contributors to use meta-analyses. Far from it! And I fully take your point that meta-analysis conclusions have to be clearly defined. However, saying that meta-analyses, with their pages full of methodological technicalities, are "easier [than what?] to understand", seems—as written—a pretty strange claim to make. Since any meta-analysis worth its salt addresses a sharply defined PICO research question, it's hardly the ideal port of call for a "general perspective of a topic"; as pointed out a few lines below, that's the role of a good narrative review.
Fwiw, the paper you cite as being more representative [10] is a systematic review which does not include any statistical meta-analysis. Here's a randomly generated sample of five more free-access reports of clinical meta-analyses recently indexed by the NLM: PMID 23639488, PMID 23637132, PMID 23587198, PMID 23558282, PMID 23558164. 86.161.251.139 (talk) 17:41, 11 July 2013 (UTC)

Ok so if the issue isn't that meta-analyses should not be recommended, but that it's incorrect to call meta-analyses "easier to understand", we can get somewhere with this. The edit that was reverted made this change: "It is usually best to use reviews and meta-analyses where possible, as these give a balanced and general perspective of a topic—and are usually easier to understand." I think the issue was that the edit removed the characterization of reviews as "easier to understand" and it looks like we all agree reviews are indeed easier to understand (although the guideline doesn't say what is it that they're easier than). I agree that an individual meta-analysis does not often provide a general, easy-to-understand overview of a topic, and they're not really designed to. They may have a paragraph backgrounder and then jump right into the data. Suggested change is:

It is usually best to use reviews and meta-analyses where possible, as these . Reviews in particular give a balanced and general perspective of a topic—and are usually easier to understand.

Comments? Support/oppose? Zad68 17:59, 11 July 2013 (UTC)

No need I can see for an RFC-type discussion with Support/oppose etc... Personally, I think your suggestion is an improvement. Though I would again question whether systematic reviews give a "general perspective of a topic" (they also have to address tightly defined PICO questions), or indeed whether they are necessarily "easier to understand" (they too are tied to strict reporting of their methodology).
In general, I feel that this part of the page could be better explained. (For instance, it would make sense to me to start a section on 'Biomedical journals' with "Peer reviewed medical journals are a natural choice as a source for up-to-date medical information in Wikipedia articles" rather than with "As mentioned above...".) Aside:I'd like to say that I'm a strong supporter of the guiding principles of MEDRS and my reverted edit was only intended as a small correction. I did the edit after making a somewhat more substantial proposal, #DARE guidance?, which I really do feel may be worthy of some consideration (eg to avoid WP:GAME?). 86.161.251.139 (talk) 18:33, 11 July 2013 (UTC)

86.161.251.139 you really seem to know your stuff and I hope you stick around and keep contributing to WP:MEDICINE-scope things. I see your IP User Talk page and won't bother with the "Welcome to Wikipedia, consider registering" song and dance. I feel like I might have run into you already. Be bold, make needed updates. The worst that happens is it gets reverted and we discuss it a bit. I'll put the small tweak to your change into MEDRS and see if anybody reverts it. Feel free to make more suggestions and changes. Zad68 18:41, 11 July 2013 (UTC)

Arbitrary break: "Review" terminology

What wording improvement can we make to make it clear we are saying reviews (PubMed Publication Type = "Review") and not systematic reviews? Zad68 18:49, 11 July 2013 (UTC)

"Literature reviews". But I believe that the PubMed search you name gives both types of review articles: literature reviews and systematic ones. WhatamIdoing (talk) 23:50, 11 July 2013 (UTC)

Regarding the terminology, the distinction generally drawn I think—though Hildabast may correct me—is between "narrative reviews" and "systematic reviews" (both of which are, strictly speaking, "reviews of the literature"). Regarding our wikilinks, I guess the narrative reviews redirect makes the distinction clear through the dab header. 86.161.251.139 (talk) 09:33, 12 July 2013 (UTC)

Well, you're right, but there's a bit more to it ;). That is the distinction usually drawn - but narrative is also used for non-quantitative (so if I can't meta-analyse, I present results narratively). It's not a satisfactory way to describe non-systematic reviews. It's on my mental list to start working on the relevant WP pages, because there's a lot that needs to be done there - having that more complete/accurate may help people decide what changes to make here. One of the biggest problems here at the moment is the guidance on searching: what's described here isn't the best way to search for reviews, because relying on publication types restricts you to MEDLINE-indexed reviews. That's problematic for two reasons: MEDLINE-indexing only indexes MEDLINE journals. Consequently, articles in purely PMC journals and Bookshelf/PubMed Health are missing. And secondly, MEDLINE-indexing takes months - as of a few days ago, for example, the most recent Cochrane reviews MEDLINE-indexed were from February. Combine those two problems, and for a search term like "asthma", you could be missing up to a 1000 reviews, the vast majority of them recent and/or full text. I'm working on an additional option at the moment, and will let you know when we get it organised. For systematic reviews, the option is PubMed Health. We include DARE, Cochrane, so it's non-MEDLINE/PubMed, and we're the ones gradually pulling in all the health technology assessment systematic reviews into the system (that are neither in DARE nor MEDLINE). PubMed Health updates DARE every week (some other places only do it once a month or once a quarter), and we'll be doing it more frequently than weekly sometime soon. Right now, your best bet is to search PubMed Health for systematic reviews, because everything is a systematic review (and there is no publication type systematic review anyway). Currently, when I look for reviews in PubMed, so that I also have a chance of seeing more non-MEDLINE-indexed ones, I do this: AND (review [pt] OR review [tw]).Hildabast (talk) 12:04, 12 July 2013 (UTC)

Sorry - didn't explain what I mean by health technology ones: I mean systematic reviews done by health technology assessment agencies. We digitise these and put them into the PubMed system as well as PubMed Health. Included are things like the NICE clinical guidelines that include full text complete systematic reviews, systematic reviews from agencies like AHRQ, the VA, IQWiG. Some of these also publish articles in MEDLINE journals, but most don't (even though they are peer-reviewed). So they meet our quality review, but not via the MEDLINE process as they are not journals.Hildabast (talk) 12:10, 12 July 2013 (UTC)

Great stuff! To try to keep the thread navigable, can I try to separate out two points here, and respond to them separately, postponing the searching question regarding the #Searching for sources section and PubMed Health etc until after another "Arbitrary break"?

• Regarding the #Biomedical journals subsection: Does this edit [11] help at all? At this point on the page, it seems to me, we're merely providing all our general users with a very brief (though hopefully terminologically correct!) introduction to the existence of different types of review articles, which may be deemed appropriate as sources in different editorial contexts. Can you perhaps improve the wording while keeping it simple for the benefit of everyone?
86.161.251.139 (talk) 15:25, 12 July 2013 (UTC)

Arbitrary break

While Zad68 was making a detailed edit, I was trying to make a more general (though perhaps hasty) one, along the lines I was trying to suggest above. For now, I've posted it as a bold edit [12] for BRD, if necessary. It will probably need some tidying anyway! 86.161.251.139 (talk) 19:21, 11 July 2013 (UTC)

I agree with the anonymous IP editor. I am happy with the recommended changes. Axl ¤ [Talk] 23:30, 11 July 2013 (UTC)

A few points - MEDLINE publication type review isn't just systematic and non-systematic reviews of literature, and it doesn't include meta-analyses if they are not reviews. I don't agree with the statement that non-systematic reviews are broader: or because of their systematic methodology that they lack contextual information. Go to a literally 1,000 AHRQ review, that includes key questions on most aspects of care, or one like lactose intolerance. Being systematic doesn't preclude these things - and being non-systematic doesn't ensure their presence. There are publication type reviews that are only letters, for example (and quite a few!).Hildabast (talk) 02:28, 12 July 2013 (UTC)

Aren't all (published) meta-analyses also reviews? Axl ¤ [Talk] 10:03, 12 July 2013 (UTC)

A meta-analysis of, say, two major trials may not involve any review (systematic or otherwise). 86.161.251.139 (talk) 10:20, 12 July 2013 (UTC)

I think I understand where Hildabast is coming from. And I agree it's complex...
A premise: As I understand it at least, unlike professional reporting guidelines [13], MEDRS has to serve in practice as both 1) an editorial tool to help ensure unbiased sourcing of medical content in keeping with Wikipedia's principles and culture, and 2) a user-friendly guide to best medical sourcing practice for the entire range of good-faith contributors. Not an easy combination, especially given the key importance of ensuring that the guideline does not lay itself open to abuse. And all this while keeping it as short and sweet as possible... What I think we're cautiously trying to do together here is to enhance the guideline without breaking something that already works pretty well.

In reply to at least a couple of the points Hildabast raises:

• "I don't agree with the statement that non-systematic reviews are broader." I've now tweaked the wording slightly [14]. IMO, we don't want to discourage in any way appropriate sourcing of up-to-date general encyclopedic information—as distinct from identification/sourcing of specific current best evidence—from recent high-quality, open-access narrative reviews and authoritative textbooks (e.g. Robbins, Harrison's et al). While this distinction may ultimately be a fuzzy one, I think there's general consensus on the project to encourage a judicious mix of high-quality journal and book sources, including open-access narrative reviews which general users can readily reference for more discursive information on a topic.
• AHRQ reviews. I wholeheartedly agree about the usefulness of AHRQ, NICE and other reliable bodies that systematically evaluate best evidence, and I'd consider this type of source to be our gold-standard "ideal source" for clinical evidence, when available and applicable. Maybe this could be expressed a bit more explicitly? (I also have a cautious idea that the focus of the #Assess evidence quality section could perhaps be enhanced by somehow incorporating AHRRQ, NICE, DARE, etc.)
86.161.251.139 (talk) 12:48, 12 July 2013 (UTC)

I'm parking this at the bottom of this discussion for lack of a better place, in case new editors here haven't encountered it: Wikipedia:Wikipedia Signpost/2008-06-30/Dispatches. Tim Vickers and Eubulides wrote it for the Dispatches in 2008 ... some of it might be outdated (haven't checked), but I think a lot of it is still useful. SandyGeorgia (Talk) 01:30, 19 July 2013 (UTC)

Guidance on searching

I seem to understand that Hildabast is suggesting (above [15]) that PubMed Health has the potential to become the default MEDRS tool for identifying "ideal sources" for clinical evidence. To me that sounds a rather attractive prospect. At present, we provide contributors with only the barest of guidance on #Searching for sources. Could it perhaps also provide an effective filtering tool to #Assess evidence quality?
86.161.251.139 (talk) 16:15, 12 July 2013 (UTC)

Well, I think it's already the main way to look for systematic reviews, for a few reasons - but not for every other type of review, or for recent non-Cochrane systematic reviews (because DARE can take weeks or months). Working on improving options at PubMed now, that has the best of both worlds. PubMed Health has additional content though - PubMed is a database of citations, PubMed Health includes a growing body of evidence-based resources for clinicians and patients. So the situation's in transition. I'm finding watching and participating in WP:Med very helpful actually for considering what improvements would be valuable (including ways of facilitating citation directly - eg a PMHID citation method that incorporated the PMID and the extras at PubMed Health, eg for Cochrane the plain language summary, for reviews in DARE the quality appraisal and so on). Hildabast (talk) 23:05, 12 July 2013 (UTC)

Handouts?

• About PubMed Health
• Finding systematic reviews in PubMed Health
• Review in Journal of the Canadian Health Libraries Association

86.161.251.139 (talk) 09:54, 14 July 2013 (UTC)

I suppose I should have read further before parking Wikipedia:Wikipedia Signpost/2008-06-30/Dispatches in the section above; some of it may be useful here. SandyGeorgia (Talk) 01:31, 19 July 2013 (UTC)

There's a confusion here between meta-analyses and systematic reviews. Meta-analysis is a statistical tool for combining the results of studies. They don't need to be trials, and they don't need to be part of a systematic review. It's just a statistical technique. Meta-analyses can be conducted on subsets of studies (eg a pharmaceutical company may meta-analyze its own trials on one of their products). Many systematic reviews have no meta-analyses in them (because there weren't multiple studies similar enough to combine). One has to be really careful with these terms, and not use them interchangeably. One is a type of research project/methodology, one is a statistical technique. Hildabast (talk) 15:49, 24 July 2013 (UTC)

Authoritative textbooks

On a related note, it would be nice to get a list of "authoritative textbooks" for each specialty area. People ask for suggestions every now and again, and it might help. I think that the pages for each WP:MEDTF task force would be the place to put it, rather than here. WhatamIdoing (talk) 23:42, 12 July 2013 (UTC)

I was going to add one to the Wikipedia:WikiProject Medicine/Neurology task force but wasn't sure where it would go. And looking at the talk page/history, is there anybody active there to read it? -- Colin°^Talk 07:22, 19 July 2013 (UTC)

MEDRS recommmended as a useful guide on how to find info on the web

I have just seen in a blog on medicine and skepticism (science-based medicine) that a comment in a post on how to find medical reliable info on the web cited MEDRS as a good guide.

The link here, the author of the post (not of the comment recommending MEDRS) is Steven Novella.

Most probably the author of the comment recommending MEDRS as a great guide on how to find good sources for medical content on the web is a regular here, but nevertheless I believe it is interesting to know that not only articles are "citable" and useful for people outside the project, but also guidelines, policies, etc.

--Garrondo (talk) 06:24, 19 July 2013 (UTC)

Proposed addition for identifying possible poor journals

While this guideline does a good job of identifying good journals and how to find them in the last paragraph of the Biomedical Journals section, I do not believe it does a good job of identifying poor journals (how to spot one, what to do when you have one). I bring up this proposal now because as I have noticed a large uptick in what can only be described as bad journal articles when I do a standard PubMed search. Unfortunately, PubMed Central is now indexing a number of non-MEDLINE indexed journals of dubious quality due these journals meeting their open access criteria (somewhat related previous discussions about open access journals and their reliability can be found here and here). Given the recent publicity about predatory journals, this problem will likely get worse, so I think we should deal with this now. This proposal is not meant to discuss open access journals per se, as there are plenty of good open access journals and poor print journals. However, I think we need to do a better job of directing editors away from poor journals, as well as dealing with the POV pusher who insists on using them.

Proposed addition:

An integral part of finding high quality sources is avoiding articles from journals without a reputation for fact-checking and accuracy. A red flag that a journal article is probably not reliable for health claims includes publication by a publisher that has a reputation for exhibiting "predatory" behavior, such as Bentham Science Publishers, Dove Medical Press, or OMICS Publishing Group, among others. These publishers may have questionable business practices and/or peer-review processes that raises concerns about the reliability of their journal articles. Other indications that a journal article is probably not reliable is its publication in a non-MEDLINE indexed journal, or the publication of an article whose content is outside the normal scope of that journal (for instance, a psychiatric nursing journal publishing an article on the treatment of cancer). Determining the reliability of any individual journal article should take into account whether the article has garnered significant citations in higher quality sources, suggesting wider acceptance in the medical literature despite any red flags suggested here.

I suspect that this addition merely writes down what many of our WP:MED editors already do when looking at the reliability of an article. Comments? Yobol (talk) 23:35, 23 July 2013 (UTC)

• I support inclusion of something of this nature, but the wording may need adjustment ... highlighting specific journals can be tricky. Also for new audiences to the MEDRS page, "predatory" might be defined on first occurrence (you have the link later). SandyGeorgia (Talk) 12:51, 24 July 2013 (UTC)

• Comments/Questions - 1) The sentences " A red flag that a journal article....reliability of their journal articles." are void for vagueness because they instantly beg the question "What is "predatory" behavior?". 2) The "MEDLINE indexed" thing might be a potentially valuable rule, but we should include instructions for naive users on how to check to see if a journal is MEDLINE indexed. 3) "garnered significant citations in higher quality sources" should be changed to "garnered significant citations in high quality sources" or "garnered significant citations in sources of undisputed reliability". 4) Completely appreciate the difficult question Yobol is trying to address. I'm not super familiar with familiar with WP:MEDRS, but it strikes me that most of the ideas being expressed here by Yobol could probably apply to other scientific literature. Do these ideas not already exist on other policy pages? NickCT (talk) 13:00, 24 July 2013 (UTC)

There is a general policy called Wikipedia:Identifying reliable sources and then there is Wikipedia:Identifying reliable sources (medicine). No other project has written its own guidelines because no other project has had a daily need to send people to a customized explanation of good sources. Blue Rasberry (talk) 18:25, 25 July 2013 (UTC)

• Comment does this include Elsevier? They have a reputation for publishing fake journals Elsevier#Sponsored_journals. Pubmed indexing says little. This article was half copied from Wikipedia (and they copied wrong stuff) and half copied from a NEJM article [16]. Doc James (talk · contribs · email) (if I write on your page reply on mine) 16:26, 25 July 2013 (UTC)
• Comment Elsewhere in this discussion some users ask for a definition of "predatory behavior". There is this - Predatory open access publishing. There may be sources which could generalize this article to create an article called Predatory academic publishing practices, but right now, the unregulated predators seem to be using open access platforms because people have confused expectations about how open access works and thus bad people attack researchers from those platforms. Blue Rasberry (talk) 18:31, 25 July 2013 (UTC)

Proposal #2

Thanks for the comments, have attempted to address them below:

An integral part of finding high quality sources is avoiding articles from journals without a reputation for fact-checking and accuracy. A red flag that a journal article is probably not reliable for health claims includes publication by a publisher that has a reputation for exhibiting "predatory" behavior, which include questionable business practices and/or peer-review processes that raises concerns about the reliability of their journal articles. (See "Notes" section below for examples of such publishers.) Other indications that a journal article is probably may not reliable is its publication in a journal that is not indexed in the bibliographic database MEDLINE, or the publication of an article whose content is outside the normal scope of that journal (for instance, a psychiatric nursing journal publishing an article on the treatment of cancer). Determining the reliability of any individual journal article should take into account whether the article has garnered significant positive citations in sources of undisputed reliability, suggesting wider acceptance in the medical literature despite any red flags suggested here.

I have removed the mention of specific publishers (redundant with a citation to the more comprehensive list anyways), and moved the description of predatory up. Also added a note on how to determine if a journal is MEDLINE indexed. Yobol (talk) 15:14, 24 July 2013 (UTC)

This doesn't address the problem. Journals don't typically fact-check (which would be a process that editors or peer reviewers may do in part but necessarily completely for an article, or that journalists do at newspapers for all facts). Journals have systems - they then rely on the systems to work, rather than themselves verifying the accuracy of everything that they publish. Not being MEDLINE-indexed does not mean "a journal article is probably not reliable". You would need some pretty strong evidence to support that claim. Citations are definitely not an indicator of quality - indeed, some of the most highly cited publications are those that are egregiously bad. Eg highly disreputable claims about vaccines for example become highly cited as egregious causes of problems. Hildabast (talk) 15:54, 24 July 2013 (UTC)

I agree "probably" may not be the right word, and have changed it to "may". Have also noted that the citations should be positive and not negative to imply reliability (probably common sense, but best to avoid ambiguity). When dealing with reliability issues, we are dealing at its core reputation (as it is based off our core policy of verifiability which discusses reputation of fact checking as of paramount importance). MEDLINE indexed journals have been through a vetting process and would seem to be a good general guide for a discussion of reliability based on reputation (with obvious exceptions on both sides). Note I also included an "out" for any of these red flags with the basis on positive citations so that articles that have been recognized by the medical community can be included. Yobol (talk) 16:20, 24 July 2013 (UTC)

This doesn't address the problem. Still using the term "predatory behavior". As written it would be really hard for a naive editor to apply the policy. NickCT (talk) 12:52, 25 July 2013 (UTC)

I'm hard pressed to think of a credible medical journal that does not subscribe to the ICMJE's Uniform Requirements for Manuscripts Submitted to Biomedical Journals, which includes a code of ethics. Are there any counter-examples? LeadSongDog come howl! 13:20, 25 July 2013 (UTC)

@NickCT:I had hoped a wikilink to our page that describes those predatory behaviors, a brief description in the next clause, and link to two different lists (one peer-reviewed) of said publishers would suffice. How would you improve upon this?

@LeadSongDog: An interesting criteria. Looking at the ICJME website here, however, it may be difficult to use as a criteria as there doesn't appear to be a comprehensive list of journals that follow it, nor does it appear that the ICJME specifically polices their own list vigorously, making this a difficult criteria to apply. (For instance, a journal could claim to follow the requirement but without policing there is no way to verify this). Do you know of a comprehensive list that is actively vetted by someone? Yobol (talk) 13:37, 25 July 2013 (UTC)

(1) just dealing with this list, journals are notorious for adopting standards that they do not consistently live up to. Anyone can say they follow this standard. Equally important, only purely or predominantly medical journals normally list themselves here. Nature & Science & PNAS & PLOS Biology do not, and they all publish articles relevant to medicine. Some journals that do list themselves would normally be considered of very low quality.

(2) more generally, predatory journals do not generally publish significant fraudulent work; anyone wanting to make an impact with such work will publish it in a journal that will be visible--as high ranking as possible, counting on the chance that the reviewers will be lazy; if they don;t succeed with one, they simply send it to another--there is no system by which a journal can know that a ms has been previously rejected. What predatory journals do is publish insignificant work, good or bad, with occasional fairly good work from authors who do not know better.

(3) Beale's list is based on his individual judgement. I greatly respect his judgment, but he has been challenged about failure to remove from his list journals that improve, or dealing adequately with those publishers who have good intentions, but nonetheless are following a model often followed by those who are not as honest. My own opinion is that some of the challenges are correct. No single judge will always be perfect.

(4) There is no foolproof test that everything published in a journal is reliable. The customary test is that people can repeat it--but since both parties may encounter the same hidden artifact, the real test is that people can build a strong structure on its basis.

(5)The best current book about biomed journals is, in my opinion, Philippa Benson and Susan Silver, What Editors Want: an Author's Guide to Scientific Journal Publishing University of Chicago Press, 2013 ISBN 9780226043142 (COI disclosure: I said so when I reviewed the book for Choice). DGG ( talk ) 16:12, 25 July 2013 (UTC)

@Yobol: Yes, Nature Med and the other giants are rather special instances, working to rather more stringent standards than required by the UCM. They have the necessary clout to demand more of their authors. At this list of links from ICMJE we find quick access to some very helpful material. The Council on Publication Ethics (COPE membership include several of the Nature Med family, mostly review journals) provides a formal mechanism for addressing cases. The Council of Science Editors' member journals include the parent Nature, plus Nature Biotechnology, Nature Genetics, and Nature Neuroscience. The CSE has its own formal approach to scientific integrity. I suppose that, in the end, the specific codification is less important than the simple fact of adopting an externally-disciplined code with third-party arbiters. LeadSongDog come howl! 18:26, 25 July 2013 (UTC)

@LeadSongDogInteresting links; though these appear to be suggestions for finding the highest quality journals, not for identifying particularly low quality journals. I wonder if we can incorporate some of this into the last paragraph already present in the Biomedical Journals section. Yobol (talk) 19:45, 25 July 2013 (UTC)

@DGG: Thank you for that additional information. Do you think adding Beall's list is too prejudicial here, or is it a reasonable starting point to look out for journals/publishers to be wary of? I specifically worded this proposal with some ambiguity knowing his list is not in any way definitive. Yobol (talk) 19:47, 25 July 2013 (UTC)

I'm not sure that creating a suitable policy that will always be easily followed by the hypothetical "naive editor" is necessarily an achievable goal. We try to dance around the point, but the truth is that there just isn't, and won't ever be, a perfect recipe that can be blindly followed to reliably determine whether or not a published source is 'reliable'. There are definitely some bright-line rules, but there are also unavoidable swathes of gray.

An individual who comes in without any familiarity with the world of academic publishing is almost always going to trip over an ugly surprise later or – more likely – sooner. What we need from this policy (rather than merely desire) is a clear framework that guides and allows experienced editors (those who come to the table with some knowledge of medical topics and academic publishing) to be able to reach reasonable conclusions about the quality of sources and how they can and should be used in Wikipedia—and to allow them to explain those conclusions to a naive-but-otherwise-competent fellow editor. TenOfAllTrades(talk) 16:45, 25 July 2013 (UTC)

What is a suitable source also depends on what other sources are available. We sometimes use less than idea sources as that is the best there are. We really just want people to use the best available source. I have been discussing issues around this Lancet article in a few other circles.[17]. The "up to 6 h" is not supported by the body of the article specifically [18]. A number of us are of the opinion that they have taken extra liberty with the English language. Additionally there are concerns that the statistical methods used were not appropriate but they used them anyway. Additionally many of the authors receive funding from the company that makes the treatment in question. Doc James (talk · contribs · email) (if I write on your page reply on mine) 18:09, 25 July 2013 (UTC)

I think we need to focus on getting a general guideline for identifying low quality journals rather than getting too deep into specific incidents. Does anyone have any specific concerns in the proposal, or ideas on how to improve it? Yobol (talk) 19:45, 25 July 2013 (UTC)

Suggest

• When looking for high-quality sources, be wary of articles from journals with a reputation for "predatory" behavior.^See: Another possible red flag is publication in a journal that is not indexed in the bibliographic database MEDLINE. An article on a topic outside a journal's normal scope may also arouse suspicion (e.g. an article dedicated to cancer treatment published in a psychiatric nursing journal). Note that such red flags may sometimes be counterbalanced by significant positive citations in reputable medical publications that suggest broader acceptance in the medical literature.
  86.161.251.139 (talk) 12:55, 26 July 2013 (UTC)

I'm neutral to your proposed change to the proposal compared to Proposal #2 above; I see merits to both versions, and would be happy with either (one is wordier, but explains the underlying policy reason for this paragraph; the other jumps to the meat of the problem). I do think that your "arouse suspicion" is a better wording, and it splits up MEDLINE indexing to its own sentence. I'm not sure discussion of "reputable" medical publications is a step forward, however, and prefer the one with "undisputed reliability" as it seems less prone to gaming. Yobol (talk) 12:01, 29 July 2013 (UTC)

I agree gaming is a key concern. To my ears at least, "undisputed reliability" seems quite a claim. Would Lancet even pass on that...? 86.161.251.139 (talk) 13:22, 29 July 2013 (UTC)

MEDLINE indexing - example query

So we're comfortable with red-flagging, say, reviews published in the Nature Clinical Practice series (eg [19]) prior to their first being indexed in MEDLINE (from 2009, in what became Nature Reviews)? 86.161.251.139 (talk) 14:14, 27 July 2013 (UTC)

I think the problem is that journals who ceased publication or changed name are not "currently" MEDLINE indexed as they are no longer published. I have added a caveat to the note in my Proposal #2 above discussing this. Yobol (talk) 12:01, 29 July 2013 (UTC)

Sorry, yes you're absolutely right. Strike query! 86.161.251.139 (talk) 13:09, 29 July 2013 (UTC)