Wikipedia talk:Identifying reliable sources (medicine)

Primary sources

I'm proposing to change this paragraph (lets reach consensus):

"ll Wikipedia articles should be based on reliable, published secondary sources. Reliable primary sources may occasionally be used with care as an adjunct to the secondary literature, but there remains potential for misuse. For that reason, edits that rely on primary sources should only describe the conclusions of the source, and should describe these findings clearly so the edit can be checked by editors with no specialist knowledge. In particular, this description should follow closely to the interpretation of the data given by the authors or by other reliable secondary sources. Primary sources should not be cited in support of a conclusion that is not clearly made by the authors or by reliable secondary sources, as defined above (see: Wikipedia:No original research). When citing primary sources, particular care must be taken to adhere to Wikipedia's undue weight policy. Secondary sources should be used to determine due weight."

To:

"ll Wikipedia articles should be based on reliable, published secondary sources. Reliable primary sources may be used with care. Edits that rely on primary sources should only describe the conclusions of the source, and should describe these findings clearly so the edit can be checked by editors with no specialist knowledge. Primary sources should not be cited in support of a conclusion that is not clearly made by the authors, (see: Wikipedia:No original research). When citing primary sources, particular care must be taken to adhere to Wikipedia's undue weight policy. "

Ryanspir (talk) 09:53, 7 April 2013 (UTC)

Strike throughs added by me for clarity. -- Brangifer (talk) 22:49, 7 April 2013 (UTC)

• Completely Disagree: it leaves too much space to cite primary sources at the discretion of editors. I believe it is a change to the worse. I am fully comfortable with what we have right now.--Garrondo (talk) 15:15, 7 April 2013 (UTC)

• Oppose There are lot of secondary sources that still need summarizing. The use of primary sources on Wikipedia generally represents an attempt to push something beyond its due weight. Doc James (talk · contribs · email) (if I write on your page reply on mine) 16:36, 7 April 2013 (UTC)

• Strong oppose. We have enough trouble trying to explain why primary studies are usually inappropriate, this will just make things worse. Yobol (talk) 17:08, 7 April 2013 (UTC)

• Strong oppose The "conclusion" of a primary research paper is nearly always a plea meaning "We've found something interesting/useful please continue to fund our research in this area / please sign the FDA documents that all allow us to sell it". They aren't neutral assessments at all. Colin°^Talk 19:19, 7 April 2013 (UTC)

• Strong oppose this actually cuts the heart out of MEDRS as a stronger statement of WP:PSTS and comes very close to violating WP:PSTS. The words "may be used as an adjunct to the secondary literature" are absolutely crucial and are far too often ignored. But thank you for crystallizing the issue with this clear statement.Jytdog (talk) 20:21, 7 April 2013 (UTC

• Oppose We rely on independent expert reviews, textbooks, and other secondary sources to evaluate and contextualise primary research. We don't do that ourselves. --Anthonyhcole (talk · contribs · email) 21:03, 7 April 2013 (UTC)

I only ever use a primary source if it has first been put into context by a reviewer with the relevant expertise in a reliable independent secondary source. --Anthonyhcole (talk · contribs · email) 17:30, 12 April 2013 (UTC)

• Oppose This section of WP:MEDRS as it currently stands is the proper application of Wikipedia sourcing policy to medical topics, the proposed change would disimprove the guideline. Zad68 22:05, 7 April 2013 (UTC)

• Strong oppose. This proposal guts the guideline of important content. It goes way too far. -- Brangifer (talk) 22:49, 7 April 2013 (UTC)
• Strong oppose. Current wording is better.Desoto10 (talk) 01:18, 8 April 2013 (UTC)

• Oppose Current wording is better. But MEDRS needs to address the occasional case where a primary source is such a whale of a study that it wags all meta-analysis down the road. Examples: Women's Health Initiative and Million Women Study. These studies have their own WP articles, so we can hardly refuse to use their primary sources, er... primarily. Secondary sources just requote the primary conclusions of whale studies. We are left in the same position as when we get data from a big study with the Hubble Space Telescope or Large Hadron Collider which isn't likely to be repeated soon. You just have to bite your tongue and quote it, noting that it's a single LARGE study that hasn't been confirmed and probably won't be, for some time. That sucks, but meanwhile what can you do? SBHarris 02:38, 8 April 2013 (UTC)

Agree that this is the exception. Doc James (talk · contribs · email) (if I write on your page reply on mine) 02:42, 8 April 2013 (UTC)

Agree. This is the type of situation the guideline allows for. Maybe it needs to be made explicit, with these as examples? -- Brangifer (talk) 03:21, 8 April 2013 (UTC)

(Sbharris) The primary-study / secondary source debate for sourcing wikipedia really has nothing to do with the power of the study. That science may want to repeat small or surprising studies to prove they weren't a one-off, or not want to repeat large conclusive studies or studies that there simply isn't enough money/interest to repeat, is a matter for our secondary sources and the academic/scientific community to decide. I don't see these as an exception to our rules. I'm sure both the studies you mentioned were covered rapidly by secondary sources of various kinds. Colin°^Talk 07:04, 8 April 2013 (UTC)

It usually does not take long before these major studies are parts of reviews. Doc James (talk · contribs · email) (if I write on your page reply on mine) 05:21, 9 April 2013 (UTC)

Of course, but I'm not sure it improves them. If there's just one big study of something, any "review" that uses it (cites it) is forced to conflate it with a lot of stuff that is only semi-confirmatory, since the extra stuff used to compare it with, won't be the same thing. For example, a big giant prospective study, the only one of its kind, might be conflated with some other really big cohort-controlled prospective epidemiological study. And that's actually exactly what happened with the two studies above, that I cited. They aren't the same-- in one study the groups are split at random, and in the other, the groups choose themselves and (of course) therefore aren't the same. So what do you do when the findings from these studies turn out the same, or when the data conflict? Which they do at many points (sometimes they agree and other times they don't). We simply end up playing off one review against another, and in that case, one would need a review of reviews, or a review of the quality of previous reviews. But suppose there's only ONE of those? How do we know it's not biased? Of course it's biased. There is no getting away from the problem that one must cut off the generalization at some level, and sometimes a lower level is appropriate is there's only one event or one big study that is relevent. In that case, the most appropriate reviewers of a set of data, are the people who generated it. Everybody else down the line is simply using data they got from somebody else and doing extra induction and comparison with similar studies, that the primary authors chose not to do.

At the most ridiculous level, we have a gigantic multicenter study that appears in the New England Journal of Medicine, peer reviewed by the best in the field, and we can't use it until somebody in the Iowa Journal of Family Practice Reviews notices the thing. Then we cite THEM. Meanwhile, why doesn't all the reviewing that contributing to getting it into the NEJM in the first place, count? It's all very arbitrary, seems to me.

Often it takes time and effort, and you must survive more scrutiny, to get some data into an exclusive high-quality primary journal, than it takes to get it they same paper into a third-tier primary one AND then reviewed by several others.SBHarris 03:10, 30 April 2013 (UTC)

We don't require a meta-analysis or some other combination with other studies. A narrative review that says little more than "This awesome study clearly proves that the foo is bar baz during full moons" is good enough for our purposes. What we're looking for is essentially an expert endorsement of someone else's work, rather than Wikipedia editors deciding which experiments were well done. WhatamIdoing (talk) 03:35, 30 April 2013 (UTC)

The whole idea of peer review is that some (bunch of) experts have to endorse a study as well-done and belieable. [Though not necessarily considered right until checked by one-or-more independent good studies elsewhere]. Anything the NEJM publishes (for example) will be picked up by secondary review journals, authomatically. And I mean automatically. It's not that some expert decides it's a great study. That's considered a given. if it even appears in the NEJM in the first place. So John Q. Editor at WP can do that as well as anybody. The secondary review journal doing that, doesn't add a thing to the credibility of the paper.

For example, today's NEJM has an article addressing a previous NEJM report of the antibiotic azithromycin causing heart deaths, based on data from Tennessee. This study had been picked up many secondary sources, including newspapers, and even caused the FDA to issue a boxed warning. Alas, none of these were any better than the first study, though they did cause the problem to make it into the Wikipedia article on the drug: [1]. Today's new study finds the supposed safety problem is not seen in a slightly different population from Denmark. So it wasn't rotten in Denmark. Now we dither, and even the FDA dithers on prescribing: to warn, or not to warn, that is the question. But no medical journals have picked up the recent finding, yet. Why should we wait till they do? They won't be doing it on the basis of anything we don't already know. They will be automatic. When they quote the new NEJM, as they inevitably will, they don't add anything by doing so. So why are we waiting for them? It's the NEJM. We don't need somebody else to read it to us. We didn't need it the first time. SBHarris 03:33, 1 May 2013 (UTC)

I disagree with your "whole idea of peer review" opinion and certainly with your view that anything the NEJM publishes can be used directly by wikipedians as though it is God's own truth. I don't think you really appreciate the WP:WEIGHT issues of new research and perhaps have got Wikipedia confused with a newspaper in terms of WP:RECENTISM. As far as Wikipedia is concerned the writers of the "Iowa Journal of Family Practice Reviews" are way ahead of Sbharris or any other Wikipedian when it comes to assessing the primary research literature. There will always be editors who think they can review the primary research literature better than the review and textbook authors we are supposed to rely on. If you feel that way, write a blog or get yourself properly published, because that isn't your role as a Wikipedian. Colin°^Talk 15:11, 1 May 2013 (UTC)

I agree with Colin above. Sbharris, there is a really important reason that both 1) the fundamental policy WP:OR and its section on primary/secondary/tertiary sources WP:PSTS, and 2) MEDRS (which is completely within PSTS), each warn very strongly away from basing content on primary sources, and that is due to the dangers of cherry-picking, SYN, and WEIGHT. As Colin said we editors need outside experts to publish a review that discusses the great NEJM study, putting it in its proper context. We need that 2ndary source to get the content in, and to defend it against people who would take the content out. While you may believe yourself to be educated/experienced enough to provide that context yourself, you could not defend the content providing the context on your own authority as an editor and editors cannot make those kind of judgements (think of the disasters that would happen if the policy were not this way, and people who thought they knew what they were talking about, but didn't, were making these kind of judgements throughout wikipedia). I realize that you find the secondary source policy frustrating and too slow and maybe too stupid, but it is a good and robust policy that is essential for Wikipedia to be the wide-open encylopedia that it is. Happily, landmark clinical studies are often accompanied by a comment letter by the editors of the journal or some thought-leader, which provides context and describes the importance of the study -- that comment letter can serve as the 2ndary source to get the Important Study into wikipedia. Jytdog (talk) 15:43, 1 May 2013 (UTC)

Letters to the editor, comments, editorials, and other opinion pieces are never peer-reviewed and are treated as primary sources on Wikipedia. They consequently aren't capable of getting the Important Study into Wikipedia. WhatamIdoing (talk) 16:25, 1 May 2013 (UTC)

That is interesting, and a good point. Thanks. Jytdog (talk) 18:22, 1 May 2013 (UTC)

I would like to point out that the devil is not as scary as he looks :). That's a Russian proverb, not sure if it makes sense in English though.

In my proposal I have indicated that:

1. "Wikipedia articles should be based on reliable, published secondary sources"

2. That only *reliable* primary sources can be used and when so is done, it should be done with care.

3. "Primary sources should not be cited in support of a conclusion that is not clearly made by the authors, (see: Wikipedia:No original research)." - I have left it as it is.

4. "When citing primary sources, particular care must be taken to adhere to Wikipedia's undue weight policy." - I have left this limitation as well in my proposal.

5. Regarding primary sources that are being adjunct to secondary sources, there is no need IMHO, as we have already mentioned that primary source should be used with care and put provided two limitations on its use. In any case, it's the consensus on every article that will be the deciding factor in case any discussion will arise if any editor has used a primary source "with care" and it will be disagreed by another editor. I repeat again, my version is more clear, shorter and more straight forward than the original and it gives just *slight ly* more air for inclusion of primary sources while including the limitations. Ryanspir (talk) 14:44, 11 April 2013 (UTC)

• Oppose - This is a move in the wrong direction. If anything, we need to tighten the loophole to ensure that primary sources are not treated as reliable unless secondary sources have first cited them in a way that shows them to be credible (i.e. the citation is not just to make a criticism, update, or retraction), significant (i.e. the secondary source uses the main findings from the primary source), and independent (i.e. no tie between the authors of the primary and of the secondary source). At that point, the utility of the primary source is simply to fill in some blanks that the secondary source omits. LeadSongDog come howl! 15:12, 11 April 2013 (UTC)

• Are you assuming that editors must be "enforced" and saying "use with care" is not adequate? Ryanspir (talk) 15:56, 11 April 2013 (UTC)
• This would simplify things enormously but it would overly constrain us when a study appears addressing a rarely-reviewed uncontroversial topic - I gather quite a bit of biology is slow to be independently reviewed. Though, in those very, very rare instances in health-related topics I'd prefer to see editors simply reporting the results and all the caveats but none of the usual evaluation, contextualisation, speculation or puffery found in the "conclusions" section, and making it clear somehow that the results have not been independently reviewed yet. ("In a recent placebo-controlled trial, blah blah, though these results are yet to be subjected to independent scholarly review" or something of the sort.) --Anthonyhcole (talk · contribs · email) 01:44, 13 April 2013 (UTC)

I agree with the position of Anthonyhcole. The example of the way how the attribution should be made in such cases can be given in WP:MEDRS. Ryanspir (talk) 13:03, 13 April 2013 (UTC)

I think the present wording of the guideline is clear enough. Just to emphasise: I've never, to my knowledge, added a health-related assertion to an article based on an un-reviewed report. And I've only seen it done appropriately a handful of times over the last five years. In most other cases the editor either doesn't understand the principle that Wikipedia simply reflects significant expert views, they disagree with that principle, or they're pushing a discredited or fringe view. --Anthonyhcole (talk · contribs · email) 03:48, 21 April 2013 (UTC)

Due to a strong consensus, may I kindly propose to remove primary sources "LA article" and "Quackwatch" on the article Medical uses of silver? Calling a substance "quack" implies that the substance is not effective for the conditions mentioned, and calling the substance "quack" in general implies that it is not effective for any conditions. Per the strong consensus in this section such determination by a primary source should have an adjunct reliable secondary source in order to comply with WP:MEDRS guidelines. Thank you. Ryanspir (talk) 09:13, 22 April 2013 (UTC)

This is so far from prior statements on this topic that it's hard to understand why you'd try this again. -- Scray (talk) 14:25, 22 April 2013 (UTC)

No. Zad68 14:30, 22 April 2013 (UTC)

No: Quack is a social term with a broader meaning that simple effectiveness. Quack is not even a medical term and proof is that search for quack in pubmed gives few results (almost all from authors with that name). In this sense it implies that there is some consensus outside the medical field to consider a treatment and its commercialization a fraud. Such consensus may be right or wrong and to indicate that we should provide secondary sources on efficacy, but if a reliable social source (such as NYT) says something is quack that clearly merits inclusion in an article since it implies that society in general does not consider the claims reliable (independently of truth). Importance is in its social aspects and that is why no secondary source is needed, since it is not a medical claim. --Garrondo (talk) 14:54, 22 April 2013 (UTC)

Ryan has a long history of making edits that conflate the substance of colloidal silver itself with the "quack" uses of it, and also of making edits confusing primary and secondary sources. Zad68 14:58, 22 April 2013 (UTC)

May I kindly bring your attention to: "Quackwatch states that colloidal silver dietary supplements have not been found safe or effective for the treatment of any condition." It clearly says "not been found effective" "for the treatment of any condition". Such statement should in my opinion have adjunct reliable secondary source. It is a medical claim, right? Ryanspir (talk) 20:27, 24 April 2013 (UTC)

Quackwatch is reporting on what the FDA has said. The FDA has stated that cs dietary supplements have not been found to be safe or effective.Desoto10 (talk) 00:28, 25 April 2013 (UTC)

FDA is not listed in WP:MEDRS as an option to which a primary source can be adjunct to. Only secondary sources are listed. Thank you. Ryanspir (talk) 01:56, 26 April 2013 (UTC)

Just a suggestion: sometimes the best way to deal with WP:IDHT is to WP:SHUN the problem. Yobol (talk) 00:54, 25 April 2013 (UTC)

I'm acting in good faith and what I'm asking is to impartially implement the strong consensus we have reached here towards primary sources. I don't think your WP:SHUN is warranted, as I'm being very reasonable and listening to everyone's opinion. Ryanspir (talk) 10:05, 26 April 2013 (UTC)

• Support proposal. I said the same thing myself before. But note that you're up against a powerful lobby here, funded via Wiki Med Inc., so this is a waste of time. Wnt (talk) 18:18, 29 April 2013 (UTC)

  Now I understand: there's a well-financed conspiracy, and all these people advocating secondary sourcing are making money from it? You're way ahead of me on this. -- Scray (talk) 00:11, 30 April 2013 (UTC)

  That's one of the daftest things I've heard in ages. Wiki Med Foundation (not Incorporated) is a charity and has next to no money. There is also no reason why the Foundation would wish the extermination of primary sources. JFW | T@lk 19:01, 17 June 2013 (UTC)

• Comment - I am fine with this part staying as it is, however I do notice that it gets misinterpreted. "In particular, this description should follow closely to the interpretation of the data given by the authors or by other reliable secondary sources." This point is covered completely in the sentence after it, and it only adds confusion. Primary sources should stay true to their own interpretation and not be dependent on another source to interpret it, that part is highly susceptible to misinterpretation. The last sentence also gets misinterpreted, "due weight".While a secondary source can critique/review/opinionate (interpret in these senses of the word) a primary source, in no way should a second source even by omission tell a primary source what it meant to say. Those two sentences are very unclear, messy, and open to misinterpretation.

I did notice the edit made by user:Wnt in the guideline article, and I agree with that part. Primary sources are important especially when they are up to date, reliable, and offer vast amounts of knowledge. Editors shouldn't be so quick to discount primary sources, when they are allowed. Strict usage of primary sources is acceptable, but it is not acceptable to discount them. Sidelight12 ^Talk 20:11, 19 May 2013 (UTC)

• Oppose. As others have pointed out, we need a policy that discourages the use of primary sources in the strongest possible terms. Where primary sources are used (on editorial judgement) they need to be supported by secondary sources as much as possible. It is most uncommon for a primary source to displace the conclusions of secondary sources, and usually you need a secondary source to support the claim that a primary source displaces the previous secondary source (yada yada). If every single health- & medicine-related editor would adhere to the simple policies on secondary sources we'd be able to avoid about 60% of all disputes. JFW | T@lk 19:01, 17 June 2013 (UTC)

Reliability of position statements from nationally or internationally recognised expert bodies

1) Sodium:

"Although the advice to restrict sodium to 1,500 milligrams a day has been enshrined in dietary guidelines, it never came from research on health outcomes, Dr. Strom said."

2) Vitamin D:

"Because of the British experience with idiopathic infantile hypercalcemia attributed to hypervitaminosis D, an inaccurate association occurred that had a profound effect on the potential of vitamin D supplementation, not only during infancy but also during pregnancy. In 1963, Black and Bonham-Carter (65) recognized that elfin facies observed in patients with severe idiopathic infantile hypercalcemia resembled the peculiar facies observed in patients with supravalvular aortic stenosis syndrome. By 1966 vitamin D was viewed by the medical community as the cause of SAS syndrome. (66,67) With the advent of molecular genetics, the children with SAS Syndrome were discovered to have Williams Syndrome, an example of unipaternal disomy, with abnormal vitamin D metabolism. (68–75)

The perception that vitamin D can inflict harm during pregnancy still lives on today as many obstetrical specialists are afraid to undertake vitamin D repletion during this period. Research efforts in this area were further hampered when in 1997, the Institute of Medicine issued guidelines that defined the adequate intake (AI) for vitamin D during pregnancy to be 200 IU/d with intakes greater than 2000 IU/d causing potential harm. (40) Recently, the IOM issued new guidelines with respect to pregnant women that define the estimated average requirement (EAR) and recommended dietary allowance (RDA) to be 400 and 600 IU/day, respectively. They also increased the tolerable upper intake limit (UL) to 4000 IU/day. (62)These new guidelines, with the exception of the UL, are based on old data since limited new data exist. The result of prior and current guidelines is that most prenatal vitamins only contain 400 IU of vitamin D. In our experience, many of today’s practicing obstetricians are unaware of the vitamin D content in prenatal vitamins or have a fear of administering additional vitamin D supplements to the pregnant women."

3) Radiation:

"In 1958, using data largely drawn from these bomb studies, scientists came up with an answer. It was called the Linear No Threshold (LNT) model and suggested all radiation, no matter how small, was dangerous.

It became the internationally recognised basis for assessing radiation risk. Yet there has always been a problem with it. The data from Hiroshima and Nagasaki were for very high levels of radiation exposure, often in the range of thousands of millisieverts. There were no significant data for lower exposures, particularly below 200 millisieverts.

The model was based on high doses and we just didn't know what was going on at lower doses of between one and 200 millisieverts," says Dr Repacholi.

Scientists simply guessed that if high-level radiation was dangerous then lower levels would also be hazardous. They made "an assumption", observes Dr Repacholi.

Chernobyl, where most people received radiation doses below 200 millisieverts, has been the first large-scale opportunity to test whether this assumption is true. The evidence from the Chernobyl Forum suggests it is not.

"Low doses of radiation are a [very] poor carcinogen," says Professor Brooks, who has spent 30 years studying the link between radiation and cancer."

"Professor Ron Chesser, of Texas Tech University, US, has spent 10 years studying animals living within the 30km exclusion zone surrounding Chernobyl.

He has found that, far from the effects of low-level radiation being carcinogenic, it appears to boost those genes that protect us against cancer.

"One of the thoughts that comes out of this is that prior exposure to low levels of radiation actually may have a beneficial effect," Professor Chesser says."

"In April, the WHO's International Agency for Research on Cancer (IARC) published a report that used the latest LNT-based radiation risk projection models to update the estimated cancer deaths from Chernobyl.

It concluded that about 16,000 people across Europe could die as a result of the accident."

Count Iblis (talk) 18:21, 14 May 2013 (UTC)

I'm sorry; I've read the above, but I'm not sure what your point is. Do you mean to illustrate that scientists and expert bodies are fallible and sometimes proven wrong with the passing of time? Or that they are sometimes forced to extrapolate and make educated guesses where evidence is lacking? Or that individual scientists sometimes disagree with the views of expert bodies? Help me out. MastCell ^Talk 18:43, 14 May 2013 (UTC)

It seems to me somewhere along the chain of primary resesearch, secondary reviews etc., all the way up to the highest authoritative expert bodies, unscientific arguments gets injected in a way that is not typical in other sciences. Of course, unlike in other sciences, in medicine expert bodies may need to make a judgement calls when the evidence is not clear, but it seems to me that such a judgement call will persist long after the science has become clear. So, in case of 2) pregnant women are still not getting the vitamin D supplementation they need, even though decades have passed since the original worry has been clarified. Count Iblis (talk) 18:59, 14 May 2013 (UTC)

Is this a suggestion to expand our content at Vitamin D? I peeked at that article and it does say that Vitamin D deficiency during pregnancy is harmful and that women often do not get enough, although it also says that supplementation is of unknown value. I do not see the source you provided in use at that article. Maybe make those edits?

Like MastCell I'm not sure what the larger point is here. Yes sometimes expert bodies get things wrong and it's hard to change entrenched positions. Still using secondary sources from expert bodies is MUCH more likely to be providing good information than using the latest individual primary sources, whatever they happen to say and of whatever quality they happen to be, or was that even the underlying point? Is there a change to WP:MEDRS being proposed? Sorry still not seeing what we're getting at here at WT:MEDRS. Zad68 19:42, 14 May 2013 (UTC)

I'm raising a point about this policy text which is written from the assumption that the highest medical expert bodies stick to science in the same was similar expert bodies in other sciences. This is not the case because these expert bodies have a dual role. They don't just review the science, they also have to come up with guidelines for doctors and patients. They need to weigh up the scientific results as published in secondary review articles based on an appropriate precautionary principle. The conclusions they reach are then not going to be conssistent with what you would get if you were to completely neutrally weigh up the scientific results. This can lead to problems that I cited above, but in general in the absense of these problems, there will still be an issue to be addressed.

A good example is vitamin D. Good review articles on toxicity are this one that considers mainly the measured effects and this one that considers it from the POV of the fundamental biological processes. But what we do in the Wiki articles is take more serious the IoM report which puts a lot of weight on a single primary reseach result which wasn't all that statistically significant, where it was found that elderly patients in some elderly care home fell more often if they were given vitamin D supplements at levels. What the IoM does here is quite obvious, in the absense of very strong evidence of health benefits (other than for bone health), you need to rule out with almost 100% certainty that there could be problems with vitamin D supplementation. So, even questionable research results hinting at problems have to be taken serious, while for evidence purporting to show benefits you would demand a very rigorous level of proof.

The problem for Wikipedia is then that with the current policy text, you can't address this issue of improper weight. E.g. in the literature you can find discussions about why these elderly people would fall more often (e.g. there is quite strong evidence that vitamin D improves neuro-muscular function, the people in non-placebo group will get out of their chairs more often and therefore be at a greater risk of falling), but these are secondary review articles and are given less weight than the IoM report, despite having a more fundamentally scientific focus on this issue. Count Iblis (talk) 14:01, 16 May 2013 (UTC)

We are not, in fact, assuming that the highest medical expert bodies stick to science. We are instead assuming that practice guidelines contain significant material about what the mainstream expert opinion is and what the actual practices are, both of which are relevant, important, and must receive significant weight to create a neutral article.

You may be confusing "neutral" with "correct". For better or worse, Wikipedia does not aim to write scientifically ideal articles. It instead aims to write encyclopedia articles that adequately and accurately reflect the current mainstream expert opinion, along with any significant minority viewpoints. If, as does happen, the experts are wrong (or divided), then it is the English Wikipedia's official policy that we, too, will be exactly as wrong (or divided) as they are. WhatamIdoing (talk) 23:42, 16 May 2013 (UTC)

Seralini Thai study on effect of glyphosate on human breast tissue, with older in vitro study from Seralini group

Note - edited the title just now as per semitransgenic's note below, with italics showing added text and crossout showing deletion.Jytdog (talk) 17:01, 16 June 2013 (UTC)

Three efforts have been made to include content in the glyphosate article based on recent primary study on the effect of glyphosate on human breast tissue from a Thai group, 'the last of which was accompanied by a 2009 in vitro study by the Seralini group Both User:Zad68 and I have reverted each time - last one is here as an example of the kind of content being created. Each time, the content has been created in the section on the Toxicity of glyphosate to humans - -indeed the only reason to include this study in the article is the effect on human health, and I believe that including it goes against MEDRS. Folks are objecting on Talk here so I wanted to get feedback on whether MEDRS applies to content generated from this source and if so, what kind of content, if any, is appropriate to generate from it. Thanks. Jytdog (talk) 22:53, 15 June 2013 (UTC) (edit note -- added the text in italics as per semitransgenic's note below. Jytdog (talk) 17:01, 16 June 2013 (UTC))

Yes would be appreciated to have more of those familiar with the application of WP:MEDRS commenting. Zad68 03:24, 16 June 2013 (UTC)

Primary sources can be included. You're right that it cannot pertain to humans (section: Avoid over-emphasizing single studies, particularly in vitro or animal studies). It says preclinical studies are invaluable for scientific research. I think it should be kept, but any mention of what it may imply for human benefit must to be deleted. Lucy346 (talk) 06:55, 16 June 2013 (UTC)

As pointed out by Zad, this source has intrinsic implications for human health- that is how it is written. Am I reading the same policy? Primary sources are strongly discouraged, and their use can only be potentially justified in very specific circumstances. This particular paper, by virtue of being in vitro is not suitable. It is very clear: the policy says, when in doubt- do not include the primary source. Lesion (talk) 08:50, 16 June 2013 (UTC)

The section "Avoid over-emphasizing single studies, particularly in vitro or animal studies" has to do with this. I don't see where it says, when in doubt, don't use the primary source. If this study is in vitro, and it assumes these results as benefits for humans, rather than potential harmful effects, then that is a problem. That part can't be used, but it doesn't necessarily make the source completely unusable. The abstract says, "potential adverse health effects," so I don't know if that makes it ok. If this stays, its dangers need to be documented too. Lucy346 (talk) 09:47, 16 June 2013 (UTC)

it's a mischaracterisation to call this a "Seralini study" in the heading, and it's intentionally misleading. But whatever, that's the kind of dishonesty we are dealing with here. Guideline does not state primary sources cannot be used, it cautions against overemphasis, it's a peer reviewed study published by a reputable journal, appropriately weighted mention, in the correct context, is acceptable. Semitransgenic _talk. 16:24, 16 June 2013 (UTC)

Good point, and thank for pointing out the conflation. It wasn't intentional, I was just moving too fast. I lumped the 2013 primary source from the Thai group with the 2009 primary source from the Seralini group into one discussion; both were included in the last attempt to add content to the article, as you can see in the dif I referenced at the beginning. Both have the same problem, in that they are primary sources and present results of in vitro studies, and as such should not be used to discuss health. Jytdog (talk) 17:01, 16 June 2013 (UTC)

When you click the bottom of "related articles" in pubmed it yields a 100 related primary studies to this one. Secondary sources provide the background to decide which content is important and which is not. To cite this specific article, and not the other 100 is by itself to give undue weight since we do not have any reason to believe that it is more important than any other one.--Garrondo (talk) 19:38, 16 June 2013 (UTC)

"To cite this specific article, and not the other 100 is by itself to give undue weight since we do not have any reason to believe that it is more important than any other one." Completely false logic, you are implying that there are 100 similar studies concerning the role of glyphosate as a potential endocrine disruptor. The relevance of the study in question (in the context of "glyphosate toxicity") is pretty clear. Semitransgenic _talk. 22:45, 16 June 2013 (UTC)

Secondary sources exist and there is no reason why we need to use primary sources. Wikipedia is not a collection of everything written. Using secondary sources helps use give proper weight. Doc James (talk · contribs · email) (if I write on your page reply on mine)

Here's an example of the text that's been disputed:

A June, 2013 article published in a peer-reviewed journal entitled: "Glyphosate Induces Human Breast Cancer Cells Growth Via Estrogen Receptors" suggests that low and environmentally relevant concentrations of glyphosate possess estrogenic activity and noted that several recent studies demonstrated glyphosate's potential adverse health effects to humans concluding that it may be an endocrine disruptor.[2][3]

Okay: The first bit is a problem per WP:MEDMOS. See all that stuff about hyping a study by claiming that it's "peer-reviewed", etc. So let's de-fluff it, in which case we get this:

One source says that low and environmentally relevant concentrations of glyphosate possess estrogenic activity. One source said that several recent studies demonstrated glyphosate's potential adverse health effects to humans concluding that it may be an endocrine disruptor.[4][5]

Now we have to process the content. Rather than adding a bunch of qualifiers like "one source that looked only at isolated cell lines, which are known to be a bit odd", let's see if we can deal with the big problems:

• For the first sentence, who really cares what just one source said? PubMed gives me more than 1500 hits on glyphosate. Why not pick one of the dozens of review articles instead of this one? This doesn't seem like such a rare opinion that we need to risk looking like we're cherry-picking.
• For the second sentence, you are presenting analytical conclusions rather than descriptions. A secondary source is required for this kind of claim. Again, these aren't exactly unusual claims in this area, so why aren't you citing a stronger source already? WhatamIdoing (talk) 23:06, 16 June 2013 (UTC)

As answer to Completely false logic, you are implying that there are 100 similar studies concerning the role of glyphosate as a potential endocrine disruptor. The relevance of the study in question (in the context of "glyphosate toxicity") is pretty clear.: says who? Several editors have already commented that they do not really see its relevancy. --Garrondo (talk) 07:58, 17 June 2013 (UTC)

Moreover, as stated above, to use it to back up results by others is highly innaproppiate, since it would be to use it as a secondary source. Similarly, the "envirommentally relevant" line is the conclussion from the authors but, by conclussions from primary articles are very commonly quite hyphenated so as to make the article more interesting". Content that would have to be included would be something like Glyphosate exerted proliferative effects only in human hormone-dependent breast cancer, T47D cells, but not in hormoneindependent breast cancer, MDA-MB231 cells, at 10-12 to 10-6 M in estrogen withdrawal condition., which is the true result from the article, and IMO absolutely undue weight. --Garrondo (talk) 07:58, 17 June 2013 (UTC)

Advice sought

I removed this section from Aloe arborescens on the grounds that it made claims not supported to the standards of WP:MEDRS, although there was some material inside the section which might possibly have been ok if carefully re-written.

Another editor has since re-written the material here, which seems to me not to make claims about effectiveness in human use, and so is ok. However, I'd like to know what colleagues here think. Peter coxhead (talk) 10:45, 16 June 2013 (UTC)

Neither source there should be used. Both are primary studies and so including them violates WP:PSTS as well as WP:MEDRS. If you search pubmed as follows [6] you get three reviews in English, two of which would be useful - one very useful for tox in cosmetic ingredients http://www.ncbi.nlm.nih.gov/pubmed/17613130 and the other which may be useful for an overview of potential dietary supplement or cosmetic ingredient uses as well as research on potential medical uses http://www.ncbi.nlm.nih.gov/pubmed/10624859 Jytdog (talk) 13:26, 16 June 2013 (UTC)

I think this is (barely) okay, since it sticks to a plain description. On stylistic grounds, I might prefer to see it condensed. But why not add the better sources yourself? WhatamIdoing (talk) 23:09, 16 June 2013 (UTC)

WP:PSTS doesn't say primary sources aren't allowed. It says be careful not to make conclusions based on primary or other sources. - Sidelight12 ^Talk 03:29, 17 June 2013 (UTC)

WhatamIdoing, you are forgetting WP:WEIGHT, which is far and away the most important reason not to use a primary research paper as a source. A five-year-old study on rats can't warrant a full paragraph of text sourced only on the primary research paper itself. Has this led to human-relevant trials or nowhere? What do secondary sources say? And the four-year-old cancer study must surely have drawn secondary commentary by now. So the question isn't whether the sources are reliable on what the article says but whether we should mention them at all, and in what detail. Whenever we mention basic (e.g. rat) research or clinical trials we imply this is relevant to the reader and has some importance to the subject. But Wikpedian's have no authority to decide which facts are relevant or important: we need to review the literature to find out. Colin°^Talk 10:25, 17 June 2013 (UTC)

Clarification

"All Wikipedia articles should be based on reliable, published secondary sources," seems to contradict "Therefore, it is vital that the biomedical information in all types of articles be based on reliable, third-party, published sources and accurately reflect current medical knowledge." so it should be changed to "All Wikipedia article topics should be based on reliable, published secondary sources." Since this is a general guideline and not a medrs guideline. The general guideline is to use secondary sources to establish notability for the article topic.

"In particular, this description should follow closely the interpretation of the data given by the authors or by other reliable secondary sources." The word "other" only muddies the meaning. if you read it slowly, and if you read it fast, the word only changes the meaning. It should be removed. The word other implies a meaning that is not there if you take the time to read it slowly.

I moved this down to the advice section from the definition section, because these aren't definitions, they are guidelines.

I added a comma and that also got reverted. Its supposed to have a comma to be grammatically correct, and it tells the reader to slow down. The meaning of it seems to change if you read it fast, as opposed to reading it slowly with or without the comma.

My edits shouldn't have been reverted, it didn't change anything, only clarified this. Leaving these edits in is promoting the mistake of misreading it. - - Sidelight12 ^Talk 03:48, 17 June 2013 (UTC) / 06:45, 17 June 2013 (UTC)

I do not understand how the first two statements are supposedly conflicting. "All Wikipedia articles should be based on reliable, published secondary sources" refers to the actual content. The use of secondary sources is not merely limited to deciding whether to have an article about it ("notability"). Most of that content should come from secondary sources, such as review articles, university-level textbooks, and high-quality reference works. WhatamIdoing (talk) 06:17, 17 June 2013 (UTC)

I mixed up third-party with tertiary sources, so I crossed that part out. - Sidelight12 ^Talk 06:41, 17 June 2013 (UTC)

I disagree that your edits should not have been reverted - it is generally unwise to change a guideline or policy without discussing it first, as they are closely watched. That aside, I don't understand what you are getting at by adding the word "topics" - can you clarify? Do you maybe mean "content" (i.e. specific passages)? Moving the passage to Basic Advice as you propose makes sense to me. Jytdog (talk) 13:09, 17 June 2013 (UTC)

It says "All Wikipedia articles," which makes it a wikipedia wide policy and not a specific medrs policy. The Wikipedia wide policy is that secondary sources be used to establish the notability of an article topic for whether the article should exist, and not the content within the article. By article topic I mean the article namespace, which is already established, not the content within the article. I don't think there is a guideline anywhere that says, all content within all wikipedia articles should be solely based on reliable, published secondary sources. Rather, all content within all Wikipedia articles should be based on reliable published sources. - Sidelight12 ^Talk 04:27, 18 June 2013 (UTC)

You are completely wrong: While notability is indeed established by secondary sorces that is not in contradiction with the fact that wikipedia articles should be based in secondary sources. MEDRS is only the application to medicine of WP:NOR, which in a section (See: WP:PSTS) says

Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources. Secondary or tertiary sources are needed to establish the topic's notability and to avoid novel interpretations of primary sources, though primary sources are permitted if used carefully. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than to an original analysis of the primary-source material by Wikipedia editors

and

Policy: Wikipedia articles usually rely on material from reliable secondary sources. Articles may make an analytic or evaluative claim only if that has been published by a reliable secondary source..

Similarly when choosing sources (See WP:RS) says

Articles should rely on secondary sources whenever possible. For example, a review article, monograph, or textbook is better than a primary research paper. When relying on primary sources, extreme caution is advised: Wikipedians should never interpret the content of primary sources for themselves.

and

Isolated studies are usually considered tentative and may change in the light of further academic research. The reliability of a single study depends on the field. Studies relating to complex and abstruse fields, such as medicine, are less definitive. Avoid undue weight when using single studies in such fields. Meta-analyses, textbooks, and scholarly review articles are preferred when available, so as to provide proper context

and

Primary sources are often difficult to use appropriately. While they can be both reliable and useful in certain situations, they must be used with caution in order to avoid original research. While specific facts may be taken from primary sources, secondary sources that present the same material are preferred. Large blocks of material based purely on primary sources should be avoided. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than original analysis of the primary-source material by Wikipedia editors

and

Ideal sources for biomedical assertions include general or systematic reviews in reliable, third-party, published sources, such as reputable medical journals, widely recognised standard textbooks written by experts in a field, or medical guidelines and position statements from nationally or internationally reputable expert bodies..

In summary: Content in Wikipedia articles should be based in secondary sources to avoid original research. In the future please discuss changes to policies and guidelines first. --Garrondo (talk) 07:12, 18 June 2013 (UTC)

Vote for this change only, removal of the word "other" from the following sentence to:

"In particular, this description should follow closely the interpretation of the data given by the authors or by other reliable secondary sources."

*Support - nominator of proposal - Sidelight12 ^Talk 11:07, 19 June 2013 (UTC)

Proposed edit to guideline

Sorry gang I am back again with the toxicity issue, this time with a specific proposal.

Briefly, IMO there is a widespread problem on Wikipedia with regard to articles that discuss toxicology -- namely, one finds content on toxicity sourced from primary studies reporting results of in vitro and animal studies of a product or active ingredient, conducted by basic researchers with no training in toxicology. Too often articles are filled with laundry lists of such content. These sections are often under section headers like "Health Effects" and in my mind, their appearance in Wikipedia violates MEDRS and PSTS and -- and they serve just to scare people. Examples are Bisphenol A, Glyphosate, Endosulfan in subsections under "Health Effects" on endocrine disruption and developmental effects, The DDT article generally handles this very well but this problem has crept into discussion of "links" to developmental problems and cancer.

Toxicology is a discipline in which scientists are trained to consider typical routes and dose of exposure to a substance, design in vitro and animal experiments to obtain useful data, and then to extrapolate those results to humans. It is special training that a typical doctor or basic biomedical researcher does not have; regulators rely on toxicologists to critically review marketing applications (and to review subsequent studies to decide whether a product should be taken off the market) and companies rely on them to decide whether to invest money in further development. Wikipedia should also rely on them for sources about toxicity, in order to ensure that we provide reliable, consensus content.

To help manage this better, I'm proposing that the following be added to MEDRS, perhaps in the "Assess evidence quality" section, as a new last paragraph. WIth respect to toxicity, that section is especially problematic because it was not written with toxicity in mind (e.g. the paragraph that starts with "The best evidence comes primarily from meta-analyses of randomized controlled trials (RCTs)..." doesn't provide guidance for dealing with toxicity, as it directed to interventions like drugs or medical devices)

I am very open to suggested changes - here it is:

"Content discussing the toxicity of substances falls within this guideline. The toxicity of drugs is studied in cells, animal models, and in RCTs during drug development, and may also emerge in post-marketing surveillance. Substances that are not intended to benefit health ("non-drug substances") will never be the subject of an RCT, since such trials would be unethical. Observational studies of the effects of non-drug substances on humans, or the presence of non-drug substances in humans, may be conducted and published after a product comes to market; these are primary studies. Toxicity studies are generally carried out in animal models or cells, and the results are extrapolated to humans; these too are primary studies. Content that discusses the toxicity of non-drug substances should be sourced from secondary or tertiary sources written by toxicologists, in which primary sources are critically reviewed for their relevance to humans under typical exposures and doses. Primary studies should not be used to support content that discusses toxicity." (note, went through and deleted all old versions Jytdog (talk) 14:20, 22 June 2013 (UTC))

Current proposal as of June 22: Include: "Content discussing the toxicity of substances falls within this guideline. Primary studies should not be used to support content that discusses toxicity." and I will generate a separate essay to Tox for everybody to consider.

Note - original post of this current proposal is at the bottom of this thread; repeated here so it easy to find. Jytdog (talk) 14:20, 22 June 2013 (UTC)

Thanks for considering this.Jytdog (talk) 12:50, 17 June 2013 (UTC)

• Support - It's not really changing the guideline at all, just specifically mentioning that there is a need for high quality secondary sources in toxicology and primary sources should be avoided. I think this is a beneficial clarification to the page per the reasons you describe above. Lesion (talk) 13:03, 17 June 2013 (UTC)

Consider "non-drug substances" instead of "not-drug substances"Lesion (talk) 13:03, 17 June 2013 (UTC)

argh, typo. I fixed it. Thanks! Jytdog (talk) 13:11, 17 June 2013 (UTC)

• Oppose per WP:CREEP. It's not clear how this would work with respect to a substance like asbestos, say. There's lots of reputable, secondary material out there and I'm not sure how you'd determine whether it passes this toxicology test or not. Different countries may have different professional standards and bodies covering such matters. Warden (talk) 14:31, 17 June 2013 (UTC)

Colonel Warden, I would love to hear your thoughts on how to manage toxicity content without more explicit language in MEDRS dealing with it.. happy to hear from you on my Talk page or yours, if you like. Or maybe you don't agree that there is a problem (if you have not considered this before, please see the Bisphenol A page and its Talk page to see what I mean) 14:58, 17 June 2013 (UTC)

• Oppose This is a major change. Jytdog proposes that he be able to reject systematic reviews that are directly about toxicity unless he agrees that the person performing the study should be described as "a toxicologist". The meat of the proposal is "should be sourced from secondary or tertiary sources written by toxicologists". This invites us to produce articles with undue attention to the views of one specialty, to ignore toxicity entirely if no True Toxicologist™ happens to have written a review about it, and to have disputes over whether each author really "counts" as a toxicologist. WhatamIdoing (talk) 14:36, 17 June 2013 (UTC)

Good point. Consider rm "written by toxicologists". Lesion (talk) 14:42, 17 June 2013 (UTC)

Done. thank you both! I am willing to strike "written by toxicologists" as long as the "critical review" language remains. See for example this source that was recently presented (right hand column, about halfway down, paragraph starting "The herbicide glyphosate..." - just cites some primary studies and doesn't comment on their design or relevance - this is the disastrous kind of secondary source that should be excluded) WAID, I am laughing about the TM, but this is not intended just for me (although I want to be able to reach for it) - it is intended for everybody. Hopefully you can accept this with the "written by toxicologists" struck out? Jytdog (talk) 14:58, 17 June 2013 (UTC)

What counts as "critically reviewed"? For example, do all systematic reviews count, or only those systematic reviews that choose primary sources according to some factor like the exposure level? WhatamIdoing (talk) 22:19, 17 June 2013 (UTC)

That is a gorgeous question. While there are advantages in policy writing to being ambiguous, my intention is that the amendment call for the critique of each primary source to be based on toxicological principles - hence, "critically reviewed for their relevance to humans under typical exposures and doses". I would primarily like the amendment (if it gets in) to disallow 2ndary or 3rdary sources that just uncritically cite primary sources; ideally the primary sources would be critiqued under tox principles. Would be happy to discuss changes to accommodate concerns that raises...Jytdog (talk) 22:44, 17 June 2013 (UTC)

follow up, WAID - if what you are looking for is "systematic review" as opposed to "critically reviewed for their relevance to humans under typical exposures and doses" I could live with that as it would exclude what I want to exclude...Jytdog (talk) 22:56, 17 June 2013 (UTC)

I don't really think you'll be happy with that outcome. Consider two papers: a mindlessly performed systematic review of garbage primaries, and a thoughtful literature review (perhaps by a team of respected toxicologists) who tell you that the primaries are all garbage and therefore nobody actually knows whether there are any significant toxicity issues. Which one do you want to be using? WhatamIdoing (talk) 11:10, 20 June 2013 (UTC)

Can a systematic review be "mindlessly performed"?  :) Anyway... based on Colin's feedback I had to move away from "systematic review" ... please see new proposal below... can you help me navigate between you and Colin? Thank you again for commenting. Jytdog (talk) 13:01, 20 June 2013 (UTC)

• Oppose The entire basis of this proposal rests on Jytog's claim that Wikipedia has a "widespread problem" with content stemming from research that has been "conducted by basic researchers with no training in toxicology." The editor claims to abhor "cognitive bias" yet believes a presumptive statement such as this justifies amending the guideline? Are we now going to willfully discriminate against interdisciplinary science? Semitransgenic _talk. 15:39, 17 June 2013 (UTC)

Hi semi, yes I do abhor cognitive bias; that same page notes that I also abhor BS. The way that the results of some experiments are blown up - sometimes by their authors, sometimes by the press, sometime by advocacy groups - to make huge claims about toxicity is sometimes BS. Sometimes attacks on those studies are BS. Sometimes not, all around. The point of the proposed language, is that it takes scientific expertise to interpret and evaluate these studies to make useful, reliable claims about toxicity. I have already accepted the striking of "by toxicologists" so critical reviews by anybody (as long as they review critically for dose/exposure) would be OK. I want to help lift Wikipedia out of the BS-y garbage heap of alarmist claims about in vitro studies and alarmist attacks on those studies - Wikipedia should provide reliable, consensus-driven content about health, not driven by RECENTISM or the most recent controversy about some chemical.Jytdog (talk) 16:13, 17 June 2013 (UTC)

Do you see what I mean, semi? Can you support this amendment, or are there any changes that would make it more acceptable to you? Thank you. Jytdog (talk) 20:10, 17 June 2013 (UTC)

unfortunately i find your additional comments above (I would primarily like the amendment (if it gets in) to disallow 2ndary or 3rdary sources that just uncritically cite primary sources) problematic, this is turning in to an agenda based creepfest. Semitransgenic _talk. 23:35, 17 June 2013 (UTC)

Thanks for replying - it is unclear to me why having only systematic reviews count as useful, is problematic to you. But if there is no way we can work to find acceptable language, then I'll just say that I appreciate your time and your responses. Thanks again.Jytdog (talk) 23:50, 17 June 2013 (UTC)

• Support the revised version (without the demand that a toxicologist review the data). Misinterpretation and panic based on toxicity data can lead to extreme things, such as the recent RCOG report in the UK essentially telling pregnant women not to buy new cars or furniture. Common sense really needs to prevail in this area, and the guideline should be seen to support common sense. JFW | T@lk 15:57, 17 June 2013 (UTC)
• Support Toxicological info as with most medical content should be based on secondary sources. Doc James (talk · contribs · email) (if I write on your page reply on mine) 16:14, 17 June 2013 (UTC)
• Support updated wording. I don't see this as a fundamental change, it simply makes more clear what the guideline already says. I've come across enough questions in this area to agree that the guideline should be clarified. Zad68 03:52, 18 June 2013 (UTC)
• Oppose As the above editor Zad68 mentions, this is not a fundamental change. It doesn't make it more clear. It is quite wordy and just adds more burden to the poor readers trying to get a grip of the sprawling landscape of Wikipedia rules and regulations. II | (t - c) 04:08, 18 June 2013 (UTC)

• Oppose most of it could be shortened to "Primary studies should not be used to support content that discusses toxicity." What if there is a primary study that says something is toxic, and secondary sources haven't caught up to it yet. It needs to be documented on Wikipedia. I think you are referring to the ethics of primary studies to imply its use for humans. Which I see no harm of ethics there. There is already a section that says preclinical tests results shouldn't be assumed for human implications.

Separately, as for ethics of a study, it should be inserted separately from this. in terms of ethics for animals, primary studies that are in situ and in vitro studies should be encouraged. I would need advice on studies that are in vivo, since the tradeoff for knowledge and for what may be unethical animal tests. Having a guideline that discourages entry from in vivo sources would encourage tests that are in situ or in vitro. I see nothing wrong with including in situ and in vitro studies that are primary. Requiring secondary sources only, does not stop this. - Sidelight12 ^Talk 04:47, 18 June 2013 (UTC)

Sidelight, if you see nothing wrong with including in vitro studies that are primary, it is no wonder you oppose this edit. Your views do not reflect the MEDRS guideline at all. Wikipedia should not have a mandate to document information in primary sources that the secondary sources have not caught up with. This is what MEDRS is all about. Lesion (talk) 08:49, 19 June 2013 (UTC)

My view is medrs related. See the section: "Avoid over-emphasizing single studies, particularly in vitro or animal studies" it says these sources are invaluable, and allows their use if used with care. medrs is made up of a collection of contributions by different people's thoughts, and it is a living document, so to say that makes no sense. As much bias as there is against primary sources, medrs doesn't prohibit them. I believe there may be a guideline or reason for a guideline to dictate primary sources, but I don't believe they should be dismissed. One is in place already, and I may understand there being something like this in place for toxicology. - Sidelight12 ^Talk 10:40, 19 June 2013 (UTC)

• Comment I think the current text (without the "written by toxicologists" part) is in alignment with MEDRS and helpful explanation. But I'm concerned that MEDRS is already too long. There's no reason we can't have a separate (short) page giving advice on toxicology. It would be classed as an Essay rather than a Guideline, but plenty good advice pages on WP are so-classed. This could be linked to from MEDRS. WP:TOXIC anyone? -- Colin°^Talk 07:39, 18 June 2013 (UTC)

Thank you for commenting. I would greatly prefer to include this in MEDRS. Would you accept a much shortened version, something like this? Jytdog (talk) 03:59, 19 June 2013 (UTC)

"Content discussing the toxicity of substances falls within this guideline, and should be sourced from secondary or tertiary systematic reviews. Primary studies should not be used to support content that discusses toxicity."

The first clause is fine and may be helpful to include. I oppose requiring systematic reviews. Or any one kind of source like "reviews". The vast majority of issues to toxicity are uncontroversial and can be sourced to any number of reliable documents. We must be careful not to write policy/guideline on the basis of solving only a few specific hard problems because then we raise the bar so high that all the easier articles can't be written at all. Colin°^Talk 09:24, 19 June 2013 (UTC)

I hear you Colin... kind of. But I believe you have stated here many times, that we need to ensure that health related content in Wikipedia reflects the actual consensus. With toxicology content, especially with the rising concern about the risks of "chemical exposure" since the 1970's, there is more and more pressure to include content that is scary and does not reflect the consensus, but instead reflects the most alarmist sides of topics where there is no consensus. Jfdwolff mentioned the recent RCOG advisory, which is here: http://www.rcog.org.uk/files/rcog-corp/5.6.13ChemicalExposures.pdf - if you look at that, it warns pregnant women away from an enormous number of things, based on pretty vaguely defined risk. At Wikipedia, we really need a higher standard for tox to avoid this kind of alarmist discussion. It is a problem and it is only getting worse. I am willing to back off systematic reviews, but would like to keep the qualifier, but only for products that are currently marketed. That way, the standard for content on those, is spilt off from tox content about products that have been withdrawn b/c their tox is well established (e.g. PCBs, asbestos). Also am willing to dump the last sentence. Current version with those modifications:

"Content discussing the toxicity of substances falls within this guideline. Content on the toxicity of substances that are still marketed should be sourced from secondary or tertiary sources in which primary sources are critically reviewed for their relevance to humans under typical routes of exposure and doses."

I disagree with dismissing primary sources altogether, but I might agree to there being a certain guideline for their use in toxicology. I think your concern is promotion of, or conflict of interest for substances. Something like primary sources must be peer reviewed, and if there is a reason for conflict of interest for this source consensus can argue against its use. Another guideline to prevent this use may be needed. I don't want that card used it used if this is not the case, just because someone doesn't like it. - Sidelight12 ^Talk 10:50, 19 June 2013 (UTC)

Using primary sources should be done rarely if ever, for health information. There is almost never a justifiable reason to use one. Jytdog (talk) 02:28, 20 June 2013 (UTC)

• Comment - The only potentially controversial aspect left here is that the edit specifies systematic reviews, whereas MEDRS for other topics requires only "reliable" secondary and tertiary sources. This wording you have compromised with after an editor above challenged the use of the phrase "critically reviewed". Also what is a tertiary systematic review? A systematic review of other systematic reviews? Not heard that term before. I think your argument is, that merely saying secondary sources is not a tight enough restriction on sources for toxicity, because the primary in vitro studies need to be interpreted by experts in the field to be of any relevance, and secondary sources which just cite these primary sources are not suitable sources? Maybe instead of using the term "systematic reviews" just use "reliable", and if need be qualify this afterwards with a rough description of what reliable means in toxicology? Lesion (talk) 09:07, 19 June 2013 (UTC)

Thanks Lesion. That is a great point, "tertiary systematic review" is goofy. I am getting stuck between Colin and WAID here, which is what I suspected might happen. Gonna try another modification above... Jytdog (talk) 02:28, 20 June 2013 (UTC)

• Work needed: I agree with your goals, but the proposal is much too wordy, containing (still) quite a few clauses which are redundant and already part of MEDRS. WP:CREEP-- please try to substantially reword so that the new text doesn't repeat what the guideline already says. Of course toxicology is part of MEDRS. Of course we should rely primarily on secondary sources. We don't need to repeat all of this. I support the proposal intent if the wording is shortened; I oppose the wording as written. SandyGeorgia (Talk) 03:06, 20 June 2013 (UTC)
  • It might be useful to write up an essay talking about how to evaluate a tox paper. It might be useful to editors who are trying to do the right thing (i.e., those not trying to win a dispute) as well as helping crystallize a few points that would be useful here. WhatamIdoing (talk) 11:10, 20 June 2013 (UTC)
• Oppose I'm concerned about the consensus which seems to be developing here, which I believe threatens to become a classic case of WP:LOCALCONSENSUS. I know it's only on the talk page, but when an editor says that "at Wikipedia, we really need a higher standard for tox to avoid this kind of alarmist discussion" and the "alarmist discussion" is a paper from the Royal College of Obstetricians and Gynaecologists, something is wrong. Wikipedia should, of course, summarize published material from such a source in an entirely neutral way regardless of what editors think about it. The purpose of MEDRS is in danger of being lost in Wikilawyering about primary/secondary/review sources. The purpose is to prevent Wikipedia being used to peddle quack remedies and make claims which may damage the health of readers if they act on them. Claims of medical efficacy and warnings of possible toxicity are not the same; much higher standards of evidence should be required for the first than the second, where the precautionary principle applies – although in all cases the strength of the evidence should be reflected in the writing. If a herbal product meets the standards of notability through its widespread use, as documented in reliable sources, and there are reviews which meet the standards of MEDRS to show some degree of efficacy plus some recent primary reports of toxicity not yet reviewed, it would be irresponsible censorship not to mention (in a properly qualified way) the possible toxicity in the article. (It might even expose Wikipedia to legal action in some jurisdictions should the toxicity be confirmed and the product cause harm to someone and it could be shown that Wikipedia editors knew of the reports of toxicity and suppressed them.) Peter coxhead (talk) 09:21, 20 June 2013 (UTC)

Peter, I strongly disagree that we should worry about "legal action". Please read the terms of use, disclaimers etc. before you make such suggestions. Irresponsible? - possibly this could be argued, fine. Illegal? - this is suggestion is a joke and does not qualify as legitimate point in this argument. No-one is going to sue me, or you any other Wikipedia editor (anonymous volunteers) if they fail to add a scientific paper about the toxicity of a substance to Wikipedia, with these terms of use etc in place. Lesion (talk) 09:33, 20 June 2013 (UTC)

Legal action or not, that is a valid ethical reason. If knowledge is available I dislike discriminating against it, when nothing is wrong with it. A vast amount of information is in primary sources that aren't in secondary sources. With good reason some primary sources may be invalid to the context of an article. What about a guideline that says sources must be peer reviewed or reliably published. Also, the source must be third party to affiliations of a substance. Third-party sources can be primary, secondary, or tertiary. - Sidelight12 ^Talk 11:17, 20 June 2013 (UTC)

Are you talking about changing the wording of this proposed edit about toxicology sources, or about changing MEDRS generally? If latter, new section with specific proposed changes delineated please. Lesion (talk) 11:57, 20 June 2013 (UTC)

It covers toxicology, but for instances I can't see now, it could cover more than that. but ok, I might propose this soon to deal with this issue. One discussion at a time seems proper. - Sidelight12 ^Talk 12:09, 20 June 2013 (UTC)

Sidelight above you write " If knowledge is available" - and this is exactly the point. MEDRS exists because we only want to state what is known (or what the medical community believes is true as stated in 2ndary and tertiary sources, which is the closest we can come) - and there is too much content in WIkipedia making claims that X or Y is toxic in A or B ways based on sources that don't take toxicology into account - heck there is too much such content based on primary studies. In o other words, too much content stated something as "known" when it is not known.Jytdog (talk) 12:58, 20 June 2013 (UTC)

Ok, forget the legal aspect (although in many countries attempts to disclaim responsibility through things like terms of use have no legal force). Please answer my main point that efficacy and toxicity aren't the same. The latter is subject to the precautionary principle. We shouldn't state on the basis of primary studies that a particular treatment is efficacious. We should state on the basis of reliable primary studies (preferably plural here) that there are preliminary reports of toxicity. Peter coxhead (talk) 19:15, 20 June 2013 (UTC)

I disagree strongly with you Peter. It is exactly because of the philosophy you are stating here that I wish to strengthen MEDRS specifically with respect to toxcity. MEDRS states "Therefore, it is vital that the biomedical information in all types of articles be based on reliable, third-party, published sources and accurately reflect current medical knowledge." This sentence (and the guideline) makes it clear Wikipedia should not include content that is "preliminary" about any health related matters, and especially not about emotionally laden preliminary conclusions that "X might be killing your baby". The goal is not to be "cutting edge" but instead to be the fat old cow walking at the end of the wagon train - we provide content on what is known - not what is speculated or preliminary. Wikipedia is not the place to go for the latest medical news. It is a place to find reliable knowledge as reflected in secondary sources. Jytdog (talk) 19:45, 20 June 2013 (UTC)

Strongly agree with Jytdog, and s/he has said it better than I could. MEDRS is not just about efficacy of treatments, but about any matter pertaining to human health. People going on about using primary sources here need to actually read MEDRS and WP:RS and realize there is encyclopedia wide policy relating to not using primary sources. Lesion (talk) 20:00, 20 June 2013 (UTC)

I have read (and regularly use) MEDRS, and I'm also very familiar with WP:RS. But editors here need to stop Wikipedia:Wikilawyering by endlessly referring to WP guideline documents, and deal with the issues: "Wikipedia policies and procedures should be interpreted with common sense to achieve the purpose of the policy..." Discuss the issues. Peter coxhead (talk) 20:20, 20 June 2013 (UTC)

Hi Peter, I tried to discuss both the letter and the spirit of policy and guideline, specifically with respect to toxicity. And I tried to directly speak to what you said about warning people. Please tell me what part of the issue you feel I have not addressed. Thanks! Jytdog (talk) 20:24, 20 June 2013 (UTC)

My comment was directed at the one above, by User:Lesion, rather than yours. But the point remains that it's not helpful to keep re-referring to MEDRS when we are discussing what should be in MEDRS. The discussion needs to focus on the purpose of the policy. So I look forward to your reply to my question below. Peter coxhead (talk) 20:34, 20 June 2013 (UTC)

• Weak support. While I agree with the implication of the current revision, it seems to be unnecessarily long-winded. How about something like "Information about the toxicity of substances (both drugs and non-drugs) should be supported by secondary sources." Axl ¤ [Talk] 10:52, 20 June 2013 (UTC)

Thank you for the support, weak as it may be! The wordiness is unfortunate - I have been trying to carve this down to the minimum that (I think) is necessary. The pressure that some editors feel to provide "fair warning" to readers about possible toxicity of currently marketed products (e.g comments by Sidelight and Peter Coxhead above) drives them to want to generate content based on conclusions drawn from primary sources (already pushing the envelope of both PSTS and MEDRS); for them (I believe), tox content is different from efficacy content and there should actually be a lower bar for adding content -- and this is exactly what has been concerning me. I don't think it is enough to just say "content about toxicity falls within MEDRS" - I think we need, within the higher standard that is MEDRS, a slightly higher standard again for content about toxicity for currently marketed products. And that takes more words. I hope that makes sense and that this was not too wordy. Jytdog (talk) 20:02, 20 June 2013 (UTC)

What is the rationale for requiring a higher standard for toxicity than efficacy? Please explain. Peter coxhead (talk) 20:20, 20 June 2013 (UTC)

Thanks for your note above. Happy to answer. I understood you to say above, that you approach tox content differently from efficacy content - that due to the precautionary principle you think that it is Wikipedia's responsibility to warn people about possible toxicities of currently marketed products based on preliminary findings described in primary sources. That is the first time I have heard the idea articulated so clearly (which I appreciate!), but in many Talk pages I have encountered similar motivation, not so clearly articulated. This approach is well-intentioned but the intention violates MEDRS and NOT/RECENTISM, and enacting it violates PSTS and MEDRS in the reliance on primary sources. I made my proposal to prevent this well-intentioned but misguided approach to tox content about currently marketed products; the higher standard serves as a counter-weight against the moral urgency some editors feel to (mis)use Wikipedia to warn the public about these possible dangers. I don't know if you agree but I hope you see the reasoning. Jytdog (talk) 20:53, 20 June 2013 (UTC)

Just re-read this and I think it is kind of dickish. Sorry about that. My intention wasn't to be so harsh, but it was to be clear. Hopefully it is clear. oy. Jytdog (talk) 21:28, 20 June 2013 (UTC)

Don't worry, I wasn't offended. Your argument for requiring higher standards for toxicity is basically to forestall what editors who disagree with you might do. Firstly, this is a poor reason: the reason should be based on a rational appraisal of the issues, not the hypothetical actions of other editors. Secondly, it's pointless, since those who disagree with you in this respect either haven't read MEDRS or aren't going to be influenced by the requirement for the same standard, let alone a higher standard.

A key issue for me is the requirement for editors to reach decisions by consensus on specific content in specific articles taking into account WP's guidelines. Trying to produce over-rigid guidelines to cover all cases (including hypothetical ones) is counter-productive (as we've seen in recent actions at the MOS generally).

My concern is not primarily with currently marketed "orthodox" medical products, and Wikipedia must never be responsible for the kind of scare which stopped parents in the UK giving their children the MMR vacine based on one individual's flawed "research". But many articles on plants which have been used in traditional herbal medicine do pose problems. Where reliable secondary sources attest the traditional and/or ethnobotanical uses of the plant, this information meets Wikipedia's requirements for notability and verifiability – verifiability in terms of the plant's cultural history and use, of course, not in terms of its efficacy. We can – and I regularly do – remove attempts to claim efficacy not supported to the standards of MEDRS. But for many readers, the mere fact that the plant has been used as a "herbal remedy" implies, even though we carefully do not say this, that it may be efficacious. Reviews of quality studies of the toxicity of many such herbal products do not exist; you surely know as well as I do that what gets reported in journals is what scientists get funded to research. For example, for plants used in ayurvedic "medicine" there are large numbers of papers and reviews in Indian journals, but very few reports of toxicity issues, even though these are known to be a problem. It cannot be acceptable to be able to say that X is used in ayurveda, and is believed by ayurvedic practitioners to be effective in the treatment of Y, but not be able to say that a primary study by Z suggests caution in the use of X as toxicity has been reported.

In summary, if it were possible to completely remove the many, many descriptions of traditional herbal "remedies" in Wikipedia, and insist that no report of traditional use can be made without evidence of efficacy or non-efficacy to MEDRS standards, then I might agree with you. But it isn't possible. So I don't agree with you. Peter coxhead (talk) 14:24, 21 June 2013 (UTC)

Hi thanks for not being offended, and for the discussion. It is a little unfair to suggest that I want to amend the guideline "basically to forestall what editors who disagree with you might do." The desire to amend definitely comes from several actual difficult conversations; what that revealed to me was that the guideline was not providing guidance sufficient to keep alarmist, and toxicologically poorly done, material about toxicity out of wikipedia. I imagine this is the same impulse that produced this guideline in the first place -- a desire to have something to help resolve disputes about what is proper to include in Wikipedia. I hear you - a lot - on the issue with traditional medicines. I have done work on articles like that too. It is a struggle to create and keep MEDRS content on them in place. However, while I share your concern, it is off topic for this request for amendment... Jytdog (talk) 22:59, 21 June 2013 (UTC)

If it were agreed that higher standards were required for toxicology reports than efficacy reports, this would (of course) also apply to articles concerned with traditional medicines. Since there are many such articles, it would potentially have a significant numerical impact. So it's not "off topic", it's very much "on topic". The way to keep alarmist, poor material out of Wikipedia is to deal with it article by article. If it's truly alarmist and methodologically poor, it can be removed for that reason, and doesn't need changes to MEDRS which may have unforeseen consequences ("you can write that herbal product X is efficacious because the sources just meet the MEDRS standards but not that it might be toxic because this needs a higher standard"). Peter coxhead (talk) 08:24, 22 June 2013 (UTC)

NOW I understand, thank you for explaining. My amendment was intended for "non-drug substances" (which has now fallen out of the amendment)... and I see what you mean that even if the "non-drug substances" were back in, traditional medicines (which are essentially dietary supplements) are "non-drug substances" too! Had not considered that and it is a great point. Especially with the language as it currently stands. Colin's suggestions about a separate essay is starting to look more sensible, to deal with all this.

Current proposal: Include: "Content discussing the toxicity of substances falls within this guideline. Primary studies should not be used to support content that discusses toxicity." and I will generate a separate essay to Tox for everybody to consider.

Note - I am going back through and deleting previous proposals so this is easier to track. Jytdog (talk) 14:20, 22 June 2013 (UTC)

I think it would be desirable for you to start with the essay. Perhaps you could use diverse classes of examples, like pesticides, prescription drugs (vaccines?), herbal treatments, and vitamins, to explain the typical concerns that editors have. Also, I hope that it will emphasize the one important area of agreement, which is that if excellent secondary sources exist, then they ought to be relied upon. WhatamIdoing (talk) 14:33, 27 June 2013 (UTC)

• Toxicity is already covered by MEDRS, and much of the material is redundant since it is already in the guideline. Maybe have a short sentence listing some of the topics covered by MEDRS instead? IRWolfie- (talk) 09:54, 22 June 2013 (UTC)
• Colin's suggestion makes eminent sense. The use of "non-drug substances" phrasing is potentially problematic in the context of CAM, at least in the US. Most herbals are normally considered as food, but garlic "for hypertension" or even "relaxing" chamomile must properly be considered a drug. Almost anything found in homeopathic materia medica is inert as used (or at most has a tiny dose of ethanol as dilutant). These things still must be considered as drugs because of associated indications, irrespective of whether there is any sign of either specific efficacy or toxicity. As Mr Bumble says, "The law is a ass", but it still is the law. Until there's an outbreak of sanity among legislators we're stuck with it, so let's ignore the drug/non-drug distinction and focus on the toxicity question. How about wp:TOXRS, aka Wikipedia:Identifying reliable sources (toxicity) as a working title for the essay? I'm thinking that we'd generally treat things such as ClinicalTrials.gov or statistical summaries from poison control centres as (low grade) secondary sources, just barely good enough to use with cautious wording. I'd expect some tension at play between avoiding alarmism on one hand and applying the precautionary principle which is so dear to environmentalists on the other. LeadSongDog come howl! 15:18, 27 June 2013 (UTC)
• The problem you appear to be having is reviews you don't like rather than editors citing primary sources directly. Also, people aren't going to change their votes based on the revised proposal, so this is dead unless you start a new section. As a general comment, as WhatamIdoing has articulated quite clearly above, I don't think you'll have much luck accomplishing what you want through editing policy pages. Personally, I don't think policies should be edited to fit specific disputes (see #8 in the cynic's guide). Your introduction and later comments revealed that you believe "we need, within the higher standard that is MEDRS, a slightly higher standard again for content about toxicity for currently marketed products". Your proposal included restrictions upon using reviews based upon the author (a "true" toxicologist), the type of review ("systematic"?) and how much analysis the review gives to the sources it cites. These look very susceptible to gaming. To get into the weeds of one example you cited (Bisphenol A), you're probably not going to be able to remove the perspective of the Chapel Hill scientists such as vom Saal just because the reviews discuss a lot of animal studies and regulators from the FDA (altho not necessarily their scientific advisors) and EU don't really agree. II | (t - c) 08:45, 29 June 2013 (UTC)

Hi ImperfectlyInformed - you are pretty much missing my point. Chapel Hill is not part of my problem with the BPA article and other articles. The main problem is the train wreck of primary sources which under PSTS and MEDRS should not be there at all. There are also some really terrible 2ndary sources cited for tox content that don't review the primary sources they cite using basic tox principles (I linked to some above relevant to BPA and glyphosate), but I have already given up the idea that I can change this guideline to exclude them, as you can clearly see in the current proposal - I have accepted that making that change would affect too many established articles. My issue with some reviews is not that I "don't like" them - by which I think you mean their conclusions - but rather that some secondary sources that make tox claims, fail to use toxicological standards and methods when citing primary sources, making their claims unreliable. The issue is quality of the review through which conclusions are arrived at, not the content of the conclusions themselves. Reliable toxicology says what it says. Which is sometimes grey, not black and white, which is fine, as long as it is actual toxicology and not just hand waving. As to your broader comments - as I have written here a bunch of times; toxicology is a discipline (an important one) within the biomedical sciences and there are way too many "armchair toxicologists" out there. I already have adapted the proposal to the several criticisms so I am not sure why you are rehashing old stuff. In any case, am looking to finish the draft essay this weekend and hope to post it Sunday evening. Will be fun. Jytdog (talk) 09:24, 29 June 2013 (UTC)

The topic of low-quality reviews has come up before - see for example a discussion from 2008 called Dated and low-quality reviews. Back then I was (not very coherently) raising the question of low-quality reviews, and I was also concerned about obscuring primary sources behind the uncritical parroting of a narrative review. There's no simple solution to that problem. Reviewing Bisphenol A, it looks to me like your approach was more deletion rather rewriting, and you also lacked persistence. You have to keep hammering away and incorporating suggestions to produce a consensus. Ultimately I think most people would be happy to see that particular article trimmed down to summary style. By the way, as you probably know there are plenty of scientists who think the Chapel Hill "consensus" is alarmist, so it's a little hard for me to understand where you would draw the line. I mean, if you've got the FDA and EU regulators disagreeing with a source, how is that source OK but the source which you personally (but perhaps no sources) disagree with as "fail[ing] to use toxicological standards and methods" not OK? (I'll admit the disagreeing sides here make it complicated.) This gets into the thorny issue of editors making authoritative judgments on which reviews are valid. I am of course not opposed to editorial research, and I've been involved in cases where, based on editorial consensus, reviews were not used because they were low-quality. However, rewriting the guideline ultimately doesn't change anything: you have no way to prove that you're right, so it comes down to editorial consensus. II | (t - c) 10:09, 29 June 2013 (UTC)

Interesting, nice to see that you struggled with this too. MEDRS itself puts boundaries on the kinds of sources that can be used, and in the Assess section, calls for editors to look at the source and consider what it is before using it. For interventions, meta reviews of RCTs are best, "Systematic reviews of bodies of literature of overall good quality and consistency" are next, etc etc. I was looking to add something to MEDRS along those lines for tox, where it would be, for tox, "reviews of bodies of literature of overall good quality and consistency that take into account route of exposure and typical doses to which humans are exposed". It is not a big leap. Your line of critique about what I "personally" view as as X or Y when no other source calls it that, a) runs up against MEDRS's call for editors to assess, and b) is a red herring - as you well know it is very rare for one review to critique another directly. And again I am not out to drive content in one direction or another. I am trying to get rid of tox content based on sources that don't address basic tox principles. Toxicologists can disagree with one another; it is not for me or Wikipedia to sort that out but rather just to describe the mainstream viewpoints. On endocrine disruptors, my understanding is that mainstream toxicology has consensus that it is a good thing that DES was taken off the market and that there is a risk that other chemicals on the market (and chemicals endogenous to various foods) ~may~ be doing similar harmful things in a more subtle way, but I think that is as far as the broad consensus goes. From there you go to individual chemicals and you find ranges of tox opinion about various risks of harm. The science appears to me, to be somewhat at the end of its rope right now, until more, and carefully done, research is completed. I am very heartened that there is a concerted effort going to coordinate research among academics, the NIH, and regulators to get definitive answers to questions raised about BPA by the Chapel Hill Group, the big NTP review, and the decisions/report by the FDA based on both reports http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3548284/ see especially this ("The consortium represents an unprecedented approach to conducting GLP-compliant research by bringing researchers and regulators together during the planning stage to ensure that results will be maximally useful for risk assessment and regulatory decision making. The grantees and FDA representatives, along with coordinators from the NIEHS/NTP, held their first in-person meeting in March 2012. This collaboration is expected to produce a robust and valuable body of work on the effects of BPA in rats, a key animal model in toxicity testing.") So no, I don't think the Chapel Hill report is outside the consensus and I have never stated a desire to remove it. Jytdog (talk) 15:21, 29 June 2013 (UTC)

DARE guidance?

I'm wondering whether the DARE Database of Abstracts of Reviews of Effects [7][8] could somehow fit into MEDRS as a useful tool to gain unbiased guidance on the reliability of a given systematic review. For many recent reviews on their database [9] DARE provides brief critical appraisals (box here [10]) of the source, with nuanced guidance on the reliability of the authors' conclusions (also accessible via TRIP). For example:

• Questions about the quality of included data and methods of synthesis mean that the authors' conclusions should be treated with caution. [11]
• In general, the authors’ conclusions appear to be appropriate, although they depended on the model assumptions. [12]

As an individual contributor, I find guidance such as this useful for nuancing the weight to give to the findings of particular systematic reviews. 86.161.251.139 (talk) 10:13, 11 July 2013 (UTC)

Agree - less than half (and reducing) have those appraisals, but they're great when they're there. There are several things to know about it - not least of which is to be careful if not going to DARE itself, how often it is updated where you're using it, because there are big differences, and how to request an appraisal. I'm building up a file with quite a few things like this missing from the systematic reviews page.Hildabast (talk) 02:16, 12 July 2013 (UTC)

Reverted edit

I have concerns about a statement in the "Biomedical Journals" section:

• It is usually best to use reviews and meta-analyses where possible, as these give a balanced and general perspective of a topic—and are usually easier to understand.

While agreeing that that narrative reviews tend to be relatively reader friendly, the same can scarcely be said of meta-analyses. For example, here is a recent open-access report I randomly selected on PubMed: [13] [14]. I doubt many general users of Wikipedia would find that easy reading. Also, I don't think that meta-analyses (a particular type of systematic review) can be said to provide a "general perspective of a topic". For methodological reasons, they have to address a rather specific research question. As noted elsewhere in the section, "whereas a narrative review may give a panorama of current knowledge on a particular topic, a systematic review tends to have a narrower focus."

I don't see the sentence adds anything useful to the paragraph and I suggest removing it. 86.161.251.139 (talk) 15:06, 11 July 2013 (UTC)

Disagree... the particular meta-analysis you selected would be readable to an editor familiar with the work in the field, I'd think. And it's a particularly "deep" example, in my opinion. Here's one that I think is more typical: [15], conclusion is "On the basis of the current research, any clinical benefit of oral glucosamine for patients with chronic LBP and radiographic changes of spinal OA can neither be demonstrated nor excluded based on insufficient data and the low quality of existing studies." - that's very accessible language to a typical editor. Don't forget that we're talking about recommended sources for editors to use. The job of the editor is to take the source and represent it accurately and accessibly to the end-reader. I find meta-analyses are useful, they're usually a dozen pages of methodology followed by one paragraph of conclusion, and it's just that conclusion we're looking for, as editors. Zad68 15:28, 11 July 2013 (UTC)

Maybe you've misunderstood my point... I've absolutely nothing against encouraging contributors to use meta-analyses. Far from it! And I fully take your point that meta-analysis conclusions have to be clearly defined. However, saying that meta-analyses, with their pages full of methodological technicalities, are "easier [than what?] to understand", seems—as written—a pretty strange claim to make. Since any meta-analysis worth its salt addresses a sharply defined PICO research question, it's hardly the ideal port of call for a "general perspective of a topic"; as pointed out a few lines below, that's the role of a good narrative review.
Fwiw, the paper you cite as being more representative [16] is a systematic review which does not include any statistical meta-analysis. Here's a randomly generated sample of five more free-access reports of clinical meta-analyses recently indexed by the NLM: PMID 23639488, PMID 23637132, PMID 23587198, PMID 23558282, PMID 23558164. 86.161.251.139 (talk) 17:41, 11 July 2013 (UTC)

Ok so if the issue isn't that meta-analyses should not be recommended, but that it's incorrect to call meta-analyses "easier to understand", we can get somewhere with this. The edit that was reverted made this change: "It is usually best to use reviews and meta-analyses where possible, as these give a balanced and general perspective of a topic—and are usually easier to understand." I think the issue was that the edit removed the characterization of reviews as "easier to understand" and it looks like we all agree reviews are indeed easier to understand (although the guideline doesn't say what is it that they're easier than). I agree that an individual meta-analysis does not often provide a general, easy-to-understand overview of a topic, and they're not really designed to. They may have a paragraph backgrounder and then jump right into the data. Suggested change is:

It is usually best to use reviews and meta-analyses where possible, as these . Reviews in particular give a balanced and general perspective of a topic—and are usually easier to understand.

Comments? Support/oppose? Zad68 17:59, 11 July 2013 (UTC)

No need I can see for an RFC-type discussion with Support/oppose etc... Personally, I think your suggestion is an improvement. Though I would again question whether systematic reviews give a "general perspective of a topic" (they also have to address tightly defined PICO questions), or indeed whether they are necessarily "easier to understand" (they too are tied to strict reporting of their methodology).
In general, I feel that this part of the page could be better explained. (For instance, it would make sense to me to start a section on 'Biomedical journals' with "Peer reviewed medical journals are a natural choice as a source for up-to-date medical information in Wikipedia articles" rather than with "As mentioned above...".) Aside:I'd like to say that I'm a strong supporter of the guiding principles of MEDRS and my reverted edit was only intended as a small correction. I did the edit after making a somewhat more substantial proposal, #DARE guidance?, which I really do feel may be worthy of some consideration (eg to avoid WP:GAME?). 86.161.251.139 (talk) 18:33, 11 July 2013 (UTC)

86.161.251.139 you really seem to know your stuff and I hope you stick around and keep contributing to WP:MEDICINE-scope things. I see your IP User Talk page and won't bother with the "Welcome to Wikipedia, consider registering" song and dance. I feel like I might have run into you already. Be bold, make needed updates. The worst that happens is it gets reverted and we discuss it a bit. I'll put the small tweak to your change into MEDRS and see if anybody reverts it. Feel free to make more suggestions and changes. Zad68 18:41, 11 July 2013 (UTC)

What wording improvement can we make to make it clear we are saying reviews (PubMed Publication Type = "Review") and not systematic reviews? Zad68 18:49, 11 July 2013 (UTC)

"Literature reviews". But I believe that the PubMed search you name gives both types of review articles: literature reviews and systematic ones. WhatamIdoing (talk) 23:50, 11 July 2013 (UTC)

Regarding the terminology, the distinction generally drawn I think—though Hildabast may correct me—is between "narrative reviews" and "systematic reviews" (both of which are, strictly speaking, "reviews of the literature"). Regarding our wikilinks, I guess the narrative reviews redirect makes the distinction clear through the dab header. 86.161.251.139 (talk) 09:33, 12 July 2013 (UTC)

Well, you're right, but there's a bit more to it ;). That is the distinction usually drawn - but narrative is also used for non-quantitative (so if I can't meta-analyse, I present results narratively). It's not a satisfactory way to describe non-systematic reviews. It's on my mental list to start working on the relevant WP pages, because there's a lot that needs to be done there - having that more complete/accurate may help people decide what changes to make here. One of the biggest problems here at the moment is the guidance on searching: what's described here isn't the best way to search for reviews, because relying on publication types restricts you to MEDLINE-indexed reviews. That's problematic for two reasons: MEDLINE-indexing only indexes MEDLINE journals. Consequently, articles in purely PMC journals and Bookshelf/PubMed Health are missing. And secondly, MEDLINE-indexing takes months - as of a few days ago, for example, the most recent Cochrane reviews MEDLINE-indexed were from February. Combine those two problems, and for a search term like "asthma", you could be missing up to a 1000 reviews, the vast majority of them recent and/or full text. I'm working on an additional option at the moment, and will let you know when we get it organised. For systematic reviews, the option is PubMed Health. We include DARE, Cochrane, so it's non-MEDLINE/PubMed, and we're the ones gradually pulling in all the health technology assessment systematic reviews into the system (that are neither in DARE nor MEDLINE). PubMed Health updates DARE every week (some other places only do it once a month or once a quarter), and we'll be doing it more frequently than weekly sometime soon. Right now, your best bet is to search PubMed Health for systematic reviews, because everything is a systematic review (and there is no publication type systematic review anyway). Currently, when I look for reviews in PubMed, so that I also have a chance of seeing more non-MEDLINE-indexed ones, I do this: AND (review [pt] OR review [tw]).Hildabast (talk) 12:04, 12 July 2013 (UTC)

Sorry - didn't explain what I mean by health technology ones: I mean systematic reviews done by health technology assessment agencies. We digitise these and put them into the PubMed system as well as PubMed Health. Included are things like the NICE clinical guidelines that include full text complete systematic reviews, systematic reviews from agencies like AHRQ, the VA, IQWiG. Some of these also publish articles in MEDLINE journals, but most don't (even though they are peer-reviewed). So they meet our quality review, but not via the MEDLINE process as they are not journals.Hildabast (talk) 12:10, 12 July 2013 (UTC)

Arbitrary break

While Zad68 was making a detailed edit, I was trying to make a more general (though perhaps hasty) one, along the lines I was trying to suggest above. For now, I've posted it as a bold edit [17] for BRD, if necessary. It will probably need some tidying anyway! 86.161.251.139 (talk) 19:21, 11 July 2013 (UTC)

I agree with the anonymous IP editor. I am happy with the recommended changes. Axl ¤ [Talk] 23:30, 11 July 2013 (UTC)

A few points - MEDLINE publication type review isn't just systematic and non-systematic reviews of literature, and it doesn't include meta-analyses if they are not reviews. I don't agree with the statement that non-systematic reviews are broader: or because of their systematic methodology that they lack contextual information. Go to a literally 1,000 AHRQ review, that includes key questions on most aspects of care, or one like lactose intolerance. Being systematic doesn't preclude these things - and being non-systematic doesn't ensure their presence. There are publication type reviews that are only letters, for example (and quite a few!).Hildabast (talk) 02:28, 12 July 2013 (UTC)

Aren't all (published) meta-analyses also reviews? Axl ¤ [Talk] 10:03, 12 July 2013 (UTC)

A meta-analysis of, say, two major trials may not involve any review (systematic or otherwise). 86.161.251.139 (talk) 10:20, 12 July 2013 (UTC)

I think I understand where Hildabast is coming from. And I agree it's complex...
A premise: As I understand it at least, unlike professional reporting guidelines [18], MEDRS has to serve in practice as both 1) an editorial tool to help ensure unbiased sourcing of medical content in keeping with Wikipedia's principles and culture, and 2) a user-friendly guide to best medical sourcing practice for the entire range of good-faith contributors. Not an easy combination, especially given the key importance of ensuring that the guideline does not lay itself open to abuse. And all this while keeping it as short and sweet as possible... What I think we're cautiously trying to do together here is to enhance the guideline without breaking something that already works pretty well.

In reply to at least a couple of the points Hildabast raises:

• "I don't agree with the statement that non-systematic reviews are broader." I've now tweaked the wording slightly [19]. IMO, we don't want to discourage in any way appropriate sourcing of up-to-date general encyclopedic information—as distinct from identification/sourcing of specific current best evidence—from recent high-quality, open-access narrative reviews and authoritative textbooks (e.g. Robbins, Harrison's et al). While this distinction may ultimately be a fuzzy one, I think there's general consensus on the project to encourage a judicious mix of high-quality journal and book sources, including open-access narrative reviews which general users can readily reference for more discursive information on a topic.
• AHRQ reviews. I wholeheartedly agree about the usefulness of AHRQ, NICE and other reliable bodies that systematically evaluate best evidence, and I'd consider this type of source to be our gold-standard "ideal source" for clinical evidence, when available and applicable. Maybe this could be expressed a bit more explicitly? (I also have a cautious idea that the focus of the #Assess evidence quality section could perhaps be enhanced by somehow incorporating AHRRQ, NICE, DARE, etc.)
86.161.251.139 (talk) 12:48, 12 July 2013 (UTC)