Wikipedia talk:Identifying reliable sources (medicine)

Add image to WP:MEDASSESS

Wonder what peoples feeling are about adding this type of illustration to the MEDASSESS section. It shows some different ways to balance evidence. I've recently had the feeling that many new editors are having trouble understanding why we use one article over the other, and this would perhaps make it easier to explain why we chose secondary over primary sources.

There are different ways to rank levels of evidence in medicine–they are similar in that they put high level reviews and practice guidelines at the top.

Please give your thoughts, and I can also prepare more of these images for different pyramids as well. Best, -- CFCF 🍌 (email) 09:20, 3 September 2015 (UTC)

very clear and informative, I think MEDRS would benefit from this/ posted talk/Med--Ozzie10aaaa (talk) 20:09, 3 September 2015 (UTC)

• Interesting. The pyramids differ on some important points about ranking sourcing. Let's use both, say where they are from, explain that conflicting advice is the norm, and also look for other differing views. Something that stands out to me is the idea that "clinical practice guidelines" are more authoritative than systematic reviews. Obviously these become outdated as compared to the latest systemic reviews, and sometimes they are not even evidence based, but I really like that one of the pyramids shows this as the highest authority because in many ways, CPGs are. Thanks for making more than one.

I wonder if we could have one that is "WikiProject Medicine's Ranking of Priority", and dictate to the world how information ought to be evaluated. We use different terms and emphasis. Blue Rasberry (talk) 18:04, 4 September 2015 (UTC)

Our "ranking" depends upon multiple factors. The "pyramid" would look a lot like the table at WP:CANVAS: several unrelated criteria, and being "strong" on Criteria #1 might not outweigh being "weak" on Criteria #2. WhatamIdoing (talk) 18:43, 4 September 2015 (UTC)

I added references in the captions to the images, but I don't really know how to best explain their differences and rationales. I also like the pyramid at [1] which makes a distinction between meta-analysis and systematic reviews. Unfortunately it has too many steps, so I'd have to make a new image for those.

As response to Bluerasberry – CPGs should ideally be at the top, but that isn't always that case. WhatamIdoing – what different criteria do you propose for such a table. I surmise you mean something like this:

Higher quality
	↕		↕		↕		↕
Lower quality

-- CFCF 🍌 (email) 23:52, 5 September 2015 (UTC)

Sorry for messing up on the formatting on adding a comment to this section, but I noticed that Medical textbooks are missing from the pyramid. I use texts all the time. Best Regards,

  Bfpage |leave a message  00:06, 23 September 2015 (UTC)

Yes, CFCF, that's exactly the table I have in mind. A comprehensive list of criteria is too long to be useful, but some of the major ones are primary vs secondary, newer vs older, independent vs conflicted, and better evidence vs weaker evidence. WhatamIdoing (talk) 04:14, 23 September 2015 (UTC)

References

Request for Comment: Country of Origin

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Should we change MEDRS, which currently reads:

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

to

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, country of origin, or conclusions.

This proposal is to address only the addition of high-quality sources into the guideline. LesVegas (talk) 23:33, 5 September 2015 (UTC)

Comments

Support Wikipedia should never get into the business of discriminating against high quality sources because of where that research might have been conducted or published. However, this has happened before, where multiple Cochrane reviews were rejected while trying to replace an old source with them, partly on the basis that "all but one of its authors were Chinese," and therefore, they must be biased. This is not an isolated incident either, with some editors rejecting all sources published in China not on the basis of case by case analysis of the source's quality, but stating they're published in China and assume they must all be unreliable. Low quality sources that, for instance, are not peer-reviewed, should always be rejected. But MEDRS does not yet make it clear whether or not high-quality foreign sources should ever be rejected on the basis of country of origin, whether it be the authors of a publication or place where high quality research is published. And interestingly, MEDRS currently allows for industry funded research not to be rejected on the basis that it's industry-funded, while it says nothing about a particular country's research. And yet, despite whatever potential problems could exist, it is very clear that industry funded research can be very problematic to add into Wikipedia. Just recently, Coca Cola came under fire for funding scientific research showing that Coca Cola doesn’t cause obesity. And bad as that may be, MEDRS currently doesn’t allow us to prohibit such research. It states:

'Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.'

There are also many well-documented problems with industry-funded research in the way of psychiatric medications failure to warn about cancer risk with certain drugs, as well as showing a huge statistical increase in "positive findings" for various pharmaceuticals.

However, just because there is a documented problem with some industry funded research, that doesn’t mean there is a problem with ‘’’all’’’ industry funded research. And oftentimes, new drugs only have industry-funded research for sources. No other sources can be found. Reasons like that are why it's not permitted to reject high quality sources on the basis of them being industry-funded.

At the same time, some editors currently reject some sources on the basis of them being published in another country. One such example is with acupuncture where a 1998 research report showed that in Hong Kong, Taiwan, Russia/USSR, China and Japan higher reports of positive findings occurred when compared to England. In some ways findings like these could possibly mirror the problems in industry-funded research. And at the same time, as with industry-funded research, it’s difficult to discount all of it on the basis that there is a chance some of the more positive findings might possibly be due to publication bias. Further complicating the issue is that more positive findings might be due to cultural differences in scientific research between various countries, such as ethical dilemmas with using placebos where the Chinese have shown statistical deviations from non-Chinese trials, creating conditions where placebo alone is not ethically justified as a control. There are many other possible confounding factors similar to this which may explain differences for findings between various countries once found within this specific modality. Undoubtedly, this seems to cloud the issue. But making the issue more problematic, for us as editors, is that some of the research mentioned in the Vickers source might not even be of high quality, thus causing making any argument against high quality research moot.

Adding "country of origin" seems to put the wording more in line with Wikipedia’s project Countering Systemic Bias, as well as our WP:BIAS policy. By ensuring Wikipedia editor-level peer review doesn’t extend to country of origin, it also makes it consistent with our current stance on liberal allowance of sources no matter what types of funding may be received.

UPDATE For anyone who thinks this issue isn't happening at all or was just one isolated incident, let me show proof it is still happening. Within the past few days, an editor rejected research in a high quality journal, Medicine, because the "authors are Chinese." Like it or not, rejecting sources based on ethnic origin of authors is a real problem. We need to examine sources, rejecting or accepting them based on their own merits and nothing else. LesVegas (talk) 20:01, 8 September 2015 (UTC)

LesVegas (talk) 23:51, 5 September 2015 (UTC)

Oppose I see no reason why this proposal is needed. We could also add that we shouldn't oppose inclusion based on authors skin color–because racism is bad, but it isn't needed. Level and quality of evidence is far more important. -- CFCF 🍌 (email) 00:56, 6 September 2015 (UTC)

There have been incidents where high quality sources were rejected for reasons that could've been prevented with amended language, such as this. I modified my first comment, adding in one example of such a rejection, although there have been others. Therefore, it appears it is needed. Let me know if you still think this way after reading the link. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Question What does "country of origin" in the proposal refer to? The country where the study was conducted, nationality of researchers or subjects, or country where the publishers of the journal or book are located? The first two are fine but perhaps unnecessary as CFCF says above (can LesVegas point to instances where such criterion was cited and prevailed?) If the last, I'll be opposed to the change since assessing the quality of journal and its publishers is crucial for assessing medical and other sources, and neither high-quality nor sham publishers are distributed uniformly across the globe, and we on wikipedia cannot solve this real world problem by simply turning a blind eye to it. Abecedare (talk) 01:04, 6 September 2015 (UTC)

Good question. "Country of origin" is all-encompassing, meaning country of author, country where research was conducted and where it was published. Keep in mind, this RfC is asking about high-quality peer-reviewed publications that meet every other standard of quality in MEDRS. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Comment This appears to be an attempt to allow Chinese studies about acupuncture published in Chinese journals to be used to support acupuncture. We need to use high quality journals with a reputation for reliability. So oppose the underlying attempt. Doc James (talk · contribs · email) 01:27, 6 September 2015 (UTC)

Actually, there was an incident where an influential editor opposed the addition of a Cochrane Review because its authors were from China. DocJames, please limit your comments to high quality sources. This is what the discussion is pertaining. LesVegas (talk) 01:52, 6 September 2015 (UTC)

To which Cochrane review do you refer? Doc James (talk · contribs · email) 04:05, 6 September 2015 (UTC)

Actually, several. One of them was a review on fibromyalgia which had several authors that were Chinese. High quality sources should never be rejected on such a basis, don't you think? By the way, upon re reading your comment, are you suggesting Chinese studies or Chinese journals are all inherently low-quality? I certainly hope that's not what you were saying. Tell me I read that wrong. LesVegas (talk) 04:24, 6 September 2015 (UTC)

Yes that Cochrane review is a fine source to use. No I am not saying all Chinese journals are inherently low-quality. Many journals in many places however are low quality. Doc James (talk · contribs · email) 04:29, 6 September 2015 (UTC)

• Agree with Doc James. I have seen zero examples of any attempt to reject high-quality foreign sources on the basis of simply being from another country. The Chinese studies about acupuncture published in Chinese journals were rejected not because they were published in China, but rather that they were not high quality journals with a reputation for reliability. There are plenty of fringe journals pushing pseudoscience in the US and UK, and plenty of good, peer-reviewed science gets published in China. --Guy Macon (talk) 01:48, 6 September 2015 (UTC)

Guy Macon , you might check out the diff below where an editor rejected one such source on this very basis. You can't get much higher quality than Cochrane Reviews, and you can't get much lower editing than to reject one all because "all but one of its authors are Chinese." It's sad we have seatbelt laws and equally as sad we have to tell editors not to reject high quality sources because of things like where they're published, but I'm afraid that's the case. It's exhausting to waste time on such silly matters and doesn't hurt anything to add it in because we're talking about high quality, peer reviewed sources here anyway. LesVegas (talk) 02:34, 6 September 2015 (UTC)

I did check out the diff. As others have pointed out, it does not say what you claim it says. --Guy Macon (talk) 05:39, 6 September 2015 (UTC)

Guy Macon, I've seen you around and know you're a good editor here. I know it's shocking to think another editor would ever argue such a thing, but I was in the middle of that one and I can tell you that's what was meant regarding a set of Cochrane Reviews used to replace old research. Kww argued it shouldn't be used to replace an outdated review per WP:MEDDATE for several reasons, one being that the Cochrane Reviews used Chinese authors. If you would like more details on that, I'm happy to provide them. Again, I know it's hard to believe editors would ever say or argue such a thing, but sadly they do. The other Guy constantly rejects sources because they're Chinese, and you can see for yourself how he stereotypes them as universally unreliable. No regard for peer review in statements like that, just outright rejection of sources because they are Chinese. We shouldn't even have to discuss such matters as ethnicity of authors or place of publication on talk pages, it's distracting and disruptive; editors need to be focusing on quality of individual journals and that's it. LesVegas (talk) 06:09, 6 September 2015 (UTC)

• Comment Pending further info Is this change meant to address a non-hypothetical problem? Excluding a study based solely on country of origin is so strange that I'd be surprised to see that anyone had attempted it, let alone done it successfully, but has it happened? What were the circumstances? When I first saw this RfC, I thought "This looks like it's probably about something else" and DocJames' scenario seems consistent with this. Other comment: What counts as "personal"? Darkfrog24 (talk) 02:07, 6 September 2015 (UTC)

Darkfrog24, you might want to check a diff out where an editor opposed updating an old source with a series of Cochrane Reviews because all but one of its authors were Chinese. Yes, rejection of high quality sources based on country of publication or origin of its authors is a real problem. As long as sources are high quality, issues like country of origin or ethnicity shouldn't matter. LesVegas (talk) 02:28, 6 September 2015 (UTC)

Well this diff doesn't show someone saying, "Exclude these sources because they're Chinese." It's more like Kiwi saying, "I think this guy wants to exclude the sources because they're Chinese," but whatever. I'm changing my comment to mild support. The idea that we shouldn't exclude sources solely on country of origin is so obvious that we all should be doing it already and if a source really is unreliable, there should be other reasons to exclude it. Darkfrog24 (talk) 03:39, 6 September 2015 (UTC)

+support based on the diff linked above, where an editor simply assumed that a study was biased because other studies by other authors of the same nationality had been accused of bias. This is out of line. DES ^(talk) 03:17, 6 September 2015 (UTC)

Darkfrog24 and DES what you see operating in the diff is WP:REDFLAG. The key word in the diff is "suspect"; which is very different from "excluded". Jytdog (talk) 13:50, 6 September 2015 (UTC)

The key phrase in the diff is "I also note that of the supposed refutations, all but one include Chinese researchers on the papers: we've discussed that bias problem extensively, and there's no reason to believe it doesn't apply here" This is drawing a conclusion that the mere presence of Chinese researchers equates to bias. This is not acceptable. DES ^(talk) 13:57, 6 September 2015 (UTC)

DES, thanks for replying. LesVegas and others have been pushing and pushing and pushing to include Chinese sources on acupuncture in our article - the issue has been discussed a zillion times but keeps coming up, and it gets frustrating, and people write things quickly that are easy to misconstrue when diffs are cherry-picked, as LesVegas has done with KWW's statement. Jytdog (talk) 14:07, 6 September 2015 (UTC)

User:Jytdog, would a rule saying, "do not exclude based solely on country of origin" make it harder for you to exclude low-quality or otherwise problematic sources? I'm not familiar with this particular dispute, but it sounds like those sources are no good regardless, so this shouldn't make any difference. Would it help if another line were added elsewhere saying, "Sources that have not survived peer review may be excluded" or "sources from publications known to routine publish and fail to retract unreliable material may be excluded"? Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

The background facts are:

a1) our article on Acupuncture has been highly contested for a long time and is subject to DS per Arbcom - the core dispute is between science-based editors and believers in acupuncture (as with many alt-med topics);

a2) some acupuncture believers actually try at some level to deal with the relevant Wikipedia policies and guidelines (NPOV and particularly PSCI, FRINGE, MEDRS);

b1) Research (and reviews) produced in China about acupuncture (and other TCM) overwhelmingly come to positive conclusions about the efficacy of TCM for various diseases and conditions, and they come to positive conclusions way more often than research and reviews produced in the West.

b2) This discrepancy is explained by believers (in published sources) in various ways (e.g Westerners don't do TCM properly or there are special facts about TCM that make it impossible to conduct sound scientific research on it); scientists have explained this discrepancy in published sources by pointing to the immature level of development of Chinese scientific institutions.

c1) When you put a) and b) together, you get some acu-proponents pushing to use reviews originating from Chinese scientific institutions to support content in our article about the benefits of acupuncture (content that is not supportable with, or even contradicted by, other sources), and science-based editors tending to reject that content and its sourcing - especially when no other sources are brought (see the first bullet of WP:REDFLAG). This has been discussed extensively on this Talk page as well as on the article Talk page and the consensus has been to treat sources about TCM originating from China as "suspect" for the time being - not excluded, but not "green flagged" as high quality. This RfC was posted by an acu-proponent. You can see my response to it below. Jytdog (talk) 15:05, 6 September 2015 (UTC)

Jytdog, what is your opinion on rejecting Cochrane Reviews because its authors are Chinese? Are there no publications in China that are reliable? Do you have a source that all are unreliable and engage in publication bias? Do you think there could be other possibilities for differing results, such as ethical dilemmas with placebo controls in more instances? And since scientists have documented the same issues you complain about in Chinese research, as with industry-funded research, why do you hold a different standard for Chinese sources? LesVegas (talk) 15:22, 6 September 2015 (UTC)

• oppose The framing of serious issues with Chinese research publications on acupuncture as "discrimination" is the worst kind of red-herring, bad-faith, manipulative argument imaginable, especially when Chinese scientists themselves point out the problems. This RfC is the pits. There are solid sources describing the problems:
  • Li J, et al The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012. BMC Complement Altern Med. 2014 Sep 26;14:362. PMID 25256890
  • Vickers, Andrew (April 1, 1998), Do certain countries produce only positive results? A systematic review of controlled trials., Control Clin Trials {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Ferguson, Cat (November 26, 2014), Publishing: The peer-review scam, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Qiu, Jane (January 12, 2010), Publish or perish in China, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)

The quality will surely improve with time but LesVegas' effort to wave a magic wand and make the actual problems disappear under the banner of "discrimination" runs hard against everything we try to do at WP:MED with regard to using high quality sources. Jytdog (talk) 04:37, 6 September 2015 (UTC)

Jytdog, this RfC is about high-quality sources, not low quality ones which are clearly not allowed. But your inability to differentiate between these two categories and continue stereotyping all Chinese studies and Chinese authors the way you just did isn't surprising since you posted that scientific institutions in China aren't independent or mature, and therefore all of their journals are unreliable. Frankly, comments like that are why RfC's like this exist. But back to the discussion at hand. You never addressed high quality sources, which is what this RfC is pertaining to. You are opposing low quality ones, which we all should oppose on the basis of things like lack of peer-review and things of that nature. But opposing them simply because they are Chinese isn't acceptable, don't you think? LesVegas (talk) 04:57, 6 September 2015 (UTC)

You continue to mischaracterize the situation and to employ circular reasoning. The systemic problems with Chinese research on acupuncture are well documented and so the assumption going in, is that they are not high quality; per Guy's diff they are "suspect." The burden is on the person bringing any such source to show that the specific source is high quality. Per WP:REDFLAG one way that could be done would be to show that sources that are high quality say the same thing. This is not racism, it is dealing with immature level of development of Chinese scientific institutions, just like it is not racist to say that the institutions of Chinese law in the fields of environment and intellectual property are not mature. The air quality in Chinese cities is often poor; copyright infringement is rampant in China; Chinese publications on acupuncture almost always have favorable outcomes. Those are simple statements of fact, reflecting a society in development. They are not essentialist, racist statements - they are existential statements of fact and will change over time, as the situation in China changes. So WP:DROPTHESTICK. And I suggest you be much more careful in deploying this tactic of cherry-picking diffs and trying to discredit editors as racist. Jytdog (talk) 13:50, 6 September 2015 (UTC)

Jytdog, just the other day you said "we do not perform peer review on sources and should never reject industry-funded sources that almost always have positive findings. But Chinese research? Nope, Chinese scientists aren't independent or mature. I'm not trying to characterize you as racist, but frankly by defending one standard for industry-funded studies, and enforcing another for Chinese research while stereotyping all Chinese research as unreliable, never to be evaluated on a case-by-case basis, makes it difficult for me not to find a diff that characterizes you any other way. But let me allow you the opportunity to prove that you're not racist. Simple question: why should we allow one standard for industry funded research despite documented publication bias issues, and another standard for Chinese research, rejecting the notion that we should evaluate Chinese sources or Chinese authors on a case-by-case basis like we do everywhere else? LesVegas (talk) 15:01, 6 September 2015 (UTC)

Your mischaracterization of what I wrote and of MEDRS is malicious, incompetent, or both. This is not about conducting peer review, which involves critiquing the quality and extent of the experiments that were done, the presentation of the resulting data, and the conclusions drawn from the data. This has nothing to do with research funding. And I said nothing about "Chinese scientists" - I addressed institutions. Jytdog (talk) 15:14, 6 September 2015 (UTC) (strike; should not have written that. my apologies. Jytdog (talk) 15:48, 6 September 2015 (UTC))

Oh but it is about conducting a peer review based on industry funding. Those have the same documented issues you complain about in Chinese research, although I would argue, worse, since there are more confounding factors between Eastern and Western cultures (like different scientific ethics) and those can explain reasons for discrepancies in results between East vs. West, not simply publication bias. Simple question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? LesVegas (talk) 15:33, 6 September 2015 (UTC)

You do not seem to understand what happens during peer review of scientific papers; I explained that above. Jytdog (talk) 15:48, 6 September 2015 (UTC)

Jytdog, this is not about peer review in the publication process. I hope you aren't WP: IDHT'ing Yes, scientific papers are peer reviewed before being published in reputable journals. But the peer review we are talking about is that "Wikipedia editors should not perform peer review", specifically, in this instance, by rejecting high-quality sources on the basis of industry funding behind piece of published research. Again, please answer the question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? I really hope you can answer it. LesVegas (talk) 16:09, 6 September 2015 (UTC)

We already do evaluate on a case-by-case basis, and since this hasn't been a problem we don't need a clause against it in our guidelines. In order that they be followed–guidelines need to be succinct. We can't indiscriminately add clauses in order to address hypothetical problems or noone will read them–making it far harder to police.-- CFCF 🍌 (email) 05:44, 7 September 2015 (UTC)

On the face of it, the OP appears to want to get his low quality pro-acupuncture sources into our Acupuncture article by claiming they are high quality. I don't think the community is quite that silly. -Roxy the dog™ (Resonate) 10:28, 6 September 2015 (UTC)

Previously uninvolved editor here: 1) Even if the sources were rightly excluded, "they're Chinese" should not be numbered among the reasons for this; their own failings should be enough. Question: Why would "don't exclude based on country" make any difference? 2) Aside from this acupuncture issue, is the argument of exclusion-based-on-country a common enough problem to merit explicitly banning it? Do the words "don't exclude based on country" earn their keep for the space they take up (in Wikipedia's already Byzantine rule structure). Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

"They're Chinese" is a poor shorthand for a complex discussion and is not how the issue should be discussed nor should that phrase be used as a description of the stance of those opposed to this RfC or who treat sources about TCM produced by Chinese institutions as suspect. I am unaware of this issue arising outside the context of TCM. Jytdog (talk) 15:19, 6 September 2015 (UTC)

I can be more succinct than JD. 1)No. 2)No (to my knowledge). 3)No. -Roxy the dog™ (Resonate) 15:38, 6 September 2015 (UTC)

Oppose. While I agree that certain fringe journals are focused in particular countries, I don't think we should be determining reliability overall by country of origin. However, that can be a first red flag on certain topics for a deeper look. That's why I would oppose this language as I don't think it is needed. If a question on reliability comes up, there will be other qualities we look at for journals as well. There is also potential this language could be abused from a WP:BEANS approach (acupuncture does come to mind), so I think it's better that this is one thing left unsaid, but consensus in discussions on the idea that country or origin alone can't be used can always be linked if someone brings it up. Kingofaces43 (talk) 16:35, 6 September 2015 (UTC)

Strong oppose The focus should remain on the quality of the research and the reliability of the source. Either the research is good, or it is not. Either the source is reliable, or it is not. In what possible way does "country of origin" factor into anything, except to imply that some countries are doing poor quality research and publishing in unreliable sources, and must therefore be held to a lesser standard? TechBear | Talk | Contributions 20:39, 6 September 2015 (UTC)

I agree with you that the focus should always be on the quality of the sources themselves. In MEDRS it states, "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions". We aren't supposed to reject sources purely on those merits, just simply focus on the quality of the source itself, and yet MEDRS has to be explicit because I can only assume in the past editors have rejected high quality sources because of things like references, inclusion criteria, etc. Given that editors have rejected clearly high quality sources on the basis of nationality of author or country of publication (as seen from the diffs I provided) why haven't we reached a point where we need to explicitly tell editors they need to be focusing only on source quality and nothing else, like country? LesVegas (talk) 22:22, 6 September 2015 (UTC)

Support I have read most of this RFC and debated commenting. Most of the responses do not answer the question. The question is specifically about High quality sources. High quality sources should never be excluded based on the country of origin or nationality of the authors. Low quality sources will be rejected based on the fact they are low quality and this change will not allow them in regardless. I have looked at the diffs and there is a problem in some instances, though how wide spread is hard to say. AlbinoFerret 15:59, 7 September 2015 (UTC)

A blog post by Edzard Ernst, a leading academic in this area, reads in large part:

"In this case, you might perhaps believe Chinese researchers. In [this systematic review http://online.liebertpub.com/doi/pdfplus/10.1089/acm.2014.5346.abstract], all randomized controlled trials (RCTs) of acupuncture published in Chinese journals were identified by a team of Chinese scientists. A total of 840 RCTs were found, including 727 RCTs comparing acupuncture with conventional treatment, 51 RCTs with no treatment controls, and 62 RCTs with sham-acupuncture controls. Among theses 840 RCTs, 838 studies (99.8%) reported positive results from primary outcomes and two trials (0.2%) reported negative results. The percentages of RCTs concealment of the information on withdraws or sample size calculations were 43.7%, 5.9%, 4.9%, 9.9%, and 1.7% respectively.

The authors concluded that publication bias might be major issue in RCTs on acupuncture published in Chinese journals reported, which is related to high risk of bias. We suggest that all trials should be prospectively registered in international trial registry in future.

I applaud the authors’ courageous efforts to conduct this analysis, but I do not agree with their conclusion. The question why all Chinese acupuncture trials are positive has puzzled me since many years, and I have quizzed numerous Chinese colleagues why this might be so. The answer I received was uniformly that it would be very offensive for Chinese researchers to conceive a study that does not confirm the views held by their peers. In other words, acupuncture research in China is conducted to confirm the prior assumption that this treatment is effective. It seems obvious that this is an abuse of science which must cause confusion.

Whatever the reasons for the phenomenon, and we can only speculate about them, the fact has been independently confirmed several times and is now quite undeniable: acupuncture trials from China – and these constitute the majority of the evidence-base in this area – cannot be trusted."

This gives us a convenient way of identifying a large section of the literature as pseudoscience sensu stricto, and not to be regarded as RS. Richard Keatinge (talk) 16:17, 7 September 2015 (UTC)

Richard, I am glad you are giving this more analysis than others have. I have asked myself the same question- why have variations been noted? While publication bias is one possibility, there are others. For instance, placebo controls are rarely done and for ethical reasons control groups receive actual treatment of some sort. Another possibility is that in China they are performing acupuncture differently. The first time I ever had acupuncture was when I was in China, and after an accident, and I can tell you it was a very different treatment than any of the acupuncture I have had since returning to the US. It is also very different from Japanese acupuncture, or Korean, all of which I have experienced. The Chinese acupuncture I experienced used thicker needles and they adjusted them in such a way that elicited very strong stimulation. It was also done in a hospital, by medical doctors, so I knew in the back of my mind this was legitimate and this gave a different experience overall. The thing is, there are many variables. But worst case scenario, statistics have shown a 24 percent increase in positive findings of Chinese research versus Western research regarding acupuncture. That's still lower than the stats on industry-funded drug trials when compared to independently-funded sources, and yet we have explicit language in MEDRS that we shouldn't reject high quality sources because of funding. So why shouldn't we be consistent? LesVegas (talk) 18:26, 7 September 2015 (UTC)

• oppose it is an attempt to run around the essential purpose of MEDRES: that we only use the highest quality sources. where there is evidence that medical sources from a particular country do not measure up to the standards generally required we should NOT be using them. -- TRPoD aka The Red Pen of Doom 17:05, 7 September 2015 (UTC)
• Support: Adding country of origin to the policy/guideline is appropriate; nothing prevents anyone from applying a rigorous critique to the quality of a particular source. It is important not to mass-categorize entire groups of authors or publications simply by where they originate. It, for example, an academic journal from nation Foo publishes bad science, it can be assessed on its own merits or lack thereof, not the ethnicity of those producing it. Montanabw^(talk) 23:41, 12 September 2015 (UTC)
• Support. Inclusion should not be arbitrated on account of the author's, nor the content's cultural origin. The opposition makes claims that the guideline amendment is unnecessary, as such is already generally accepted, however I do believe in a differentiation between the two. We need a reference-able guideline to control cultural prejudice, which serves no place on Wikipedia. There should be no question of interpretation, the guidelines should be definitive. ExParte ^{talk | contribs} 06:18, 13 September 2015 (UTC)
• Oppose quote The framing of serious issues with Chinese research publications on acupuncture as "discrimination" is the worst kind of red-herring, bad-faith, manipulative argument imaginable, especially when Chinese scientists themselves point out the problems. This RfC is the pits. There is an awful lot of bad faith characterisation in this thread. Of course country, nationality etc. should not in itself be a criteria, but if the problem really did exist (which I'm not convinced of by the instances given), this wording would do nothing to solve it but simply distract from the central question of the quality of the source in a POINTY manner. Pincrete (talk) 19:04, 19 September 2015 (UTC)

There were several instances, one where Cochrane Reviews were being suppressed because of Chinese authorship as well as high quality journals such as Medicine where the same objections based on Chinese authorship were made. Since this RfC was on high quality sources only, can you please clarify: when we have obviously high quality sources, do you think country of origin should be an issue at all? LesVegas (talk) 22:50, 19 September 2015 (UTC)

Re country of origin question. Obviously not in itself. I don't ordinarily edit in Med. areas, (summoned by bot), however it is simply an established fact that some sources are less reliable than others in all areas (US Govt. 'collateral damage' figures?). I don't see why 'country' would be identified any more than any other factor. If there is real evidence of rejecting sources solely, or principally, on grounds of ethnicity or provenance, this is not the way to solve it. I was not persuaded that there was such evidence. You (and others) are repeatedly saying these are high quality sources, AS A FACT, but is that not what is disputed? Pincrete (talk) 14:20, 21 September 2015 (UTC)

I have never seen anyone dispute, for instance, that Cochrane Reviews or publications from Medicine, are not high quality publications unless they have Chinese authors in the studies they publish (which has been disputed). In fact, MEDRS says they unequivocally are high quality sources. One aspect that has been disputed is that if they have authors who are Chinese, then that makes them unreliable. My position is that country of origin shouldn't ever be used as an argument to exclude high quality sources. What we instead should be focusing on as editors is if the source passes MEDRS's established high-barometer for reliability and that's it, never rejecting sources because of things like country of origin. Do you agree? LesVegas (talk) 17:14, 21 September 2015 (UTC)

I was invited by a bot to make a comment, I am not either competent or willing to discuss the reliability of specific research. The comment I made is that the examples given do not persuade me that research is being rejected solely or primarily on the grounds of ethnicity or nationality, and that further, if it were, this would not be an effective means of remedying the problem. The provenance of any source, and its reputation for checking, is always a factor in assessing its reliability, whether we are discussing Russia Today, Fox News, Daily Mail, or a Govt. statement. I am in-expert on the protocols of medical trials, but even I know that allowing for 'placebo effect' is a cornerstone of such trials, yet you dismiss this factor above as 'Chinese doctors have an ethical objection'. Fine, then such trials have not been conducted according to long-established medical standards. IF the proof of 'racism' or irrational 'national prejudice' were as clear-cut as you appear to believe it to be, you should be taking this matter to a much higher 'court' in WP than this RfC. I am not persuaded that you are correct. Pincrete (talk) 07:47, 22 September 2015 (UTC)

• Oppose There's no noted problem with this occurring. The objections to Chinese sources for acupuncture and Indian sources for ayurveda is due to noted and repeated bias problems, not due to some imaginary racist motivation.—Kww(talk) 16:05, 23 September 2015 (UTC)

Actually, one of the noted problems was with you, when you deemed a series of Cochrane Reviews suspect because they had Chinese authors. I just wanted to correct the record here. I'll also note you continue to defend this behavior even today. Nobody is saying there's racist motivations on your or anyone's part for doing this, btw, just a wrongful assumption where you believe that because some studies have at one time shown possible (not proven) bias, all are therefore unreliable, even extending to what we consider the highest quality sources like Cochrane. LesVegas (talk) 20:17, 23 September 2015 (UTC)

You confuse "LesVegas is concerned about an issue" with "there actually is an issue".—Kww(talk) 20:57, 23 September 2015 (UTC)

• Oppose. Where there is a proven bias, as in the case of Chinese studies of acupuncture, we should just ignore them. (Off topic, but I also believe we should treat industry-funded reviews with a high degree of skepticism, and should ignore them when good, independently funded reviews are available.) --Anthonyhcole (talk · contribs · email) 02:05, 26 September 2015 (UTC)
• Oppose It's unnecessary to add that kind of language. We accept a source if it's published in reputable journals regardless of their origin. This proposal will only work for POV pushers as an avenue to weasel their not so reliable sources into articles, claiming regional bias. Darwinian Ape ^talk 08:50, 30 September 2015 (UTC)

Call for close

This thread should be closed as POV pushing and trolling. When Jytdog wrote "When the institutions that support science in China become independent and mature, their journals will become valuable sources. They are not there yet"[2], only to have LesVegas characterize it as " Chinese scientists aren't independent or mature"[3] -- something Jytdog never wrote or implied -- it became clearly evident that LesVegas is more interested in winning the argument than seeking the truth. Demonizing Jytdog as a racist is trolling, and we should close down this discussion rather than rewarding such behavior with further attention. --Guy Macon (talk) 20:00, 6 September 2015 (UTC)

Oppose closeI find it a little odd that nobody seems to want to answer why we have one standard for industry funded sources (written into MEDRS), and an entire other standard for Chinese sources where we can reject them on the basis of being Chinese and not on their own merits. Nobody has dared answer that question, and here, less than 24 hrs after the RfC was filed, an attempt has been made to close it before anyone answers this obvious and frankly embarrasing inconsistency. How paradoxical to then accuse me of not trying to seek the truth. And please don't accuse me of demonizing Jytdog as racist. He accused me of accusing him of racism, which is not the same thing at all. He said I was finding diffs that made him look racist and I told him essentially that was his problem, not mine. As for institutions, are they faceless, humanless entities or are they made up of human beings? So I suppose the institutions can be immature or not independent, but the scientists and publishers composing the institutions are mature and independent? Really? Especially when Jytdog and other editors are on record rejecting sources because they're Chinese, and not because of, say, lack of peer-review or lack of Medline indexing. If you read Jytdog's many diffs in full context he makes it clear it is impossible for anything Chinese to ever be considered reliable. I have not once seen him state, even here, otherwise. And at the same time he defends the practice of not rejecting sources because of industry-funding that has been documented to skew results, but opposes all Chinese research on acupuncture for those same reasons. I have asked multiple times for an answer, and he ignores it. Please, somebody, can somebody answer it? If we can come to a consensus on how to reconcile that glaring issue, and we can come to a consensus that high quality sources shouldn't be rejected on the basis of ethnicity of author, nor should they be rejected purely because of where they're published, I'm fine with that. When editors reject friggin Cochrane Reviews because authors are Chinese, we have a serious problem. LesVegas (talk) 22:11, 6 September 2015 (UTC)

I'm reminded of MastCell's WP:CGTW#8: Anyone who edits policy pages to favor their position in a specific dispute has no business editing policy pages. Corollary: these are the only people who edit policy pages. Yobol (talk) 22:16, 6 September 2015 (UTC)

• Support close This RfC is an obvious attempt to win a content dispute by declaring that editors objecting to certain sources are racist. Johnuniq (talk) 23:45, 6 September 2015 (UTC)

Oppose close - I do not think it is appropriate that everyone is focusing on the editor instead of answering the basic question this thread is asking - some of those opposed to the policy edit proposed asked some version of "why would country of origin matter?" Well, exactly, it shouldn't matter. And yet sources are rejected because "they're Chinese" and that is wrong. If there were reliable ways of gauging the quality of the research, each review can evaluated on its own merit, as it should be. The fact that the authors are Chinese shouldn't even come up, but it frequently does. So, we have a problem.Herbxue (talk) 14:12, 7 September 2015 (UTC)

Note to closer: Herbxue also has argued for using sources stemming from Chinese institutions in acupuncture articles and claimed discrimination with regard to applying REDFLAG to them Jytdog (talk) 14:50, 7 September 2015 (UTC)

Actually read that link again: I say "fine" to rejecting Chinese journals that do not have a reputation for quality, what I call "outrageous" is rejecting a review in a mainstream journal because the authors names are Chinese, and yes that actually occurred, and thus we have a true problem.Herbxue (talk) 15:25, 7 September 2015 (UTC)

You continue to conflate systematic problems with Chinese science on acupuncture with racism. This is not OK. Please provide me with any diff where you express an acknowledgement of the problem (there is not one in that diff btw - you acknowledge occasional problems, not institutional ones) I very much agree that the issue is not about individual Chinese scientists. Jytdog (talk) 15:59, 7 September 2015 (UTC)

Focusing on individual editors is taking the focus away from the debate. I have acknowledged problems I saw with TCM research firsthand in China, but I do not generalize them, and I do not believe there is adequate evidence to claim there is a "systematic" problem with "Chinese research". For me the issue here is whether Chinese researchers are inherently unreliable, and I don't believe WP editors can say that they are without evidence showing individual journals or educational institutions to be unreliable. If a particular review appears to be of low quality, then that in itself is reason not to use it. But if it appears to have solid methodology, I don't think it should be rejected on the basis of being "Chinese". Herbxue (talk) 21:20, 7 September 2015 (UTC)

• Support close Despite what our resident acupuncturist says, we should not use unreliable sources, (which is what they are) wherever they come from. -Roxy the dog™ (Resonate) 14:28, 7 September 2015 (UTC)

• This might have been an interesting RfC and it asks some very real questions in terms of sources. I would have been interested in pursuing the ideas presented here from all sides however, once again, discussion is shut down with name calling and personal accusations so that mature discussion is thrown out.(Littleolive oil (talk) 14:35, 7 September 2015 (UTC))

Note to closer: Littleolive oil also has argued for using sources stemming from Chinese institutions in acupuncture articles and per this agrees with Herbxue who commented above Jytdog (talk) 14:50, 7 September 2015 (UTC)

Thank you for including a link which belies what you mean to be an accusation and attack and links to my cmts which you paraphrase inaccurately.(Littleolive oil (talk) 15:17, 7 September 2015 (UTC))

• • I am becoming increasingly concerned with Jytdog's constant attacks and his inaccurate representations of editors and what they are saying and doing.(Littleolive oil (talk) 15:28, 7 September 2015 (UTC))

Naming you as an alt-med advocate is not a personal attack; it is a description that is easily supportable with diffs. And as I noted here, you are the one making blatant misreprentations here, as did LesVegas as noted in post opening this section. I don't know what you think this drama-mongering gains for you. Jytdog (talk) 16:36, 7 September 2015 (UTC)

Just so you know, being an alt-med advocate and being an editor who wants to make sure alt-med is treated neutrally and fairly on Wikipedia are two completely different things. LesVegas (talk) 18:08, 7 September 2015 (UTC)

Jytdog. No diif or diffs support overarching, sweeping generalizations regarding other people. Making false statements about people is a personal attack on those people and what they are and stand for. Wikipedia is not the real world, a place in which people are multifaceted and carry on their lives in complex ways. Do you realize that the use of exercise and the studies on its impact on health are relatively new. Is this alternative medicine and if so how many sensible people in the world today support this alternative medicine. If I use antibiotics but support exercise does this make me a supporter of alternative medicine. You are constantly making statements which attach motive and meaning to people and their actions. What do I get out of this. Nothing expect that I am tired of seeing false narratives created by editors which in the end are used and applied to sanction. It is possible to disagree with people and to even be aware that our own perspectives are based on our own point of view. Discussion which does not attack but attempts to understand and compromise can go along way towards creating good articles. You have once again made statements about me which are false with no proof whatsoever. You are creating a false narrative about another editor. You know nothing about what I support and do not support. This the third time. (Littleolive oil (talk) 17:25, 7 September 2015 (UTC))

Oppose close RFC's should run 30 days, this one is only 2 days old. Regardless if some are reading into the question something it doesnt say, it needs to run its course. AlbinoFerret 16:04, 7 September 2015 (UTC)

Support close, this RfC is going nowhere and is sufficiently ill-framed that it stands minimal chance of producing anything useful. Richard Keatinge (talk) 16:19, 7 September 2015 (UTC)

• Oppose close: RfC was simple, and straightforward concept; IDHT problem of editors reading more into it than exists notwithstanding. Montanabw^(talk) 23:41, 12 September 2015 (UTC)

• Oppose Close. This proposition brings up a valid point. Country of origin should not be, in general, a criteria for inclusion of a medicinal article. It doesn't matter what the purpose of presentation was, this should be a part of the guidelines. What's the issue with additional guidelines for inclusion arbitration? ExParte ^{talk | contribs} 05:34, 13 September 2015 (UTC)

• Country of origin should not be a blanket basis for condemnation across all possible subjects, but it might be for specific countries in specific subjects. It would certainly be the case for the subject and country being discussed be on the evidence a reason for at least the greatest skepticism. The prosposed modification is too great. The cases will need to be discussed individually. DGG ( talk ) 20:11, 13 September 2015 (UTC)
• Support close … This RfC is an obvious attempt to win a content dispute by declaring that editors objecting to certain sources are racist. and , this RfC is going nowhere and is sufficiently ill-framed that it stands minimal chance of producing anything useful. This seems like a lot of airing of bad feeling on the part of those who want different standards to apply to Alt Meds. Pincrete (talk) 19:12, 19 September 2015 (UTC)

This problem is still occurring

I have seen comments suggesting editors here never rejected sources because their authors are Chinese, despite diffs, so allow me to provide one more. Just yesterday, an editor did just that, rejecting a high-quality source because it had Chinese authors, so clearly this is a serious and ongoing problem that must be dealt with. This editor rejected research in a high quality journal, Medicine, because the "authors are Chinese," and therefore assumed to be incapable of not being biased Like it or not, rejecting sources based on ethnic origin of authors is a real problem. Sources in low quality journals should be rejected based on their own lack of merits. Sources in high quality journals should be accepted on their own merits. But sources should never be excluded for embarrasing reasons like this and it is a shame that in 2015 we have to have to write explicit language into our guidelines to keep behavior like this from happening. LesVegas (talk) 20:18, 8 September 2015 (UTC)

I think you're grossly oversimplifying a significant issue. It has been recognized for some time that research in certain countries tends to be uniformly positive about acupuncture, to the extent that it raises serious questions about publication bias. This line of thought is not—as you try to portray it—a form of racism among Wikipedia contributors. The relevance of national origin of research has been expounded both qualitatively and quantitatively in the reputable scholarly literature, going as far back as Vickers et al., 1998, who found that trials from China, Japan, Hong Kong, and Taiwan were uniformly positive about acupuncture and urged caution in integrating these almost-certainly-biased results into systematic reviews or meta-analyses. More recently, a systematic review conducted by Chinese authors in 2014 found striking evidence of publication bias in studies on acupuncture reported in Chinese journals. I think it's worth having a serious discussion about this, but your post is pretty much the opposite of serious discussion. MastCell ^Talk 17:56, 21 September 2015 (UTC)

MastCell, I'm all for having a serious discussion on this and hope you are too. For the record, I never claimed racism was a factor here, and my rather long-winded complex posts, if you read them, should show that I haven't oversimplified anything. In fact, we have written into MEDRS not to deny sources based on how they're funded, and yet we have extensive documentation of publication bias regarding industry funded research. No such bias has ever been proven with Chinese research, merely speculated as one of many possibilities for difference in findings. Why would we treat Chinese research different than industry funded research? A subject which, by the way, has much more extensive documentation of bias! That said, when editors reject Cochrane Reviews because they have Chinese authors, or reject obviously high quality Western-published journals because they have Chinese authors, maybe they're just belligerent POV pushers rather than racists. But they put their POV above all else and that's a problem. LesVegas (talk) 21:25, 21 September 2015 (UTC)

But this has never happened, your diffs don't support it. CFCF 💌 📧 00:09, 22 September 2015 (UTC)

Ummm, there is a diff at the top of this very thread where an editor objected to a journal article in Medicine because its authors were Chinese. Did you not see that? LesVegas (talk) 00:22, 22 September 2015 (UTC)

The diff doesn't say that, it speaks of 'a well documented bias' and expands later. Mis-quoting people doesn't strengthen your case. In other areas of WP should we not be allowed to say that govt X, TV station Y or news outlet Z, has 'a well documented bias' and should therefore be treated with extreme caution ? Pincrete (talk) 07:56, 22 September 2015 (UTC)

Pincrete, I am sorry I forgot to show the actual source, but, anyway, here is the source that was removed. As you can see, it is a high quality journal that just so happens to have Chinese authors who did the meta-analysis. The diff in question was trying to apply the "well-documented bias" to this meta-analysis and systematic review. There is another one where the same thing happened regarding several Cochrane Reviews used to update an old claim. They just so happened to have Chinese authors and that's a problem. LesVegas (talk) 22:43, 22 September 2015 (UTC)

When you actually read the conclusions in that paper they say very little, and above all they comment on the lack of qualitative studies in the field. It states "Finally, the included RCTs were all conducted in China, so more studies are needed" and "Reporting biases could not be detected by funnelplot due to lack of adequate RCTs."
From the entire paper a single positive sentence was taken and copied verbatim into Wikipedia (amounting to copyright violation). Maybe the article has a role in the sources, but it did not properly support the statement it was used to support and the comment you linked reflects that. CFCF 💌 📧 23:12, 22 September 2015 (UTC)

CFCF, I agree with most of what you said above. Yes, it was a copyright violation. And yes, the quote wasn't fully representative of the source. And by the way, I didn't add the quote or source, that was another editor. I had nothing to do with it beyond watch it get removed. But the reasons it was removed were wrong. Removing it or amending it for reasons you stated are perfectly fine. I have no problem with that. Editors shouldn't be supporting removal of sources because the authors are Chinese. I hate to belabor the point, but you said above that it never happened and my diffs don't support it, so I just have to clear my name here. LesVegas (talk) 02:27, 23 September 2015 (UTC)

The article states it is a problem that their study has included so many chinese studies, so I find nothing wrong with the comment you've linked. CFCF 💌 📧 07:23, 23 September 2015 (UTC)

CFCF Do you have access to the full article? Would you mind posting what they say re Chinese studies? It was not in the free text they provided. LesVegas (talk) 20:08, 23 September 2015 (UTC)

LesVegas, I am not competent to assess the virtues or weaknesses of specific medical research, my opinion on that would be valueless. However, specific pieces of research are not the subject of this RfC, rather a general principle. Pincrete (talk) 08:25, 23 September 2015 (UTC)

Question for Kww

User:Kww is mentioned repeatedly above, and it doesn't look to me like anyone has pinged him about this or given him a fair chance to explain his off-the-cuff comment that most of the "authors are Chinese" on a particular source. (Does that refer to their race? Their citizenship? Where they were trained? Where they're currently working? A quick guess based upon last names? It could mean almost anything.) IMO it would be fair to let him have his say if he's interested. WhatamIdoing (talk) 04:32, 23 September 2015 (UTC)

The diff shows an edit by Guy (JzG). The history of Talk:Acupuncture says Kww hasn't posted there since July. Johnuniq (talk) 04:53, 23 September 2015 (UTC)

Yes I'd love for him to explain himself because I and other editors could never get an explanation back when I confronted him about it. But as I understand it, he doesn't edit much anymore since his desysopping, correct? LesVegas (talk) 15:00, 23 September 2015 (UTC)

Actually I just looked at his edit history and I see he occasionally and casually edits here and there still, so maybe we can finally get an explaination on the specifics. LesVegas (talk) 15:03, 23 September 2015 (UTC)

I don't respond much unless pinged. The diff you are providing is JzG's, but I'll defend it. The bias of Chinese medical journals and Chinese studies submitted to Western medical journals is well-documented. For such sources to discover beneficial effects to acupuncture isn't particularly surprising, nor is it likely to indicate that acupuncture has any particular effect beyond placebo. There are similar problems with sources related to ayurveda. While I understand the slippery slope of racism, it's also important for us to note that associated with some of the rituals that are misrepresented as medicine there are one or two countries that have a vested interest in portraying that ritual as if it were effective. As for Les Vegas's proposal above, I don't think anyone is attempting to remove sources due to a personal objection to the country of origin: it's due to well-documented and objectively provable objections to the country of origin.—Kww(talk) 16:03, 23 September 2015 (UTC)

Thanks for the reply, Kevin; I appreciate it. Thanks also to John for pointing out that I had the wrong diff. I apologize for the sloppy post; I meant this diff. WhatamIdoing (talk) 16:12, 23 September 2015 (UTC)

As for the corrected quote, note that my primary objection was that even taken at face value, the sources didn't support the proposition Les Vegas was making: "not effective for a wide range of conditions" isn't contradicted (or even particularly weakened) by evidence of positive effect for a narrow range of conditions. I didn't get into a detailed analysis of the sources themselves. I will say, however, that it's a fair bet that Z Zheng, CCL Xue, J Shang, X Shen, J Xia, X Zhu, L He, and J Song being among the reported researchers in a small handful of reviews represents a reasonable foundation to suspect bias. If there was actually a meaningful claim being made, I'd scrutinize the sources more carefully.—Kww(talk) 16:29, 23 September 2015 (UTC)

Thank you for the explanation. Kww's answer goes right to the heart of the matter. If authors have Chinese names, we should suspect bias even if it's in a Cochrane Review, as if Cochrane is incapable of vetting the material themselves, but we, the lowly Wikipedia editor are. Frankly, I'm really not surprised. LesVegas (talk) 20:04, 23 September 2015 (UTC)

Nor should you be. What's surprising is that you would think that we should close our eyes to an indication of bias noted in reliable sources. Note that I did not reject the sources outright, but was arguing primarily on the basis that your conclusions didn't follow from their statements even if we presumed they were completely accurate.—Kww(talk) 20:54, 23 September 2015 (UTC)

Yes, I recall very well. Your primary argument was that the Cochrane Reviews didn't support replacing a 6 year old claim with an update. Your secondary argument was that they might be unreliable anyway because they have Chinese authors. That's what is being disputed here. Please note that the wording of the MEDRS amendment in this RfC says "do not reject high quality sources because of x,y,z...or country of origin." It does not say do not reject high quality sources if they are being used to support a claim that isn't supported by those sources, or is more accurately supported by another source. Policies elsewhere already cover those objections. LesVegas (talk) 21:41, 23 September 2015 (UTC)

If you had found material that actually supported an update, I would listen. Once again, "not effective for a wide range of conditions" and "is effective for condition x, y, and z" are not contradictory. They are barely related. The original source surveyed the impact on over thirty conditions and you found a source that weakened the position on, at most, two of those conditions. As for your propensity for misinterpreting sources and using them to support claims they don't support, I hope this debate makes that problem more obvious to a wider variety of editors. Your motivation for requesting this change has been noted by numerous editors: you are upset that people want to see corroboration in other sources when you can only offer Chinese sources supporting a claim about acupuncture. Given the noted bias problems, that's a quite reasonable demand. If something is actually true, Western sources will eventually catch up.—Kww(talk) 22:08, 23 September 2015 (UTC)

You're back to omnisciently assuming my motivations again. Just for the record, my motives are simply to encourage editors to examine source quality and not reject them for paltry reasons like the last name of their authors. LesVegas (talk) 22:21, 23 September 2015 (UTC)

Examination of source quality does require taking bias considerations into account. Your requested modification is basically a demand to ignore one potential source of bias. As for omniscience? No, pretty much any observer of your edits and discussions will come to a similar conclusion.—Kww(talk) 22:38, 23 September 2015 (UTC)

"Examination of source quality does require taking bias considerations into account" is just another way of saying high quality sources that we never question anywhere else for any other claims might not be high quality at all if we see it has Chinese authors, whom we ought to be suspicious of because they might be biased. Thank you for making your position clear, over and over again. LesVegas (talk) 23:03, 23 September 2015 (UTC)

I support being suspicious of any biased source (including Coke-paid researchers on the effects of soda pop on obesity [on which I didn't comment, apparently], ayurvedists on the benefits of heavy metal poisoning, Christians on the historical existence of Jesus of Nazareth, among many other examples a quick search of my edit history will uncover).—Kww(talk) 23:58, 23 September 2015 (UTC)

Ok, I'm glad to see you are consistent. I'm glad you brought up the Coke funding of obesity studies which is an example I have brought up here as well. MEDRS currently states that editors shouldn't reject high quality sources based on their source of funding. So that would mean Coke studies are perfectly ok to use, as are countless numbers of other industry funded research which has been shown to be far worse than Chinese studies on acupuncture. Why should we have one guideline where we don't reject industry funded studies, but we do reject sources because they originate in China or have Chinese authors? To me, I wouldn't even have raised this issue if I didn't see an inconsistency in our guideline where we allow carte blanche acceptance of industry funded, obviously tainted and conflicted research, but we reject Chinese sources because they might be tainted. I have never accused anyone here of being racist. I honestly believe editors here are good people, some just have blindspots and I know I do too. But frankly, if we continue to greenlight biased industry funded research but disallow Chinese sources, that is racist. If editors here want to consider Chinese sources suspect, while I deeply disagree with that, I can understand the point. But we had damn well amend MEDRS to treat industry funded sources the same way. We're not going to have two standards here. LesVegas (talk) 04:00, 24 September 2015 (UTC)

You shouldn't attempt to fix the inconsistency in the wrong direction: two wrongs don't make a right and all that.—Kww(talk) 04:15, 24 September 2015 (UTC)

The question of industry funding was discussed at length very recently on this very page. There, WP:DNFT was invoked and I think it now applies here too. Time to close. Alexbrn (talk) 04:24, 24 September 2015 (UTC)

You can ignore a problem or inconsistency, you can IDHT, but editors shouldn't be insulted for pointing out problems. And previously, that was invoked by an editor who much of the Wikipedia community has complained about for GMO advocacy and personal attacks and I have no reason to doubt that his invoking DNFT wasn't a great illustration of both his advocacy and insults. I can certainly see why a GMO advocate doesn't want to limit industry funded sources in any way. This same editor also believed Chinese sources should be limited rather inconsistently. But I expected much better from you, Alexbrn, than to ignore and insult and hope this embarrassing inconsistency just goes away. And as much as I deeply disagree with Kww on this issue, I at least have to respect and admire his consistency here. And we should be consistent, one way or another. While I think the best way with our policies is to allow high quality sources regardless of industry funding or country of origin (because it's too hard to say that because some have been bad, all are bad), we cannot have one set of guidelines for some sources and another set of guidelines for others. I know it's uncomfortable for some editors to admit this partisan application, but If we bury our heads in the sand and pretend it's not inconsistent, that won't magically make it consistent. LesVegas (talk)

You've illustrated the problem quite nicely: there's no particular correlation between the conflict of interest inherent in the Coca-Cola funded obesity research and the general agreement of all responsible researchers that GMO as currently practiced presents no health or public safety issues. While I abhor the inconsistency, I can understand why people would see a slippery-slope problem on the funding issues. After all, most good research is funded by people connnected with science, and science pretty much rules out the anti-vax, anti-GMO, pro-ayurveda, and pro-acupuncture camps, giving those people an incentive to object to nearly all research as being biased in one way or the other.—Kww(talk) 00:05, 25 September 2015 (UTC)

Well, I don't claim to be an expert on the latest in all things biotech, but I know Monsanto funds much biotech safety research and even when" purely independent research" is touted in the press it's often funded by The American Society for Nutrition whose own website says is funded by Monsanto. They fund the science all over the place. And this idea that industry-funded research was only a problem with Coke today and big tobacco in yesteryears is not accurate at all. What I do know is there is well documented issues with pharmaceutical industry funded research in the way of psychiatric medications and in failing to warn about cancer risk with certain drugs, as well as in showing a huge statistical increase in "positive findings" for various pharmaceuticals just to name a few. They were all connected with "science" yet clearly have shown worse issues than the Chinese because they're not explainable by a slew of confounding factors from an entirely different culture and a medicinal art many researchers have openly admitted is very difficult to study anyway. I see no argument for why we should treat Chinese studies differently than those paid for by Monsanto, Merck or the Marlboro Man. LesVegas (talk) 19:29, 25 September 2015 (UTC)

It's a balancing act: whether an individual drug does or does not create a cancer risk is something with gradations of bias that can range from damning (a study by the manufacturer that contradicts all other studies would need to be treated with great suspicion, for example) to minor (a study by a research group funded by multiple sources, including the manufacturer, which falls in line with similar studies by groups not involving the manufacturer). In the case of Chinese studies of acupuncture, we're dealing with a group that tends not to publish negative results, refuses to use standard placebo-based methodologies, and comes to conclusions that researchers using more rigorous research techniques cannot reproduce. There's no reason to suspect that the results are accurate, and every reason to suspect that the results are due to flawed research and reporting. There's nobody arguing to exclude all Chinese research, just to avoid relying on a known trouble spot: Chinese studies of treatments based on Chinese folk remedies and superstitions. Is there anyone objecting to Chinese studies of pharmaceutical treatment of tuberculosis or anything like that? I think not.—Kww(talk) 23:01, 25 September 2015 (UTC)

Actually, negative results in China weren't seen in 1998, they are now. But sure, in more recent times it's noted with different results seen in US studies. Some of that could be publication bias, some of that could be due to other factors. Fact is, we don't know. But industry funded pharmaceutical results are also seen to have much better results than independent research shows and those are comparing results within the same country. We have one variable to deal with there, not countless, so it makes publication bias much more likely a factor than is the case with the Chinese. I'm talking hard facts and statistics here, not your opinions of Chinese culture or your opinions of alternative medicine, but documented facts alone. And by the documented facts alone, even in the worst case-scenario with the lowest quality published research in China, Chinese studies on acupuncture have actually less possibility of publication bias than industry funded pharmaceutical studies do from the standpoint of variables. Any reasonable person examining facts alone and not swayed by opinions or POV must agree with this. And yet we specifically allow industry funded pharmaceutical studies on Wikipedia, and some of these same editors want to reject Chinese studies on acupuncture. LesVegas (talk) 16:59, 26 September 2015 (UTC)

Do we all understand the outcome here?

So we have a bit of a procedural issue with RFCs, which is that one editor lists almost all the expired RFCs at WP:ANRFC, so that some busy WP:NAC can carefully add a colored box and a closing statement to each of them. I think this is an ongoing WP:NOTBURO problem, a waste of time for most RFCs, and sometimes even an insult to the participants' good sense and ability to understand their own conversations, but let me ask just to be sure:

Does anybody here need any outside help in figuring out whether or not this discussion shows support for the proposed addition of the words "country of origin" to that sentence in this guideline at this time?

If nobody actually needs any help with understanding the outcome, then perhaps we can prevent him from wasting someone's time with telling us what we all already know. WhatamIdoing (talk) 17:49, 6 October 2015 (UTC)

The discussion above is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

Mid Quality sources

I have noticed a trend in the last few threads... debates that focus on defining the source in question as either "low-quality" or "high-quality"... and not considering whether it might be "mid-quality". I think we all agree that high-quality sources are preferred, and that low-quality sources should be avoided. But what about mid-quality sources? What often happens is that supporters of the mid-quality source try to elevate it to high-quality status... while opponents try to demote it to low-quality status. I think it important to assess sources realistically... so perhaps we need some discussion on how to present mid-quality sources. Blueboar (talk) 12:33, 6 September 2015 (UTC)

I think the reason we don't speak of mid-quality sources is because there of is such a divide between the low and high quality sources. It might seem confusing but what we are saying is basically that we accept high quality and top of the line Cochrane-level sources. Mid-quality sources would be primary sources such as RCTs and well-performed single studies–check out the pyramids I added above. We don't discuss these so much because Wikipedia doesn't allow them (or at least strongly discourages them). -- CFCF 🍌 (email) 21:51, 6 September 2015 (UTC)

You know, this is an interesting query well worth exploring. I think what we'd first have to do would be define what a mid-level source is. Is it peer-reviewed, but not in a highly well-known journal like JAMA? Does impact factor have any bearing on level? Or is it less about the publication and more about the type of study? So are meta-analyses superior to systematic reviews for identical claims? Do higher level sources use established reporting standards for their field? I would think that despite what mid-level sources are, we might still allow them but give them less weight obviously excellent sources. What that would probably mean for the article is that we should only use high-level sources in the lede while mid-level sources could be used elsewhere. High level sources might ought to be listed first in a paragraph, while mid-level sources would come later. All of that would, of course, be dependent on what we agree mid-level sources even are. LesVegas (talk) 22:39, 6 September 2015 (UTC)

This is the same idea that you have proposed already at the talk page of the acupuncture article and has been rejected. See Talk:Acupuncture/Archive_28#Chinese_Source_Idea and Talk:Acupuncture#The_Chinese_question Jytdog (talk) 00:11, 7 September 2015 (UTC)

Its appropriate for an editor to bring comments and questions to the larger community especially since in this case LesVegas is responding to Blueboar above (Littleolive oil (talk) 01:30, 7 September 2015 (UTC))

You also argued in those threads for inclusion of Chinese sources on acupuncture in some way like this. I do understand that both and LesVegas and likely Herbxue are pleased to find an opening in what Blueboar wrote above, for what you already wanted; likewise it is reasonable that Blueboar should be informed of the conversation into which he/she is walking Jytdog (talk) 14:54, 7 September 2015 (UTC)

Once again your response is a personal attack. No, I did not argue for inclusion of any sources carte blanche, Chinese or otherwise. Blueboar has been around along time and is policy expert in my opinion. I'm sure he can handle this discussion. Sheesh Jytdog give it a rest.(Littleolive oil (talk) 15:13, 7 September 2015 (UTC))

No it isn't an attack, it is a simple description. You regularly advocate for alt-med in WP. You also blatantly misrepresent me. I did not write anything like "any sources carte blanche," - I wrote "in some way like this." I don't know what you think that gains for you. Jytdog (talk) 15:48, 7 September 2015 (UTC)

• I've moved this back to its own section, here. This was posed as a separate issue and question from the RfC above, and should be dealt with as such. There may be a better way to do this if so please feel free.(Littleolive oil (talk) 14:29, 7 September 2015 (UTC))

We should not be judging the quality of something based on whether it is published in a "highly well-known" journal, nor should impact factor have any bearing. There are hundreds (thousands?) of reputable journals applying the same level of peer review and oversight. We should reject fringe journals etc., but as a scientist I give exactly the same weight to something whether it is published in something like JAMA, Heart or Gut, or if it is in a less well-known, lower impact factor journal like International Journal of Social Research Methodology, Knowledge-Based Systems or Informatics in Primary Care (all IF < 1). Bondegezou (talk) 17:04, 14 September 2015 (UTC)

Bondegezou, I believe that some of our editors unfortunately believe that IF < 1 defines "fringe journal". It can be difficult to identify fringe journals if you're not in the field, so some of them try to take shortcuts like that. User:DGG was meditating on a page at Wikipedia:Impact factors (since redirected by User:LeadSongDog to WP:Scholarly journal) to explain the limitations of that approach.

User:Blueboar, I'm not sure that I can agree with User:CFCF's statement that mid-quality sources are primary sources, or that we are really that close to banning them. One of MEDRS's structural problems is that it was written primarily to handle treatments for major diseases. It applies poorly to the other (90%) article contents for major diseases, and to the 90% of diseases that aren't so "popular".  ;-) Fixing that is going to require encouraging editors to use their best judgment, instead of imposing one-size-fits-some rules such as "you must only use secondary sources" or "six-year-old sources are bad". WhatamIdoing (talk) 04:41, 23 September 2015 (UTC)

Obviously, it depends on the topic. In an area where the are first rate journals, it is safe to assume that the second-rate journals publish the less important work. Within a popular, field, for the same type of journal, the IF can be a good guide to this. In narrow fields, where relative few people work, the IFs of all the journals will be low, and we simply have to use the best material. It can take a certain amount of knowledge of the literature in the area to figure this out. A mechanical approach to reference selection or article writing doe not work well. DGG ( talk ) 04:53, 23 September 2015 (UTC)

WP:MEDASSESS and hierarchies of evidence

In a recent WikiProject Medicine discussion, I noted that WP:MEDASSESS largely follows an evidence-based medicine approach, but that there are many valid critiques of the EBM movement (examples given in that discussion). I note Pearce et al. (2015) says:

"the notion of ‘hierarchies’ of evidence within evidence-based medicine is no longer prevalent in the literature, being replaced by more nuanced typologies of evidence demonstrating how different research methods are appropriate for answering different types of research question [72, 73]."

72. OCEBM Levels of Evidence Working Group. The Oxford Levels of Evidence 2. Oxford: Oxford Centre for Evidence-Based Medicine; 2011. 73. Howick J, Chalmers I, Glasziou P, Greenhalgh T, Heneghan C, Liberati A, et al. Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document). Oxford: Oxford Centre for Evidence-Based Medicine; 2011.

Perhaps we are a bit behind the times now in following a "pyramid" hierarchy and should work on some alternative text.

Here has the set of papers, all open access, that includes Pearce et al., a series edited by Trish Greenhalgh. Bondegezou (talk) 16:32, 14 September 2015 (UTC)

Yes, we need to address this. I've been thinking about it for a long time—not because there's something "wrong" with the evidence hierarchies, but because they're only useful for some kinds of information. You will never see a systematic review of randomized controlled studies on what the current name of body part is. It'd be silly. And yet we have editors who believe that this is the be-all and end-all of all sources, and that it's better to cite a systematic review that mentions something like that in passing than to cite a more authoritative source, e.g., a reputable anatomy textbook.

But I think it's going to take a lot of re-writing, and I've got limited time for it. WhatamIdoing (talk) 04:20, 23 September 2015 (UTC)

I'll try to find time, but the issue in the case you've linked is not that EBM is weak, but that it does not apply to all fields. CFCF 💌 📧 07:07, 23 September 2015 (UTC)

Writing for Wikipedia on scientific topics is not "doing" peer reviewed science, more especially it is not authoring reviews. It is, or should be, non-credible mostly-pseudonymous editors reporting on where the existing consensus of scientists lies. This is the underlying reason that we cannot use our own individual assessment of primary sources, even the experts among us. The result is that we cannot simply adopt a ranking hierarchy intended for use by experts: we must have published sources behind our application of wp:WEIGHT, as inconvenient as that may be when we all know the primary source is right.LeadSongDog come howl! 13:06, 23 September 2015 (UTC)

One of the (multiple) problems that we create with mindless application of a "no primary sources!" rule is that we exclude interesting information as a result. Primary sources need to be given due weight, which doesn't always mean excluding them. For very rare diseases, due weight might even mean emphasizing them—because the options are sometimes scholarly primaries or non-scholarly secondaries, and the "scholarly" part can matter more than the "secondary" part. We need to stop painting ourselves into a corner by declaring a subject notable because independent secondary sources exist, and then refusing to permit any sources at all to be used: all the academic sources get rejected on the grounds that they're primary or too old, and all the secondary and recent sources get rejected on the grounds that they aren't peer-reviewed.

Also, encyclopedia articles aren't just summaries of current mainstream scientific consensus. They should also include (largely scholarly, largely mainstream) views from multiple disciplines, such as:

• history (why does the word cancer come from crabs?),
• law (where is this drug legal?),
• business (how much money did they make selling this?),
• economics (what would happen to the global economy if depression disappeared?),
• sociology (why does this affect wealthy countries more than poor ones?),
• religion (can this surgery be done bloodlessly?),
• culture (why do people bring you dinner if your child has cancer, but not if he has autism?),
• practical life (can a heart attack make you too weak to lift a pot of coffee?),

and other subjects. None of that is science, and all of it belongs in a medicine-related encyclopedia articles. WhatamIdoing (talk) 16:08, 23 September 2015 (UTC)

MEDRS has exceptions for all those (including primary sources for rare diseases)–so I don't see the issue here.
As I understand it the problem is that misinformed editors are mass-tagging for example 6 year old studies as unreliable. While that is a major issue, it isn't an issue with the current rules.
As for notability I don't understand, would you care to clarify? CFCF 💌 📧 22:05, 23 September 2015 (UTC)

On notability: Notability, in the form of the WP:GNG, basically requires the existence of two or more secondary sources. MEDRS, however, "requires" (in the minds of our most zealous defenders of the wiki) the exclusive use of scholarly secondary sources for WP:Verification. This means that it is possible to have an article about, e.g., an experimental drug, that is WP:Notable (because multiple non-scholarly secondary sources exist) but which produces endless disputes about the lack of "MEDRS-compliant" sources (because zero scholarly secondary sources exist). We are handling this poorly. WhatamIdoing (talk) 17:57, 6 October 2015 (UTC)

MEDDATE

Here's an approximate idea of what I'm thinking about MEDDATE issues.

The ideal maximum age for a source depends upon the subject

Subject	Example	Maximum recommended age
Major topics in a major, actively researched area	first-line treatments for hypertension	Review articles published within approximately the last five years
Minor topics in a major area	treatment of hypertension in a person with kidney cancer	Approximately five years or the three most recent review articles, whichever is longer
Major topics in a minor area	treatment of cystic fibrosis	Approximately five years or at least three reviews, whichever is longer
Minor topics in a minor area	treatment of hypertension in a person with cystic fibrosis	The several most recent review articles, and any primary sources published since the penultimate review article
Very rare diseases	most genetic disorders	The several most recent peer-reviewed articles, regardless of absolute age

Does this seem approximately like what you all would expect to find if you were looking for sources? (On the fourth line, it may help to know that hypertension does not seem to be a common complication of cystic fibrosis.) WhatamIdoing (talk) 18:22, 6 October 2015 (UTC)

I like the idea of a table of different variants like this, but it's going to need some work. There are a couple of issues here:

 1. Ideal source are obviously always new up to date systematic reviews regardless of topic area

 2. Do we use number of reviews to determin which are major/minor topics and how do we know ehn a topic falls under any of these groups?

 3. How do we determine if reviews/articles have been published? You have WebOfScience and Scopus for this but very few editors have access. Pubmed doesn't really cut it.

 4. The wording "regardless of absolute age" is problematic because all you need to do is go back to a 1970s East German source and you can promote a wealth of alt-med diseases. I'd be more comfortable is we used something akin to Orphanet [4] to determine what rare diseases are - many old purported diseases are just that, and aren't considered real today.

 5. "any primary sources" is far to inclusive

 6. Best possible sources don't depend on subject but rather how much research has been performed - this means that even some very rare diseases have quite significant bodies of research.

CFCF 💌 📧 20:31, 6 October 2015 (UTC)

I agree that it needs work; that's why I posted it.  ;-)

1: Ideal sources are not always systematic reviews. The ideal sources for treatment efficacy are systematic reviews – assuming any exist – but systematic reviews are not the ideal source for 90% of article content.

2 and 3: I think that the number of reviews available might be one reasonable metric for major/minor (and all the things in between). We can base this on PubMed and treat it as a rebuttable presumption: if I find nothing in PubMed, but you've got access to Scopus and find more, then you can share your information with me. By the way, here are some quick numbers:

• "Hypertension" is mentioned in 12,730 (tagged) reviews on PubMed in the last five years, and is present in the title of 3,162.
• "Breast cancer" is mentioned in 6,447 and in the title of 3,741.
• "Pneumonia" is mentioned in 2,808 and in the title of 695.
• "Cystic fibrosis" (a heavily researched rare disease) is mentioned in 1,543 and in the title of 692.
• "Preeclampsia" is mentioned in 779 (plus more under the hyphenated spelling "pre-eclampsia") and in the title of 279.
• "Down syndrome" is mentioned in 469 and in the title of 160.
• "Kidney cancer" is mentioned in 225 (plus 174 non-duplicates for "renal cancer") and in the title of just 64 (plus 51 for "renal cancer").
• "Wilson disease" is mentioned in 62 and in the title of 17 (plus more for "Wilson's disease").
• "Oculodental digital dysplasia" (incredibly rare disease) is mentioned in zero.

(These are all quoted-phrase searches on PubMed, merely for illustration rather than ideal searches for these subjects.)

As a quick rule of thumb, then maybe this would work: If there are more than 100 hits among reviews published on the subject in the last year, then you should probably be using the "major" criteria for the bulk of your sources. If there are less than 100, then that might not be possible (because "hits" ≠ "reviews actually about the subject"). Or we could build it based on in-title searches: Use good reviews if you've got more than a couple dozen, but when you've only got 20 (or fewer) to choose from, the fact is that the available sources might not cover all of the material that ought to be in the article. For example, there is exactly one review that has both "cystic fibrosis" and "hypertension" in the title during the last five years, and if you need to source a sentence about non-pulmonary hypertension (perhaps to mention the need to control hypertension in advance of getting a lung transplant), then there are zero recent reviews available on that exact subject.

However, I think that most experienced editors are going to have an easy time deciding where a subject falls on the scale. If I have no trouble discovering sources, then it's a major topic. If my searches come up empty, then it's not. You should be using the best of what you've got, unless and until someone demonstrates that better ones; conversely, when better ones don't exist, then you should not be hassled by people who care about the date on the paper more than they care about the content of the article.

4. Bad sources are bad sources. Age is not the sole, or even main, determinant of whether a source is bad.

5. Bad sources are bad sources. Primary vs secondary status is not the sole, or even main, determinant of whether a source is bad.

6. Best possible sources do depend on the subject, because the subject determines how much research has been published. I believe that you meant to say that the best possible sources don't depend upon disease prevalence.  ;-) Also, it's necessary to write these rules to work for non-disease subjects, such as drugs and surgical techniques. WhatamIdoing (talk) 22:00, 6 October 2015 (UTC)

While I appreciate the thinking behind this proposal, I suspect that it will make things worse rather than better. As we all know, there is already a tendency – usually but not always editors who aren't familiar with how to read and use the published literature – to treat MEDRS as a series of yes/no checkboxes that must be met, rather than as a set of rules of thumb which a skilled editor might consider in evaluating a given source-assertion-context triple. (See also the related problem of editors who think that "reliability" is a magical inherent trait possessed by a source, without regard for how or where that source is being used. And editors who had WT:MED watchlisted earlier this year will be familiar with the individual who thought evaluate this article meant make a complete list of its citations older than 5 years and declare them not MEDRS-compliant, regardless of context.)

Creating a more-specific-looking set of criteria increases the tendency for slavish adherence to the letter of the rule rather than to the purpose of the rule. Saying that "most experienced editors are going to have an easy time deciding where a subject falls on the scale" misses the likely source of the problem— most experienced and competent editors already grasp the need for flexibility in applying MEDRS' guidelines. Where a question about a source arises under these new criteria, the discussion will be diverted from the central question of whether or not the source-assertion-context triple at hand is appropriate, and into bickering over whether a particular topic and area are major/major, major/minor, minor/major, or minor/minor. Once that binary categorization is achieved, there will be blind counting of number of reviews or blind adherence to the five-year criterion—which is the same problem we already encounter. And since the new criteria look more specific and 'scientific', then we're probably going to have more trouble dislodging individuals from their mistaken belief that these rules of thumb are etched in stone. TenOfAllTrades(talk) 11:49, 16 October 2015 (UTC)