Wikipedia talk:Identifying reliable sources (medicine)

Request for a list of high quality journals

There has been a request for use to create a list of high quality journals people can feel comfortable using. Wondering what peoples thoughts on this are? Have begun drafting something here User:Doc_James/RS#Journal_articles Doc James (talk · contribs · email) 19:56, 25 June 2015 (UTC)

Great project. Since "Good-vs-bad" is not exactly a black-and-white issue, I'd say the more annotation and explanation of how conclusions are arrived at, the more useful it will be. -Pete (talk) 20:17, 25 June 2015 (UTC)

I also see the need for this. But I think it's neither pragmatic nor is it possible to create such a list. Plus, I think some editors are likely to edit war anything that's not on the official list right off Wikipedia. What I think might be more workable is to tighten our guidelines for what journals are acceptable and what journals aren't. Can we come up with a better list of guidelines, based on principles of what an unreliable journal versus a reliable one are? The ones we have are a bit too vague, I think. LesVegas (talk) 16:45, 30 June 2015 (UTC)

As I suggested on Wikipedia talk:WikiProject Medicine, I agree with Las Vegas that a clearer guideline would be more useful than a list. A list of the highest quality journals could be created, but couldn't the contents be summed up nearly as well with a generalized 3 - 5 point criteria for inclusion?Dialectric (talk) 17:24, 30 June 2015 (UTC)

I am concerned that such a list will be misunderstood, as others note above, as a binary yes-no type of list. We'll get editors who insist that since they read claim A in a journal on the list, it must be added to our Wikipedia article; we'll get editors who insist that because claim B isn't from a journal on the list, it must be removed from Wikipedia. Both attitudes will be problematic. We already have a similar difficulty with editors who don't understand – or wish to abuse – the 'try to use sources from the last 5 years' rule of thumb. This will likely be even more messy.

Sometimes bad papers appear in 'good' journals. Reputable, sturdy, solid science involving unsexy topics or with unflashy results ends up in middle-of-the-road or lower-impact journals—regrettably, often side-by-side with dramatic, sexy results produced by crappy science. (With a high variance, the product of "study importance" and "study quality" is proportional to a journal's impact factor.)

Evaluating the quality of a journal article is difficult. Doing it with competence requires, literally, years of experience and training. While I appreciate the impulse to try to help editors understand the standards for sourcing on Wikipedia, there is a limit to what we can do with rules of thumb and examples and suggestions. There isn't a magic six-point checklist that will let someone with little or no experience reading and evaluating scientific literature to learn to do so in five minutes (or five hours, or even five months). An extraordinary amount in such an evaluation depends on context and contextual knowledge. TenOfAllTrades(talk) 19:47, 30 June 2015 (UTC)

Something along the lines of WP:ELPEREN might be doable - creating a list of source that are frequently discussed where there is strong consensus on how they should be used in MEDRS situations. Such a list could even include Quackwatch. --Ronz (talk) 01:19, 1 July 2015 (UTC)

A list of high quality journals does not equal these sources are useable. Sources are reliable per the content they support and not because they are deemed to be high quality. While it might be helpful to have a list of high quality journals that list in no way determines reliability per Wikipedia content. (Littleolive oil (talk) 01:33, 1 July 2015 (UTC))

But if a list of "high-quality journals" is created, even if it is festooned with disclaimers about not being an absolute complete list, then it absolutely will get abused to say that X must be included because it was published in a journal on The List, and Y must be excluded because it was not published in a journal on The List.

Also, does anyone else remember a discussion at WPMED a few years back, in which a bunch of mostly pre-med students tried to figure out what "common" or "typical" textbooks were being used for one specialty (gynecology, I think)? It basically failed. If we can't even get a list of a half-dozen textbooks created, by people who had to buy such textbooks, then I believe that an effort to pick "good" journals is doomed. WhatamIdoing (talk) 04:03, 8 July 2015 (UTC)

Agree with the skepticism above. I think in general, having a list of sources that are generally considered high quality (as an essay, perhaps) would be helpful. However, I agree that such a list will lead probably lead to more issues than less if incorporated as a part of this guideline. Yobol (talk) 13:07, 9 July 2015 (UTC)

• Comment Isn't this the list/place you were looking for https://en.wikipedia.org/wiki/Medical_literature#Instances Links from lede ? prokaryotes (talk) 21:30, 16 August 2015 (UTC)

This is exactly what you're looking for: Brandon–Hill list. Unfortunately Dorothy Hill retired in 2004 and it ceased publication. Some people call Doody's List a replacement, but Doody's List isn't quite the same (and is much more expensive). --Nbauman (talk) 09:46, 6 September 2015 (UTC)

The new wording "Primary sources should almost never be used for medical content."

With this edit, Alexbrn changed "Primary sources should generally not be used for medical content." to "Primary sources should almost never be used for medical content." Given the various discussions on this talk page about there being cases where primary sources are allowed, whether it's because of what WP:MEDDATE states about there being areas where "little progress is being made or few reviews are being published" or because an editor chooses to provide a primary source in conjunction with the secondary source, I felt that we should perhaps discuss if this wording is what we definitely want to go with in the lead of WP:MEDRS. For an example of a medical topic significantly lacking review articles, see clitoromegaly. Flyer22 (talk) 14:59, 8 July 2015 (UTC)

I'm as big a stickler for using reviews as the next person, but recognize that there are times when primary sources are not only appropriate, but the only sources to use. I'm not sure this change is an improvement. Yobol (talk) 13:05, 9 July 2015 (UTC)

Agree this new wording is not an improvement for the reasons previously given. Ward20 (talk) 17:56, 9 July 2015 (UTC)

• My position is "Primary sources should almost never be used for medical content before discussion and consensus on the talk page." Doc James (talk · contribs · email) 18:19, 9 July 2015 (UTC)

I agree with this wording too. If a topic does not have secondary sources, we generally don't consider it WP:NOTABLE to include an article on it in the first place. "Almost never" still does not exclude primary sources, but basically shows that someone is going to need to show a really unique case related to sourcing to use them. If someone can somehow demonstrate that a topic has weight for coverage, but we don't have great secondary sources and should instead carefully rely on some primary sources, I don't think it should be hard to demonstrate that. I don't see issues with the wording with that in mind. Kingofaces43 (talk) 00:47, 10 July 2015 (UTC)

• I am good with the new language or Doc James' version. I like the economy of the new language. Doc James' longer version could go in the body.... but I would change the new language to "content about health" instead of "medical content". Jytdog (talk) 00:52, 10 July 2015 (UTC)

Thinking about it more, I think looking at the body and lede this way may be the way to go. Alexbrn's original addition above (adding in content about health instead) makes for a good stern piece in the lede. Further along in the body, Doc James' addition could go in the Respect secondary sources section a bit later to show that a primary source is a huge exception to the rule and requires some work to use if that does happen. Kingofaces43 (talk) 00:59, 10 July 2015 (UTC)

Agree with the above suggestions. Doc James (talk · contribs · email) 03:09, 10 July 2015 (UTC)

Doc James's wording makes it seem like we should discuss using a primary source before adding it; as made clear above, first by me and then by Yobol, that should not always be the case. Then again, I think what Doc James means is that we should almost always discuss using a primary source before adding it; I agree with what he means on that. As for adding his wording lower on the page, the "Definitions" section is already clear that use of primary sources for health content should be a rare case; it uses the word rare twice. And the "Respect secondary sources" section is also very clear about not using primary sources, except occasionally. How much more do we need to stress this? Either way, because of past discussions at this talk page about the use of primary sources for health content, I still feel that Alexbrn's wording is too strong. For example, I've been clear that I sometimes like to provide the primary source in conjunction with the secondary source (which the "A primary source may be presented adjunctively to a secondary source." wording of the "Respect secondary sources" section currently allows); there is no problem with that since the secondary source is there for viewing. The bolded "generally" wording in the lead was strict enough with regard to discouraging primary sources. That was us enforcing the "be wary of primary sources" aspect of the guideline. We keep tightening the "primary source" aspect, and it happens each time a WP:Med editor finds misuse and/or excessive use of primary sources. Alexbrn's edit summary seems like recent misuse and/or excessive use of primary sources inspired him to add "almost never." But the misuse is not going to stop, no matter what wording we use. We have no proof that the bolded "generally" wording was working; furthermore, it had not been there for long, and already we've moved on to "almost never." The next step after "almost never" is "never"; and, well, "never" shouldn't be the case. All that stated, I agree with Jytdog that we should change "medical content" to "health content."

And, Kingofaces43, what do you mean by "adding in content about health instead"? Alexbrn simply changed a small piece of the text; "medical content" is in the previous and current wording. Flyer22 (talk) 04:19, 10 July 2015 (UTC)

My comment was about using "health content" instead of "medical content" as Jytdog pointed out. Kingofaces43 (talk) 04:26, 10 July 2015 (UTC)

I was confused by your wording because the "adding in content about health instead" part seemed liked you attributed it to Alexbrn. Flyer22 (talk) 04:33, 10 July 2015 (UTC)

Also, Kingofaces43, regarding your statement of "If a topic does not have secondary sources, we generally don't consider it WP:NOTABLE to include an article on it in the first place.", I don't think that the others were simply referring to no secondary sources. When I mentioned clitoromegaly above, I mentioned that it is significantly lacking review articles. It is reported in secondary sources, but review articles are obviously a different matter. So when citing studies for that topic, we will be citing primary sources...or secondary sources noting the primary studies. And some other medical topics on Wikipedia are similar. But you are correct that "If someone can somehow demonstrate that a topic has weight for coverage, but we don't have great secondary sources and should instead carefully rely on some primary sources, I don't think it should be hard to demonstrate that." Flyer22 (talk) 04:48, 11 July 2015 (UTC)

Often the first thing to come out is primary sources by those closely related to the topic at hand. They are thus often promotional and IMO we would do well to, much of the time, wait for decent independent secondary sources. Think of the topic CCSVI for example. Doc James (talk · contribs · email) 04:55, 11 July 2015 (UTC)

I don't dispute that. Flyer22 (talk) 05:09, 11 July 2015 (UTC)

The problem with this wording is that it creates confusion for editors who happen to be working on topics that are new and emerging fields. In these cases, primary sources might be the only sources available. In developed aspect of a field where there are plenty of secondary sources, yes, primary sources shouldn't ever be used. But not in other cases. I'm concerned some of our editors whom can't see in shades of grey might use wording like this to edit war primary sources that are necessary unless we temper it a bit. LesVegas (talk) 00:42, 13 July 2015 (UTC)

That is my concern, and the "areas where little progress is being made or few reviews are being published" and "new and emerging fields" aspects are why I started this discussion; we need editors to know that primary sources are not banned (WP:MEDRS is a guideline, not a policy anyway), and that there are legitimate exceptions to excluding them for biomedical content. Some editors who were and/or still are less well-versed in WP:MEDRS didn't even know that History sections, and Society and culture sections, are usually exempt from strict WP:MEDRS-compliant sourcing. WP:MEDDATE states as much of History sections. Flyer22 (talk) 01:01, 13 July 2015 (UTC)

I agree, that's quite a take-off from the rest of RS policy. "Should almost never" is not very helpful expression, but more confusing instead. If we want to add such definition, then we should also cover when the primary sources may be used. So far, "generally" leaves room for personal discretion, and we already have policies saying that "reliable secondary sources" should be preferred when available. For example, we might have a topic that has already established notability, but where some new findings emerge. As we know, it takes time before secondary sources are actually available, so primary sources may be considered, being careful not to break WP:UNDUE of course. Maybe a more common example would be that: we have a secondary source "A" making statement "B". The statement "B" is a bit ambiguous, so we may look up to the actual primary source that "A" is quoting to get a better picture, Cheers! Jayaguru-Shishya (talk) 19:25, 13 July 2015 (UTC)

That is not how you write policy/guidance in WP or anywhere. The idea is not micromanaging but setting broad and clear principles. With the language we are discussing, it is clear that someone would have to justify using a primary source; the justification would be discussed at the article talk page and DR from there. Jytdog (talk) 19:32, 13 July 2015 (UTC)

Reboot

It strikes me that much of the aversion to the "should almost never" wording is likely grounded in its prescriptive tone. A more descriptive approach could be simply "Primary sources are rarely useful for medical content." Comments? LeadSongDog come howl! 20:00, 20 July 2015 (UTC)

I see no issues there. Kingofaces43 (talk) 21:09, 20 July 2015 (UTC)

In the hands of an expert editor, primary sources can be extraordinarily useful. We discourage them because they result in WP:DUE violations as a result of cherry-picking by POV pushers, not because they are intrinsically non-useful for medical content.

I think that we need to write a WP:MEDDUE section or page, and to tie our preference for secondary sources to that. WhatamIdoing (talk) 08:19, 21 July 2015 (UTC)

I just noticed that you reverted back to "generally". As for LeadSongDog's wording, I wouldn't mind "rarely"; I mentioned above that "rare" is stressed lower (past the lead) in the guideline. As for WhatamIdoing's suggestion to write a WP:MEDDUE section, I trust that she could type up a decent or good section to address these issues. Flyer22 (talk) 03:04, 25 July 2015 (UTC)

As an addendum I've just said to an editor that per MEDRS, primary sources are "not reliable for health content". They then apparently consulted MEDRS and responded thusly:[1]

To claim that they "not reliable for health content per WP:MEDRS" is a misrepresentation at best or an outright lie at worst.

I see this again and again. Whatever we think we're saying, the repeated message that readers seem to take away from this guideline is still that primary sources for health content are okay. Alexbrn (talk) 06:13, 6 August 2015 (UTC)

Whether in a medical context or not, simply saying that primary sources are [not/rarely/generally not/...] useful isn't the point. The point is the purpose for which the primary sources are used. Thus primary medical sources should not be used to establish the efficacy of a treatment – no qualification needed. But once the efficacy has been established, if the article sets out details of the treatment, primary sources may be the most reliable for these details (e.g. the chemistry and pharmacology of any drugs involved).

This is a classic example of the danger of trying to reduce an issue which ultimately depends on human judgement (i.e. determining the purpose for which a source is used) to a rigid set of rules. It's bound to fail. Provide guidance, examples, explanations, but don't try to provide infallible rules. Peter coxhead (talk) 06:59, 6 August 2015 (UTC)

Broadly speaking, if new readers of MEDRS came away with the understanding you articulated, that would be great, particularly "primary medical sources should not be used to establish the efficacy of a treatment" (I would say "establish or imply" rather than just "establish"). But IME they often don't. Alexbrn (talk) 07:09, 6 August 2015 (UTC)

@Alexbrn: (Adding "imply" is clearly right.) IME too they often don't. But the solution isn't to keep writing more and more prescriptive statements, which just alienate experienced editors and confuse inexperienced ones. Peter coxhead (talk) 08:02, 6 August 2015 (UTC)

I don't believe that you (or anyone else) ought to say that primary sources are not reliable for health content. Some of them are: if you see a press release from a politician that says he's been diagnosed with a medical condition, then that is a self-published primary source about health content. It is also reliable for a statement that Politician Paul said that he had been diagnosed with ____. This guideline is about WP:Biomedical information, which is not the same thing as health content.

I think that Peter's example is a good one. I'll add, though, that "efficacy of a treatment" is maybe as much as 10% of the information that a good encyclopedic article should cover. We may need to re-write MEDRS to better explain what to do about the content.

Also: "ideal sources" (what MEDRS encourages) and "reliable sources" (the rock-bottom minimum that is required) are not synonyms. A primary source is reliable for quite a lot of material, even if it's ideal for relatively little. WhatamIdoing (talk) 03:08, 22 August 2015 (UTC)

Yes, 'health content' is too broad a term in general. I certainly didn't intend to outlaw the type of case you mention, the context in which I used the term was a dispute about whether Mate tea was a potentially effective treatment for certain conditions. Note that para 3 of MEDRS currently scopes it as follows: "This guideline supports the general sourcing policy at Wikipedia:Verifiability with specific attention given to sources appropriate for the medical and health-related content in any type of article, including alternative medicine" - which further demonstrates that MEDRS is a bit of a mess. Alexbrn (talk) 08:19, 22 August 2015 (UTC)

Thanks for the nudge. I've cleaned that up. WhatamIdoing (talk) 15:48, 22 August 2015 (UTC)

Beall’s List of Predatory Publishers 2015

FYI: Beall’s List of Predatory Publishers is a list of scientific publishers who will publish any paper, regardless of quality, for a processing fee. In some cases the predatory publishers uses a name that is similar to or identical to the name of a legitimate journal. --Guy Macon (talk) 16:51, 15 August 2015 (UTC)

Problems with Systematic review usage

The current lede mentions Systematic review, the 1st article cite in the lede links to a page which appears to be based on publications from 1996 about Evidence based medicine (scope is patient care), and does not include most of the content given in the lede of SR. The second cite of the SR lede points to CRD’s guidance for undertaking reviews in health care, the 3rd cite goes to this publication Systematic Reviews in the Social Sciences A PRACTICAL GUIDE, which explains, "In the book we use the term ‘‘systematic review’’ to cover both those reviews that include a statistical summary of the included studies (a metaanalysis), and those that don’t. While we also use the phrase systematic ‘‘literature’’ reviews, not all evidence which may be useful for a review will of course appear in the published ‘‘literature.’’ Because of this, the term ‘‘research synthesis’’ (or ‘‘evidence synthesis’’) is becoming increasingly common. We have used the term ‘‘systematic review’’ in preference, as it is still so widely used, but we do not assume that only ‘‘published’’ literature can be reviewed.". Hence the current lede of SR is entirely at odds (see WP:OR) with Identifying reliable sources (medicine). We should rather mention something about meta analysis, and literature reviews from peer-reviewed journals unless the SR article can be fixed. prokaryotes (talk) 22:03, 16 August 2015 (UTC)

More on point http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/syst-review.pdf prokaryotes (talk) 22:24, 16 August 2015 (UTC)

Suggest you take it to that articles talk page. -- CFCF 🍌 (email) 22:30, 16 August 2015 (UTC)

I did, but since this policy here overlaps with SR, and SMcCandlish suggested above to change the term as well, i brought it here for discussion.prokaryotes (talk) 22:33, 16 August 2015 (UTC)d

I mean the problem isn't with this page, but at the article. The What is a systematic review? page you linked is a good introduction and explains why we are so intent on keeping this wording here. -- CFCF 🍌 (email) 22:46, 16 August 2015 (UTC)

Ok, i agree the paper is much less confusing. prokaryotes (talk) 23:23, 16 August 2015 (UTC)

Industry Funded Studies

Do we allow industry funded studies on Wikipedia articles? There has been a documented problem of funding bias with industry funded studies, so I was curious if MEDRS restricted such sources of information in some way. LesVegas (talk) 02:10, 30 August 2015 (UTC)

In general, we do not disallow sources based on funding: "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions." (emphasis mine). There are all sorts of biases that can affect a study, we should be careful how we assess the sources so as to not disallow otherwise good studies based on issues that may have no relevance. Yobol (talk) 16:31, 30 August 2015 (UTC)

We do also state Wikipedia:Identifying_reliable_sources_(medicine)#Use_independent_sources. But we should be using reviews independent of the primary source anyway. Doc James (talk · contribs · email) 21:20, 30 August 2015 (UTC)

Thanks for your reply, Yobol. If industry funded studies have shown to have bias (much more positive outcomes compared to neutrally conducted research), why should we allow it on our articles? Shouldn't we change what MEDRS says? I mean, why did we add that tidbit into MEDRS in the first place allowing any research despite possible shady funding sources which might skew its results, and why shouldn't we change MEDRS now to disallow such sources? Isn't it a real problem if a pharmaceutical company funds research showing excellent results for their product if neutral research shows mediocre ones? LesVegas (talk) 02:41, 31 August 2015 (UTC)

I think there are two issues here: 1) Doc James' point is well taken, most of the funded studies you are discussing are primary studies, which we should be avoiding anyways in favor of independent secondary sources such as reviews; avoiding primary sources in favor of secondary will help us avoid those funded sources already. 2) the fact that something is funded does not mean it is automatically "shady" or suspect. There are plenty of good quality industry funded studies as well. We don't want to throw out the baby with the bathwater. Yobol (talk) 02:56, 31 August 2015 (UTC)

We just want to use the best available sources. And yes often these are reviews of industry funded trials. Doc James (talk · contribs · email) 03:05, 31 August 2015 (UTC)

Yes, industry funded studies do eventually make up reviews and meta-analyses, so if there is a problem with industry funded sources, there's a problem with reviews that use them. But I guess it is being argued that even though there is a well-documented history with industry funded research being unreliable with far too many results toward the positive side, it's too problematic for us as editors to determine what's reliable and what's not based on funding sources. Some of it could be good and we'd throw out the baby with the bathwater as Yobol said, so we really shouldn't be questioning it, just ensuring it comes from a source that meets all of our other requirements. But is that really all we're supposed to do as editors? Really? LesVegas (talk) 22:20, 31 August 2015 (UTC)

Not necessarily. Often a good review will take that into account. Doc James (talk · contribs · email) 22:46, 31 August 2015 (UTC)

Should we toss out reviews if they review industry funded studies but don't seem to acknowledge the possibility of funding bias? I mean, if the reviewer just treats the industry funded studies like they're the highest quality in existence and seems oblivious to a well-documented problem in the field, there's no way that can be reliable, right? And the reviewers too would be suspect, right? LesVegas (talk) 23:31, 31 August 2015 (UTC)

No. Aside from that being WP:OR on our part, if a study has a problem (industry funded or not), that will be mentioned in the review of the methodology or specific claims made in the conclusions. If there is an issue with funding or lab-specific related, the reviewers will say, "Studies from lab Y, funding sources Z, etc. all had inadequate methodology for assessing X." This happens every now and then when labs are found doing shoddy work. There are lot of times industry funded trials as designed and analyzed entirely properly, and reviews in that case wouldn't mention any issues from the get go. The sometimes subtle difference that people miss is industry-designed studies. Those are the studies where there isn't independence in terms of the scientists doing the actual work and interpretation. Kingofaces43 (talk) 23:41, 31 August 2015 (UTC)

In a perfect world, reviewers pick up on problems with funding sources that skew results. But since this is an imperfect world, if they miss problems in these studies, why exactly should we not disallow these reviews as sources? LesVegas (talk) 23:53, 31 August 2015 (UTC)

this has been discussed to death in the archives. No. we do not perform peer review on papers. We do look for independent, high quality sources of the types described in MEDRS. Jytdog (talk) 23:56, 31 August 2015 (UTC)

Good quesion LesVegas, though even if funded a study can be useful, and it is not clear which studies are skewed or not. But we can report on skewed results with follow up studies, as we do frequently. However, there is a problem when studies are funded and then not published, but used for decision making.prokaryotes (talk) 00:05, 1 September 2015 (UTC)

Addition to MEDRS for clarity

There have been documented [issues with industry funded studies] and even though they have shown to be overly positive and could even be unreliable, it is clear that, while possibly unreliable, these are allowed as several editors here have shown MEDRS saying we should not be performing a peer review ourselves. And yet, this stance is difficult to reconcile with the stance some editors on Wikipedia have taken regarding studies that come from other countries, which have shown issues similar to possible bias seen in industry funded research. As a result, I propose changing MEDRS which currently reads:

• Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

to:

• Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, conclusions, or country of origin.

Since the country issue has been brought up repeatedly and editors have rejected research because of the country it was published in, I don't see any alternative but to amend MEDRS to what I did above. If there are no objections (and I don't see why there would be, based on everyone's arguments for industry funded studies), I will make the change. LesVegas (talk) 20:26, 1 September 2015 (UTC)

You misrepresent the discussions that have been held here about Chinese publications about acupuncture. Those are about a specific topic and specific country, and the generalization you want to make here is not helpful. Your persistent efforts to portray this as a bias fall flat when you look at the publications about the general poor quality of acupuncture research publications in China. Jytdog (talk) 12:37, 2 September 2015 (UTC)

Any "problems" they may have are the same as seen in industry funded studies. You might want to familiarize yourself with these in order to discuss this material at substantive depth. LesVegas (talk) 23:02, 2 September 2015 (UTC)

I think it may be helpful to remove "funding sources": there's certainly been some questioning of cigarette/e-cigarette material where the publications are industry-funded, but I do worry that this question - essentially of whether a source is a WP:IS or not - is too tricky to codify in a few words, especially if we're going to get into questions about Chinese/Russian research e.g. - so would leave the change at that. It's also often struck me that the sentiment in the current MEDRS text sits uneasily with WP:FRIND for fringe topics. Alexbrn (talk) 13:08, 2 September 2015 (UTC)

Arthur, I appreciate your intellectual honesty and your desire for consistency here. I can see both sides to allowing industry funded studies on the article. On the one hand, correlations have been shown with industry funding bias regarding psychiatric medications failure to warn about cancer risk, and have shown a huge statistical increase in "positive findings" for various pharmaceuticals over those funded by other sources such as non-profits or the government. On the other hand, there could be other factors at play, and not all industry funded research isn't tainted so we can't throw the baby out with the bathwater, and so on. The same can be said for Chinese or Russian research as well. It's never been shown to be as simple as "fraud" or "lies". It could be that acupuncture performed on Chinese people by Chinese doctors in China has a more powerful effect than it does on unbelieving Westerners in the US when it's performed by caucasians. It could be that they perform the acupuncture stimulation differently, or use differently sized needles. It could also be that they conduct studies differently for ethical reasons, choosing a control group that receives medication instead of sugar pill, and that's why we see different results. If things like that are the case, it's much harder to dismiss it as simply unreliable. Russian research is also performed quite differently than US/European research as well, confounding the issues. I'm sure all around the world these things we see different results for different reasons. Ultimately, things just need to be consistent in our guideline here so that editors don't get confused. LesVegas (talk) 23:02, 2 September 2015 (UTC)

I added "country of origin" because I doubt editors here would want to remove funding sources in order to remain consistent, and I am also in that camp now. I really think there are far too many variables at play with these things and as Yobol said we ultimately can't throw out the baby with the bathwater. LesVegas (talk) 00:40, 4 September 2015 (UTC)

Your whole argument here is incredibly WP:POINTY. Jytdog (talk) 01:15, 4 September 2015 (UTC)

Thanks for the substantive argument. See, I knew you had it in you. LesVegas (talk) 12:19, 4 September 2015 (UTC)

On the merits of the proposal: We shouldn't add this. One of the important differences between "acupuncture in China" and "industry-funded research" is that we have a wealth of other sources about acupuncture, but for some subjects, there are no sources that are entirely independent of "industry funding". If we banned all sources from China, we could still write a decent article about Acupuncture; however, if we banned all sources that either received "industry funding" (broadly defined) or were based on sources that received industry funding, we could not write any articles about experimental drugs or new drugs. For example, the FDA just approved rolapitant, and I have just looked up all of the registered clinical trials. Every single one of them was sponsored by a commercial drug manufacturer. You will not find a source that provides biomedical information about rolapitant, that meets MEDRS's quality standards, and that does not use data produced through "industry funding". This gap is why the two cases (one subject in one country vs entire industries) are not comparable. WhatamIdoing (talk) 06:14, 4 September 2015 (UTC)

WhatamIdoing, as usual you make an interesting point here. Two problems I see: one, that we have no checks or balances on industry funded sources the way MEDRS is currently written. We have many independent studies on soft drinks, but research making health claims about soft drinks that's funded by Coca-Cola is still as allowable as anything else. There's no cutoff whatsoever. Ultimately, we just need some consistency, somehow. Two, have Chinese sources ever been proven inferior, biased or less reliable to Western sources on any subject? The critical literature looks pretty much the same with Chinese sources for acupuncture as it does for industry funded sources, although with better explanations for discrepancies in findings, such as different ethical standards the Chinese have, (for instance placebo controls aren't allowed in China because half the study group wouldn't receive treatment). So for those two reasons, there is a huge logical issue preventing consistent application of MEDRS here. We just need to fix it in some way. I'm open to whatever, but MEDRS needs to be remedied for universal consistency, somehow. LesVegas (talk) 12:19, 4 September 2015 (UTC)

Never feed the trolls.

I don't really understand what you mean that placebo controls aren't allowed–and I would very much like that sourced. Often placebo consists of current treatment which is then compared with a group receiving a new treatment – seldom is the alternative no treatment.-- CFCF 🍌 (email) 14:43, 4 September 2015 (UTC)

This is becoming tendentious. the sources discussing the poor quality of Chinese research on acupuncture (including self-reflective sources by Chinese authors) were discussed to death on this board not long ago. I don't want to be mean, but it is time to WP:SHUN this WP:POINTY and incompetent (e.g discussion of placebo) effort. LesVegas has been claiming systemic bias since his first month here - before he understood any of the policies and guidelines, letter or spirit, and LesVegas still does not understand what we are up to here. Jytdog (talk) 15:18, 4 September 2015 (UTC)

Jytdog, you should know to focus on the content, not the contributor. You calling me tendentious is highly KETTLISH. I have yet to see you focus on anything relevant whatsoever. I will respond to your personal attacks, one by one, and after that I will cease to respond here because it's not the forum. If you still have a bone to pick, please do it on my talk page. First, this subject was discussed to death here not long ago, here, but not in the context of keeping MEDRS consistent. I am suggesting we either have to change MEDRS's stance on carte blanche acceptance of funding-source tainted research, or allow research from all countries equally. Frankly, to maintain consistency, we cannot have it both ways. I'm not sure you actually understand WP:POINTY since I'm talking about being consistent rather than rigidly applying policies I disagree with consistently. I happen to understand many policies here better than you do, including systemic bias, because when I started editing Wikipedia I sought out training in the adopt-a-user program, which is something you never did. Since you raised issue with my competence for stating the fact that placebo controls are ethically problematic in Chinese research, I could do the same with you while pointing out that you've been editing Chinese articles in a highly opinionated, yet ignorant, fashion all along, but I wouldn't do that because I don't expect my fellow editors to be all-knowing. Again, I won't be addressing or even responding to your personal attacks again. I won't acknowledge anything but true content on article talk pages with you again, and frankly wouldn't have even replied to this here if it weren't for the fact that you have persistently attacked me on this talk page. Going forward, if you have a valid point about my suggestions, I would love to hear them. LesVegas (talk) 01:38, 5 September 2015 (UTC)

Do we allow sources that reference unpublished studies?

Do we allow reviews as sources for claims, in which some or most of the studies referenced in the review are unpublished and unavailable to the public? An example might be an FDA review which refers to many unpublished studies provided by the drug company, but not accessible to the public. SageRad (talk) 16:36, 30 August 2015 (UTC)

Conveniently, the sentence I reference above is applicable here, "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions." (emphasis mine). Yes, sources that reference unpublished data can be used. High quality sources such as the FDA should not be dismissed because editors themselves do not have access to the same data as the source. Yobol (talk) 16:42, 30 August 2015 (UTC)

Hmm. Thanks. I find that disturbing, personally, as it flies against all notions of transparency and peer-review which are supposed to be the basis of resiliency and error-correction in science. SageRad (talk) 18:29, 30 August 2015 (UTC)

{e/c} On the contrary, published data often have significant biases (see publication bias), so the addition of unpublished data to a secondary source such as a meta-analysis is often more neutral and accurate than only using published data. Either way, the use of unpublished data has no real relevance to the reliability of a source, per WP:MEDRS. Yobol (talk) 18:37, 30 August 2015 (UTC)

I question whether the FDA is a "medical" organization as per WP:MEDORG. SageRad (talk) 18:32, 30 August 2015 (UTC)

Of course it is, just as much as the CDC or any major governmental medical/scientific body. Those who dispute the conclusions of the FDA/WHO/CDC often dispute whether the FDA/WHO/CDC are reliable, but that says more about the biases of the individual editor and their dispute with the conclusion of the major medical body than it does about the reliability of the medical body themselves. Yobol (talk) 18:37, 30 August 2015 (UTC)

There was an authenticated letter sent by a group of FDA employees in 2010 that read in part:

... many other FDA managers who have failed to protect the American public, who have violated laws, rules, and regulations, who have suppressed or altered scientific or technological findings and conclusions, who have abused their power and authority, and who have engaged in illegal retaliation against those who speak out, have not been held accountable and remain in place. (CBS News)

I have a problem with trusting governmental agencies, especially when they have a history of corruption and are using information that is not available to the public to pass judgments. This article in JLME also speaks to the regulatory capture of the FDA. I am concerned about trusting a source that other reliable sources say is not reliable. I'd say it should not be viewed as flawless or without reproach, and we should not place blind trust in the FDA especially when their primary sources are unpublished and therefore not peer-reviewable. SageRad (talk) 20:07, 30 August 2015 (UTC)

Wikipedia is not a place to WP:RIGHTGREATWRONGS. The FDA is very much part of the scientific establishment that is the mainstream of scientific/medical thought in the world. Jytdog (talk) 20:10, 30 August 2015 (UTC)

(e/c) There are no publisher of sources that are "flawless" or "without reproach", and you can find examples of almost all high quality sources having some sort of black mark(s) in their record. In general, the FDA is a reliable source for medical information especially as related to their field of expertise (food and drug regulation). We are probably going past what the role of this talk page is for now, so unless there is a particular change you would like to suggest here, it's probably best to leave specific discussions of specific sources to the talk page of the article in question or at WP:RSN. Yobol (talk) 20:13, 30 August 2015 (UTC)

I'd like to be able to discuss general matters of this sort without being shut down. Please stop the shutting down of conversations. And i am not "righting great wrongs". Let us be able to speak about generalizations, with others, and let it stand for a while. Let's allow people to speak here. I'm completely serious in querying the reliability of the FDA and similar sources that use unpublished data to reach conclusions that are then represented as truth. I understand that no source is "perfect", but there is a built-in mechanism that enables peer review for most sources that we would accept as reliable. In other words, most reliable sources are not typically relying on inaccessible data for their reportage. SageRad (talk) 20:16, 30 August 2015 (UTC)

Countercultural conspiracy theories don't travel too far in discussions about policies guidelines. WP is mainstream and conservative when it comes to sourcing and institutions that produce them. By "conservative" i don't mean right wing, I mean trusting the institutions that produce and disseminate scientific knowledge. Jytdog (talk) 20:23, 30 August 2015 (UTC)

(e/c) I get that you believe you are right, and I'm just expressing my own view that attempts to get the FDA disallowed as a reliable source of Wikipedia would be an incredible waste of time, as the vast majority of medical sources and medical expertise recognizes the FDA as a highly reliable source. Best of luck trying to get a consensus around the dubious proposition that it can't be used as a source here. Yobol (talk) 20:29, 30 August 2015 (UTC)

I posed a question, a topic for discussion, on the topic that this page is about. If you have a problem with that, then don't take part in the conversation. Please don't use such a tone as "i get that you believe you are right".... what does that even mean? And i am not attempting to get the FDA disallowed as a source. I am questioning the mechanics of some sources that are accepted as secondary sources, when they are not peer-reviewable, rely on secret documents, and have a documented history of corruption. SageRad (talk) 20:42, 30 August 2015 (UTC)

Please show me any human institution that is free of corruption. Just one. Instances of corruption =/= rotton to the core, and you seem to be proposing the latter, which is just conspiracy theorizing. In general WP trusts the knowledge-producing institutions of our society. Jytdog (talk) 20:48, 30 August 2015 (UTC)

Of course, you're correct that no institution is inherently free of corruption, but that's a red herring because the point i was making was that there is a difference when a source uses inaccessible primary sources to reach a conclusion, and when that source has a documented history of corruption. It's not a good combination. We do need to think critically about credibility of sources, and i would hope there are no sacred cows. This is the place where we talk about general principles about how we decide what sources to trust. We cannot trust all "knowledge producing institutions" equally because they do not merit equal trust. That is why we have guidelines about what is a reliable source and what is not, and why. SageRad (talk) 20:53, 30 August 2015 (UTC)

This talk page is for the concrete discussion of improvements to this page, not for a general philosophical discussion. What is your concrete suggestion to improve this guideline? You've spent a lot of time trying to belittle the FDA but then say you don't want it disallowed as a source. What exactly is your concrete proposal to improve this guideline? Yobol (talk) 21:02, 30 August 2015 (UTC)

Where does it say that? Who made that rule that this talk page is only for concrete proposals and not discussions of principle? Seriously, i've seen several very interesting discussions get shut down in the last couple of weeks here, and i don't like that. Sometimes we need to have a general discussion to arrive at concrete discussions.

My concrete suggestion if i were to propose one, is to state that secondary sources that are peer-reviewable are preferable to sources that rely heavily on unpublished data. SageRad (talk) 21:07, 30 August 2015 (UTC)

You might try reading the very first sentence at the top of this page, which states, "This is the talk page for discussing improvements to the Identifying reliable sources (medicine) page." Discussion should be centered around improving the page, not just a general philosophical discussion about sources (that belongs on user talk page, talk pages of specific articles for discussion of specific sources in context, or on RSN). As to your proposal to deprecate sources that use unpublished data, I will oppose strongly based on my first comment above, which is that the published data is often biased, so better sources can be the one using unpublished data. I should also note that I find your determination that papers that use unpublished data are by definition not "peer-reviewable" as bizarre, but I think I've wasted enough of my time on this thread already, and will take my leave. Yobol (talk) 21:14, 30 August 2015 (UTC)

Nobody is requiring you to take part in this discussion. I'm interested to hear what others think as well. I do believe that this question is a discussion on possible improvements to the guideline. I don't see how a paper that cites unpublished data is reviewable by anyone who doesn't have access to the underlying data. I'm puzzled by that. SageRad (talk) 21:26, 30 August 2015 (UTC)

I would say that you have it exactly backwards. Reviews should absolutely take into account unpublished studies. If they do not then due to publishing bias they have a real chance of overestimating any effect. Also I see earlier you are referring to unpublished studies as "secret documents". There is a world of difference between difficult to access because no one wanted to publish it and a secret document. 2601:645:C201:3840:88CF:A3D4:4EFF:F687 (talk) 03:45, 31 August 2015 (UTC)

• Answer is yes Doc James (talk · contribs · email) 21:15, 30 August 2015 (UTC)
• Comment I think that is a very interesting question, and relates directly to WP:RS and transparency of RS: So far Wikipedia doesn't care. prokaryotes (talk) 23:48, 31 August 2015 (UTC)

Yes. We expect the authors of reliable sources to determine which sources they use, many of which would fail rs for Wikipedia articles. For example in writing about a figure in the news, they may rely on what that person, his or her friends, enemies and acquaintances have said about them, high school year book stories, etc. It is similar to what a court does. They rely on the evidence of people whose testimony does not meet rs and determine the facts. TFD (talk) 01:26, 4 September 2015 (UTC)

In fact, we go further than the question asks: we not only do not require our sources to cite published studies, we do not even require them to cite any sources at all. WhatamIdoing (talk) 06:01, 4 September 2015 (UTC)

Add image to WP:MEDASSESS

Wonder what peoples feeling are about adding this type of illustration to the MEDASSESS section. It shows some different ways to balance evidence. I've recently had the feeling that many new editors are having trouble understanding why we use one article over the other, and this would perhaps make it easier to explain why we chose secondary over primary sources.

There are different ways to rank levels of evidence in medicine–they are similar in that they put high level reviews and practice guidelines at the top.

Please give your thoughts, and I can also prepare more of these images for different pyramids as well. Best, -- CFCF 🍌 (email) 09:20, 3 September 2015 (UTC)

very clear and informative, I think MEDRS would benefit from this/ posted talk/Med--Ozzie10aaaa (talk) 20:09, 3 September 2015 (UTC)

• Interesting. The pyramids differ on some important points about ranking sourcing. Let's use both, say where they are from, explain that conflicting advice is the norm, and also look for other differing views. Something that stands out to me is the idea that "clinical practice guidelines" are more authoritative than systematic reviews. Obviously these become outdated as compared to the latest systemic reviews, and sometimes they are not even evidence based, but I really like that one of the pyramids shows this as the highest authority because in many ways, CPGs are. Thanks for making more than one.

I wonder if we could have one that is "WikiProject Medicine's Ranking of Priority", and dictate to the world how information ought to be evaluated. We use different terms and emphasis. Blue Rasberry (talk) 18:04, 4 September 2015 (UTC)

Our "ranking" depends upon multiple factors. The "pyramid" would look a lot like the table at WP:CANVAS: several unrelated criteria, and being "strong" on Criteria #1 might not outweigh being "weak" on Criteria #2. WhatamIdoing (talk) 18:43, 4 September 2015 (UTC)

I added references in the captions to the images, but I don't really know how to best explain their differences and rationales. I also like the pyramid at [2] which makes a distinction between meta-analysis and systematic reviews. Unfortunately it has too many steps, so I'd have to make a new image for those.

As response to Bluerasberry – CPGs should ideally be at the top, but that isn't always that case. WhatamIdoing – what different criteria do you propose for such a table. I surmise you mean something like this:

Higher quality
	↕		↕		↕		↕
Lower quality

-- CFCF 🍌 (email) 23:52, 5 September 2015 (UTC)

'

References

Request for Comment: Country of Origin

Should we change MEDRS, which currently reads:

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.

to

Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, country of origin, or conclusions.

This proposal is to address only the addition of high-quality sources into the guideline. LesVegas (talk) 23:33, 5 September 2015 (UTC)

Comments

Support Wikipedia should never get into the business of discriminating against high quality sources because of where that research might have been conducted or published. However, this has happened before, where multiple Cochrane reviews were rejected while trying to replace an old source with them, partly on the basis that "all but one of its authors were Chinese," and therefore, they must be biased. This is not an isolated incident either, with some editors rejecting all sources published in China not on the basis of case by case analysis of the source's quality, but stating they're published in China and assume they must all be unreliable. Low quality sources that, for instance, are not peer-reviewed, should always be rejected. But MEDRS does not yet make it clear whether or not high-quality foreign sources should ever be rejected on the basis of country of origin, whether it be the authors of a publication or place where high quality research is published. And interestingly, MEDRS currently allows for industry funded research not to be rejected on the basis that it's industry-funded, while it says nothing about a particular country's research. And yet, despite whatever potential problems could exist, it is very clear that industry funded research can be very problematic to add into Wikipedia. Just recently, Coca Cola came under fire for funding scientific research showing that Coca Cola doesn’t cause obesity. And bad as that may be, MEDRS currently doesn’t allow us to prohibit such research. It states:

'Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions.'

There are also many well-documented problems with industry-funded research in the way of psychiatric medications failure to warn about cancer risk with certain drugs, as well as showing a huge statistical increase in "positive findings" for various pharmaceuticals.

However, just because there is a documented problem with some industry funded research, that doesn’t mean there is a problem with ‘’’all’’’ industry funded research. And oftentimes, new drugs only have industry-funded research for sources. No other sources can be found. Reasons like that are why it's not permitted to reject high quality sources on the basis of them being industry-funded.

At the same time, some editors currently reject some sources on the basis of them being published in another country. One such example is with acupuncture where a 1998 research report showed that in Hong Kong, Taiwan, Russia/USSR, China and Japan higher reports of positive findings occurred when compared to England. In some ways findings like these could possibly mirror the problems in industry-funded research. And at the same time, as with industry-funded research, it’s difficult to discount all of it on the basis that there is a chance some of the more positive findings might possibly be due to publication bias. Further complicating the issue is that more positive findings might be due to cultural differences in scientific research between various countries, such as ethical dilemmas with using placebos where the Chinese have shown statistical deviations from non-Chinese trials, creating conditions where placebo alone is not ethically justified as a control. There are many other possible confounding factors similar to this which may explain differences for findings between various countries once found within this specific modality. Undoubtedly, this seems to cloud the issue. But making the issue more problematic, for us as editors, is that some of the research mentioned in the Vickers source might not even be of high quality, thus causing making any argument against high quality research moot.

Adding "country of origin" seems to put the wording more in line with Wikipedia’s project Countering Systemic Bias, as well as our WP:BIAS policy. By ensuring Wikipedia editor-level peer review doesn’t extend to country of origin, it also makes it consistent with our current stance on liberal allowance of sources no matter what types of funding may be received.

LesVegas (talk) 23:51, 5 September 2015 (UTC)

Oppose I see no reason why this proposal is needed. We could also add that we shouldn't oppose inclusion based on authors skin color–because racism is bad, but it isn't needed. Level and quality of evidence is far more important. -- CFCF 🍌 (email) 00:56, 6 September 2015 (UTC)

There have been incidents where high quality sources were rejected for reasons that could've been prevented with amended language, such as this. I modified my first comment, adding in one example of such a rejection, although there have been others. Therefore, it appears it is needed. Let me know if you still think this way after reading the link. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Question What does "country of origin" in the proposal refer to? The country where the study was conducted, nationality of researchers or subjects, or country where the publishers of the journal or book are located? The first two are fine but perhaps unnecessary as CFCF says above (can LesVegas point to instances where such criterion was cited and prevailed?) If the last, I'll be opposed to the change since assessing the quality of journal and its publishers is crucial for assessing medical and other sources, and neither high-quality nor sham publishers are distributed uniformly across the globe, and we on wikipedia cannot solve this real world problem by simply turning a blind eye to it. Abecedare (talk) 01:04, 6 September 2015 (UTC)

Good question. "Country of origin" is all-encompassing, meaning country of author, country where research was conducted and where it was published. Keep in mind, this RfC is asking about high-quality peer-reviewed publications that meet every other standard of quality in MEDRS. LesVegas (talk) 03:16, 6 September 2015 (UTC)

• Comment This appears to be an attempt to allow Chinese studies about acupuncture published in Chinese journals to be used to support acupuncture. We need to use high quality journals with a reputation for reliability. So oppose the underlying attempt. Doc James (talk · contribs · email) 01:27, 6 September 2015 (UTC)

Actually, there was an incident where an influential editor opposed the addition of a Cochrane Review because its authors were from China. DocJames, please limit your comments to high quality sources. This is what the discussion is pertaining. LesVegas (talk) 01:52, 6 September 2015 (UTC)

To which Cochrane review do you refer? Doc James (talk · contribs · email) 04:05, 6 September 2015 (UTC)

Actually, several. One of them was a review on fibromyalgia which had several authors that were Chinese. High quality sources should never be rejected on such a basis, don't you think? By the way, upon re reading your comment, are you suggesting Chinese studies or Chinese journals are all inherently low-quality? I certainly hope that's not what you were saying. Tell me I read that wrong. LesVegas (talk) 04:24, 6 September 2015 (UTC)

Yes that Cochrane review is a fine source to use. No I am not saying all Chinese journals are inherently low-quality. Many journals in many places however are low quality. Doc James (talk · contribs · email) 04:29, 6 September 2015 (UTC)

• Agree with Doc James. I have seen zero examples of any attempt to reject high-quality foreign sources on the basis of simply being from another country. The Chinese studies about acupuncture published in Chinese journals were rejected not because they were published in China, but rather that they were not high quality journals with a reputation for reliability. There are plenty of fringe journals pushing pseudoscience in the US and UK, and plenty of good, peer-reviewed science gets published in China. --Guy Macon (talk) 01:48, 6 September 2015 (UTC)

Guy Macon , you might check out the diff below where an editor rejected one such source on this very basis. You can't get much higher quality than Cochrane Reviews, and you can't get much lower editing than to reject one all because "all but one of its authors are Chinese." It's sad we have seatbelt laws and equally as sad we have to tell editors not to reject high quality sources because of things like where they're published, but I'm afraid that's the case. It's exhausting to waste time on such silly matters and doesn't hurt anything to add it in because we're talking about high quality, peer reviewed sources here anyway. LesVegas (talk) 02:34, 6 September 2015 (UTC)

I did check out the diff. As others have pointed out, it does not say what you claim it says. --Guy Macon (talk) 05:39, 6 September 2015 (UTC)

Guy Macon, I've seen you around and know you're a good editor here. I know it's shocking to think another editor would ever argue such a thing, but I was in the middle of that one and I can tell you that's what was meant regarding a set of Cochrane Reviews used to replace old research. Kww argued it shouldn't be used to replace an outdated review per WP:MEDDATE for several reasons, one being that the Cochrane Reviews used Chinese authors. If you would like more details on that, I'm happy to provide them. Again, I know it's hard to believe editors would ever say or argue such a thing, but sadly they do. The other Guy constantly rejects sources because they're Chinese, and you can see for yourself how he stereotypes them as universally unreliable. No regard for peer review in statements like that, just outright rejection of sources because they are Chinese. We shouldn't even have to discuss such matters as ethnicity of authors or place of publication on talk pages, it's distracting and disruptive; editors need to be focusing on quality of individual journals and that's it. LesVegas (talk) 06:09, 6 September 2015 (UTC)

• Comment Pending further info Is this change meant to address a non-hypothetical problem? Excluding a study based solely on country of origin is so strange that I'd be surprised to see that anyone had attempted it, let alone done it successfully, but has it happened? What were the circumstances? When I first saw this RfC, I thought "This looks like it's probably about something else" and DocJames' scenario seems consistent with this. Other comment: What counts as "personal"? Darkfrog24 (talk) 02:07, 6 September 2015 (UTC)

Darkfrog24, you might want to check a diff out where an editor opposed updating an old source with a series of Cochrane Reviews because all but one of its authors were Chinese. Yes, rejection of high quality sources based on country of publication or origin of its authors is a real problem. As long as sources are high quality, issues like country of origin or ethnicity shouldn't matter. LesVegas (talk) 02:28, 6 September 2015 (UTC)

Well this diff doesn't show someone saying, "Exclude these sources because they're Chinese." It's more like Kiwi saying, "I think this guy wants to exclude the sources because they're Chinese," but whatever. I'm changing my comment to mild support. The idea that we shouldn't exclude sources solely on country of origin is so obvious that we all should be doing it already and if a source really is unreliable, there should be other reasons to exclude it. Darkfrog24 (talk) 03:39, 6 September 2015 (UTC)

+support based on the diff linked above, where an editor simply assumed that a study was biased because other studies by other authors of the same nationality had been accused of bias. This is out of line. DES ^(talk) 03:17, 6 September 2015 (UTC)

Darkfrog24 and DES what you see operating in the diff is WP:REDFLAG. The key word in the diff is "suspect"; which is very different from "excluded". Jytdog (talk) 13:50, 6 September 2015 (UTC)

The key phrase in the diff is "I also note that of the supposed refutations, all but one include Chinese researchers on the papers: we've discussed that bias problem extensively, and there's no reason to believe it doesn't apply here" This is drawing a conclusion that the mere presence of Chinese researchers equates to bias. This is not acceptable. DES ^(talk) 13:57, 6 September 2015 (UTC)

DES, thanks for replying. LesVegas and others have been pushing and pushing and pushing to include Chinese sources on acupuncture in our article - the issue has been discussed a zillion times but keeps coming up, and it gets frustrating, and people write things quickly that are easy to misconstrue when diffs are cherry-picked, as LesVegas has done with KWW's statement. Jytdog (talk) 14:07, 6 September 2015 (UTC)

User:Jytdog, would a rule saying, "do not exclude based solely on country of origin" make it harder for you to exclude low-quality or otherwise problematic sources? I'm not familiar with this particular dispute, but it sounds like those sources are no good regardless, so this shouldn't make any difference. Would it help if another line were added elsewhere saying, "Sources that have not survived peer review may be excluded" or "sources from publications known to routine publish and fail to retract unreliable material may be excluded"? Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

The background facts are:

a1) our article on Acupuncture has been highly contested for a long time and is subject to DS per Arbcom - the core dispute is between science-based editors and believers in acupuncture (as with many alt-med topics);

a2) some acupuncture believers actually try at some level to deal with the relevant Wikipedia policies and guidelines (NPOV and particularly PSCI, FRINGE, MEDRS);

b1) Research (and reviews) produced in China about acupuncture (and other TCM) overwhelmingly come to positive conclusions about the efficacy of TCM for various diseases and conditions, and they come to positive conclusions way more often than research and reviews produced in the West.

b2) This discrepancy is explained by believers (in published sources) in various ways (e.g Westerners don't do TCM properly or there are special facts about TCM that make it impossible to conduct sound scientific research on it); scientists have explained this discrepancy in published sources by pointing to the immature level of development of Chinese scientific institutions.

c1) When you put a) and b) together, you get some acu-proponents pushing to use reviews originating from Chinese scientific institutions to support content in our article about the benefits of acupuncture (content that is not supportable with, or even contradicted by, other sources), and science-based editors tending to reject that content and its sourcing - especially when no other sources are brought (see the first bullet of WP:REDFLAG). This has been discussed extensively on this Talk page as well as on the article Talk page and the consensus has been to treat sources about TCM originating from China as "suspect" for the time being - not excluded, but not "green flagged" as high quality. This RfC was posted by an acu-proponent. You can see my response to it below. Jytdog (talk) 15:05, 6 September 2015 (UTC)

Jytdog, what is your opinion on rejecting Cochrane Reviews because its authors are Chinese? Are there no publications in China that are reliable? Do you have a source that all are unreliable and engage in publication bias? Do you think there could be other possibilities for differing results, such as ethical dilemmas with placebo controls in more instances? And since scientists have documented the same issues you complain about in Chinese research, as with industry-funded research, why do you hold a different standard for Chinese sources? LesVegas (talk) 15:22, 6 September 2015 (UTC)

• oppose The framing of serious issues with Chinese research publications on acupuncture as "discrimination" is the worst kind of red-herring, bad-faith, manipulative argument imaginable, especially when Chinese scientists themselves point out the problems. This RfC is the pits. There are solid sources describing the problems:
  • Li J, et al The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012. BMC Complement Altern Med. 2014 Sep 26;14:362. PMID 25256890
  • Vickers, Andrew (April 1, 1998), Do certain countries produce only positive results? A systematic review of controlled trials., Control Clin Trials {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Ferguson, Cat (November 26, 2014), Publishing: The peer-review scam, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)
  • Qiu, Jane (January 12, 2010), Publish or perish in China, Nature {{citation}}: Italic or bold markup not allowed in: |publisher= (help)

The quality will surely improve with time but LesVegas' effort to wave a magic wand and make the actual problems disappear under the banner of "discrimination" runs hard against everything we try to do at WP:MED with regard to using high quality sources. Jytdog (talk) 04:37, 6 September 2015 (UTC)

Jytdog, this RfC is about high-quality sources, not low quality ones which are clearly not allowed. But your inability to differentiate between these two categories and continue stereotyping all Chinese studies and Chinese authors the way you just did isn't surprising since you posted that scientific institutions in China aren't independent or mature, and therefore all of their journals are unreliable. Frankly, comments like that are why RfC's like this exist. But back to the discussion at hand. You never addressed high quality sources, which is what this RfC is pertaining to. You are opposing low quality ones, which we all should oppose on the basis of things like lack of peer-review and things of that nature. But opposing them simply because they are Chinese isn't acceptable, don't you think? LesVegas (talk) 04:57, 6 September 2015 (UTC)

You continue to mischaracterize the situation and to employ circular reasoning. The systemic problems with Chinese research on acupuncture are well documented and so the assumption going in, is that they are not high quality; per Guy's diff they are "suspect." The burden is on the person bringing any such source to show that the specific source is high quality. Per WP:REDFLAG one way that could be done would be to show that sources that are high quality say the same thing. This is not racism, it is dealing with immature level of development of Chinese scientific institutions, just like it is not racist to say that the institutions of Chinese law in the fields of environment and intellectual property are not mature. The air quality in Chinese cities is often poor; copyright infringement is rampant in China; Chinese publications on acupuncture almost always have favorable outcomes. Those are simple statements of fact, reflecting a society in development. They are not essentialist, racist statements - they are existential statements of fact and will change over time, as the situation in China changes. So WP:DROPTHESTICK. And I suggest you be much more careful in deploying this tactic of cherry-picking diffs and trying to discredit editors as racist. Jytdog (talk) 13:50, 6 September 2015 (UTC)

Jytdog, just the other day you said "we do not perform peer review on sources and should never reject industry-funded sources that almost always have positive findings. But Chinese research? Nope, Chinese scientists aren't independent or mature. I'm not trying to characterize you as racist, but frankly by defending one standard for industry-funded studies, and enforcing another for Chinese research while stereotyping all Chinese research as unreliable, never to be evaluated on a case-by-case basis, makes it difficult for me not to find a diff that characterizes you any other way. But let me allow you the opportunity to prove that you're not racist. Simple question: why should we allow one standard for industry funded research despite documented publication bias issues, and another standard for Chinese research, rejecting the notion that we should evaluate Chinese sources or Chinese authors on a case-by-case basis like we do everywhere else? LesVegas (talk) 15:01, 6 September 2015 (UTC)

Your mischaracterization of what I wrote and of MEDRS is malicious, incompetent, or both. This is not about conducting peer review, which involves critiquing the quality and extent of the experiments that were done, the presentation of the resulting data, and the conclusions drawn from the data. This has nothing to do with research funding. And I said nothing about "Chinese scientists" - I addressed institutions. Jytdog (talk) 15:14, 6 September 2015 (UTC) (strike; should not have written that. my apologies. Jytdog (talk) 15:48, 6 September 2015 (UTC))

Oh but it is about conducting a peer review based on industry funding. Those have the same documented issues you complain about in Chinese research, although I would argue, worse, since there are more confounding factors between Eastern and Western cultures (like different scientific ethics) and those can explain reasons for discrepancies in results between East vs. West, not simply publication bias. Simple question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? LesVegas (talk) 15:33, 6 September 2015 (UTC)

You do not seem to understand what happens during peer review of scientific papers; I explained that above. Jytdog (talk) 15:48, 6 September 2015 (UTC)

Jytdog, this is not about peer review in the publication process. I hope you aren't WP: IDHT'ing Yes, scientific papers are peer reviewed before being published in reputable journals. But the peer review we are talking about is that "Wikipedia editors should not perform peer review", specifically, in this instance, by rejecting high-quality sources on the basis of industry funding behind piece of published research. Again, please answer the question: why should we not reject research on the basis that it's industry-funded, but we should reject research because it's Chinese, instead of evaluating it on a case by case basis? I really hope you can answer it. LesVegas (talk) 16:09, 6 September 2015 (UTC)

We already do evaluate on a case-by-case basis, and since this hasn't been a problem we don't need a clause against it in our guidelines. In order that they be followed–guidelines need to be succinct. We can't indiscriminately add clauses in order to address hypothetical problems or noone will read them–making it far harder to police.-- CFCF 🍌 (email) 05:44, 7 September 2015 (UTC)

On the face of it, the OP appears to want to get his low quality pro-acupuncture sources into our Acupuncture article by claiming they are high quality. I don't think the community is quite that silly. -Roxy the dog™ (Resonate) 10:28, 6 September 2015 (UTC)

Previously uninvolved editor here: 1) Even if the sources were rightly excluded, "they're Chinese" should not be numbered among the reasons for this; their own failings should be enough. Question: Why would "don't exclude based on country" make any difference? 2) Aside from this acupuncture issue, is the argument of exclusion-based-on-country a common enough problem to merit explicitly banning it? Do the words "don't exclude based on country" earn their keep for the space they take up (in Wikipedia's already Byzantine rule structure). Darkfrog24 (talk) 14:34, 6 September 2015 (UTC)

"They're Chinese" is a poor shorthand for a complex discussion and is not how the issue should be discussed nor should that phrase be used as a description of the stance of those opposed to this RfC or who treat sources about TCM produced by Chinese institutions as suspect. I am unaware of this issue arising outside the context of TCM. Jytdog (talk) 15:19, 6 September 2015 (UTC)

I can be more succinct than JD. 1)No. 2)No (to my knowledge). 3)No. -Roxy the dog™ (Resonate) 15:38, 6 September 2015 (UTC)

Oppose. While I agree that certain fringe journals are focused in particular countries, I don't think we should be determining reliability overall by country of origin. However, that can be a first red flag on certain topics for a deeper look. That's why I would oppose this language as I don't think it is needed. If a question on reliability comes up, there will be other qualities we look at for journals as well. There is also potential this language could be abused from a WP:BEANS approach (acupuncture does come to mind), so I think it's better that this is one thing left unsaid, but consensus in discussions on the idea that country or origin alone can't be used can always be linked if someone brings it up. Kingofaces43 (talk) 16:35, 6 September 2015 (UTC)

Strong oppose The focus should remain on the quality of the research and the reliability of the source. Either the research is good, or it is not. Either the source is reliable, or it is not. In what possible way does "country of origin" factor into anything, except to imply that some countries are doing poor quality research and publishing in unreliable sources, and must therefore be held to a lesser standard? TechBear | Talk | Contributions 20:39, 6 September 2015 (UTC)

I agree with you that the focus should always be on the quality of the sources themselves. In MEDRS it states, "Do not reject a high-quality type of study due to personal objections to the study's inclusion criteria, references, funding sources, or conclusions". We aren't supposed to reject sources purely on those merits, just simply focus on the quality of the source itself, and yet MEDRS has to be explicit because I can only assume in the past editors have rejected high quality sources because of things like references, inclusion criteria, etc. Given that editors have rejected clearly high quality sources on the basis of nationality of author or country of publication (as seen from the diffs I provided) why haven't we reached a point where we need to explicitly tell editors they need to be focusing only on source quality and nothing else, like country? LesVegas (talk) 22:22, 6 September 2015 (UTC)

Call for close

This thread should be closed as POV pushing and trolling. When Jytdog wrote "When the institutions that support science in China become independent and mature, their journals will become valuable sources. They are not there yet"[3], only to have LesVegas characterize it as " Chinese scientists aren't independent or mature"[4] -- something Jytdog never wrote or implied -- it became clearly evident that LesVegas is more interested in winning the argument than seeking the truth. Demonizing Jytdog as a racist is trolling, and we should close down this discussion rather than rewarding such behavior with further attention. --Guy Macon (talk) 20:00, 6 September 2015 (UTC)

I find it a little odd that nobody seems to want to answer why we have one standard for industry funded sources (written into MEDRS), and an entire other standard for Chinese sources where we can reject them on the basis of being Chinese and not on their own merits. Nobody has dared answer that question, and here, less than 24 hrs after the RfC was filed, an attempt has been made to close it before anyone answers this obvious and frankly embarrasing inconsistency. How paradoxical to then accuse me of not trying to seek the truth. And please don't accuse me of demonizing Jytdog as racist. He accused me of accusing him of racism, which is not the same thing at all. He said I was finding diffs that made him look racist and I told him essentially that was his problem, not mine. As for institutions, are they faceless, humanless entities or are they made up of human beings? So I suppose the institutions can be immature or not independent, but the scientists and publishers composing the institutions are mature and independent? Really? Especially when Jytdog and other editors are on record rejecting sources because they're Chinese, and not because of, say, lack of peer-review or lack of Medline indexing. If you read Jytdog's many diffs in full context he makes it clear it is impossible for anything Chinese to ever be considered reliable. I have not once seen him state, even here, otherwise. And at the same time he defends the practice of not rejecting sources because of industry-funding that has been documented to skew results, but opposes all Chinese research on acupuncture for those same reasons. I have asked multiple times for an answer, and he ignores it. Please, somebody, can somebody answer it? If we can come to a consensus on how to reconcile that glaring issue, and we can come to a consensus that high quality sources shouldn't be rejected on the basis of ethnicity of author, nor should they be rejected purely because of where they're published, I'm fine with that. When editors reject friggin Cochrane Reviews because authors are Chinese, we have a serious problem. LesVegas (talk) 22:11, 6 September 2015 (UTC)

I'm reminded of MastCell's WP:CGTW#8: Anyone who edits policy pages to favor their position in a specific dispute has no business editing policy pages. Corollary: these are the only people who edit policy pages. Yobol (talk) 22:16, 6 September 2015 (UTC)

• Support close This RfC is an obvious attempt to win a content dispute by declaring that editors objecting to certain sources are racist. Johnuniq (talk) 23:45, 6 September 2015 (UTC)

Oppose close - I do not think it is appropriate that everyone is focusing on the editor instead of answering the basic question this thread is asking - some of those opposed to the policy edit proposed asked some version of "why would country of origin matter?" Well, exactly, it shouldn't matter. And yet sources are rejected because "they're Chinese" and that is wrong. If there were reliable ways of gauging the quality of the research, each review can evaluated on its own merit, as it should be. The fact that the authors are Chinese shouldn't even come up, but it frequently does. So, we have a problem.Herbxue (talk) 14:12, 7 September 2015 (UTC)

Note to closer: Herbxue also has argued for using sources stemming from Chinese institutions in acupuncture articles and claimed discrimination with regard to applying REDFLAG to them Jytdog (talk) 14:50, 7 September 2015 (UTC)

Actually read that link again: I say "fine" to rejecting Chinese journals that do not have a reputation for quality, what I call "outrageous" is rejecting a review in a mainstream journal because the authors names are Chinese, and yes that actually occurred, and thus we have a true problem.Herbxue (talk) 15:25, 7 September 2015 (UTC)

• Support close Despite what our resident acupuncturist says, we should not use unreliable sources, (which is what they are) wherever they come from. -Roxy the dog™ (Resonate) 14:28, 7 September 2015 (UTC)

• This might have been an interesting RfC and it asks some very real questions in terms of sources. I would have been interested in pursuing the ideas presented here from all sides however, once again, discussion is shut down with name calling and personal accusations so that mature discussion is thrown out.(Littleolive oil (talk) 14:35, 7 September 2015 (UTC))

Note to closer: Littleolive oil also has argued for using sources stemming from Chinese institutions in acupuncture articles and per this agrees with Herbxue who commented above Jytdog (talk) 14:50, 7 September 2015 (UTC)

Thank you for including a link which belies what you mean to be an accusation and attack and links to my cmts which you paraphrase inaccurately.(Littleolive oil (talk) 15:17, 7 September 2015 (UTC))

• • I am becoming increasingly concerned with Jytdog's constant attacks and his inaccurate representations of editors and what they are saying and doing.(Littleolive oil (talk) 15:28, 7 September 2015 (UTC))

Mid Quality sources

I have noticed a trend in the last few threads... debates that focus on defining the source in question as either "low-quality" or "high-quality"... and not considering whether it might be "mid-quality". I think we all agree that high-quality sources are preferred, and that low-quality sources should be avoided. But what about mid-quality sources? What often happens is that supporters of the mid-quality source try to elevate it to high-quality status... while opponents try to demote it to low-quality status. I think it important to assess sources realistically... so perhaps we need some discussion on how to present mid-quality sources. Blueboar (talk) 12:33, 6 September 2015 (UTC)

I think the reason we don't speak of mid-quality sources is because there of is such a divide between the low and high quality sources. It might seem confusing but what we are saying is basically that we accept high quality and top of the line Cochrane-level sources. Mid-quality sources would be primary sources such as RCTs and well-performed single studies–check out the pyramids I added above. We don't discuss these so much because Wikipedia doesn't allow them (or at least strongly discourages them). -- CFCF 🍌 (email) 21:51, 6 September 2015 (UTC)

You know, this is an interesting query well worth exploring. I think what we'd first have to do would be define what a mid-level source is. Is it peer-reviewed, but not in a highly well-known journal like JAMA? Does impact factor have any bearing on level? Or is it less about the publication and more about the type of study? So are meta-analyses superior to systematic reviews for identical claims? Do higher level sources use established reporting standards for their field? I would think that despite what mid-level sources are, we might still allow them but give them less weight obviously excellent sources. What that would probably mean for the article is that we should only use high-level sources in the lede while mid-level sources could be used elsewhere. High level sources might ought to be listed first in a paragraph, while mid-level sources would come later. All of that would, of course, be dependent on what we agree mid-level sources even are. LesVegas (talk) 22:39, 6 September 2015 (UTC)

This is the same idea that you have proposed already at the talk page of the acupuncture article and has been rejected. See Talk:Acupuncture/Archive_28#Chinese_Source_Idea and Talk:Acupuncture#The_Chinese_question Jytdog (talk) 00:11, 7 September 2015 (UTC)

Its appropriate for an editor to bring comments and questions to the larger community especially since in this case LesVegas is responding to Blueboar above (Littleolive oil (talk) 01:30, 7 September 2015 (UTC))

You also argued in those threads for inclusion of Chinese sources on acupuncture in some way like this. I do understand that both and LesVegas and likely Herbxue are pleased to find an opening in what Blueboar wrote above, for what you already wanted; likewise it is reasonable that Blueboar should be informed of the conversation into which he/she is walking Jytdog (talk) 14:54, 7 September 2015 (UTC)

Once again your response is a personal attack. No, I did not argue for inclusion of any sources carte blanche, Chinese or otherwise. Blueboar has been around along time and is policy expert in my opinion. I'm sure he can handle this discussion. Sheesh Jytdog give it a rest.(Littleolive oil (talk) 15:13, 7 September 2015 (UTC))

No it isn't an attack, it is a simple description. You regularly advocate for alt-med in WP. You also blatantly misrepresent me. I did not write anything like "any sources carte blanche," - I wrote "in some way like this." I don't know what you think that gains for you. Jytdog (talk) 15:48, 7 September 2015 (UTC)

• I've moved this back to its own section, here. This was posed as a separate issue and question from the RfC above, and should be dealt with as such. There may be a better way to do this if so please feel free.(Littleolive oil (talk) 14:29, 7 September 2015 (UTC))