Wikipedia talk:Identifying reliable sources (medicine)

Research reviews

MEDRS strongly recommends the use of research reviews, but gives little guidance as to how to determine whether a particular research review is reliable. Currently the only criteria are peer-reviewed secondary sources and publications on Brandon/Hill. Would it be useful to identify other considerations? For example impact factor seems already to be widely referred to. Listing on major indexes has recently been suggested at RSN[1], as have citation of the review and citation of other work by the authors. Other possibilities, raised here[2], are the standing of the authors of the review (well credentialed, widely published?) and the status of their research center; also a consideration of the quality of the research included in the review (publication of the primary sources in top medical journals? the primary research conducted at top medical centers?). Support from the NIH or other respected organization is another suggestion.

A further consideration could be whether the journal carrying the review is put out by a prominent association, organization, or university.

Would the inclusion of factors such as these be helpful? Spicemix (talk) 13:20, 7 February 2012 (UTC)

I'm not sure that the benefits would outweigh the costs. We have a fairly serious problem with POV pushers who want to discredit apparently good sources, and this would pretty much turn into a checklist of reasons to reject any publication that I personally disagree with. That's why the last paragraph of WP:MEDASSESS tells editors not to personally evaluate studies.

Impact factor and citations, in particular, are poor markers for quality. They tell you something about the publication's popularity or prominence. A review that gets cited ten times as an example of flawed thinking has the same citation count as a review that gets cited ten times for its brilliance. WhatamIdoing (talk) 19:30, 20 February 2012 (UTC)

Many thanks for this helpful response. Spicemix (talk) 11:24, 25 February 2012 (UTC)

Status of this page

Following up from a conversation elsewhere, I'd like to begin a discussion here on making several changes to this guideline:

1. Upgrade it to a policy page
2. Emphasize the importance of accuracy, the risk of harm, and the need for the strictest application of WP:V (in the same vein as WP:BLP)
3. Provide guidance for enforcement (perhaps including the creation of an actual noticeboard, a la BLP/N)

I think at least #1 should be pretty straightforward (the guideline has broad support, relative stability over the long-term, and serves the purpose of codifying in more detail the general policies on verifiability and reliable sourcing). The others will take some work and some back and forth. I'd suggest #1 be completed first, and then work can commence on improving the new policy to better describe the importance of MEDRS and support those working to enforce it. Any objections to posting the proposed policy template on the page as a first step? Nathan ^T 19:49, 26 February 2012 (UTC)

Something else to consider. There's more to ensuring appropriate coverage of health and medicine articles than identifying reliable sources, and the guideline as it is goes beyond its current title. Might WP:Health and medicine articles be a more appropriate title for the long-term goal of focusing more resources in this topic area? Nathan ^T 20:04, 26 February 2012 (UTC)

Nathan, you've basically got two different proposals here, and that will confuse the results below. How about discussing this some more and then starting a formal RFC? -- Brangifer (talk) 17:10, 27 February 2012 (UTC)

I support these measures; the prominence of Wikipedia as a trusted source of health information places a significant burden of responsibility on our medical article editors to ensure the greatest accuracy. I think a more stringent, careful attitude to content in this area, and less tolerance for borderline editing is called for relative to the rest of the encyclopaedia, and Nathan's suggestions represent steps in the right direction in this respect. Skomorokh 14:31, 27 February 2012 (UTC)

• Support per previous.  Kiefer.Wolfowitz 14:38, 27 February 2012 (UTC)

• Support. I thought it was policy until recently, to tell the truth! It certainly should be. I'd also support the rename — as Nathan points out, the scope is broader than identification of reliable sources. Jakew (talk) 16:33, 27 February 2012 (UTC)

• I've been beating this drum for a couple of years now, so I'm excited at the possibility that there might actually be support for something like this. I'm actually much less concerned about whether WP:MEDRS is a guideline vs. a policy. I want to know whether the community is ready to take this issue seriously. If so, then it doesn't really matter whether this is policy, a guideline, or even an essay; good things will happen. On the other hand, official "policy" status would probably head off at least one level of wikilawyering.

  Not to look a gift horse in the mouth, but the cynic in me has some reservations. We've built something here that works. WP:MEDRS (in its guideline form) has been accepted by the community and even formed the basis of ArbCom findings. We have a group of knowledgeable, sensible editors who (while we often disagree on individual issues) share a commitment to high-quality medical information and thus "enforce" the guideline in a de facto sense. There is obviously still a vast quantity of wrong, misleading, or outright dangerous crap in this site's medical articles, but we have the tools right now to fix any of them that we turn our attention to (and we have fixed quite a few already).

  More visibility is good, but along with a debate about policy status we're going to get the downsides of "community" involvement—namely, attention from the growing class of Wikipedians who lack or have lost interest in actual content and instead crowd onto whatever project-space discussion seems the most politically active. This guideline works because it was formulated, revised, and implemented by people with in-the-trenches experience. If it becomes a policy, then I'm worried it's going to become like most of our other policies—ossified, impossible to change without three preceding RfCs, and divorced from the reality of actual editing.

  I realize that this is a cynical, if not frankly elitist, viewpoint, but after six years on this site, that's where I am. MastCell ^Talk 17:24, 27 February 2012 (UTC)

(ec) ::That's a fair point. (And the part about "editors" who don't add much content certainly applies to me -- at some point a couple of years ago, the feeling of negativity I got from reading and interacting in the metapedia overwhelmed my desire and ability to contribute regularly). I agree that the WP:MED / WT:MEDRS crowd of dedicated regulars should drive any changes to MEDRS and lead any effort to make enforcement more effective. It's also clearly true that drawing in a wider audience of Wikipedians will net both experienced and serious editors and people whose contributions will be less appreciated.

Drawing the parallels with BLP... For a long time it was the preserve of a small number of administrators and editors who worked tirelessly, and without much support or recognition. Over time, a number of factors contributed to repeated crises and controversies - administrator and editor burnout, decisive but controversial decision-making, attention from the media and external critics, and the involvement of high-profile internal actors (like Jimbo and the WMF board). While the ultimate outcome clearly benefited BLPs and their subjects, to say the process was painful for many community members is an understatement.

My hope is that the final product of BLP, and its status in the community, can be attained for MEDRS with more peaceful means. It may be that the best time to push for broader awareness of MEDRS and related issues is when the guideline, and its use, is at a sort of stable equilibrium and enjoys broad (if quiet) support. It may also be that the potential benefits aren't worth the chance at upsetting a system that, while far from perfect, "works" and represents years of heroic effort. It's for the participants in this discussion to decide if the uncertainty and risk to the status quo is worth the attempt to achieve a more complete solution. My personal opinion is that it's been unfair of the project to expect this Sisyphean task of monitoring and improving to be handled by a small group of people, and I think that you have suffered far more than is necessary to accomplish something everyone should agree is important. At some point we need to find a better way, a way that doesn't burn out great Wikipedians one after another year after year. Nathan ^T 18:35, 27 February 2012 (UTC)

Goodness, where have you been all my life !?!?! Thanks, good stuff. But we've got work to do. SandyGeorgia (Talk) 18:39, 27 February 2012 (UTC)

• All very accurate, we need to take this slow, and be very aware of how any proposal is formulated. An excellent interim step would be some sort of education of admins about how to determine reliable medical sourcing. I'm waiting for Colin to weigh in here, as I have some of the same concerns as MastCell, and the page isn't ready to be simply converted to policy. We need a careful discussion of how to give MEDRS more teeth, to make it work more like BLP vios, and wider visibility, regardless if it's policy. SandyGeorgia (Talk) 17:51, 27 February 2012 (UTC)

• Somewhat related to these concerns, I'm a bit worried about the tendency for some editors to demand the best sources as an end, rather than as a means. No matter what the subject is, the source needs to be strong enough to support the claim being made. It does not need to be any stronger than that. We don't need a review article to support a lightweight claim like "Cancer is a disease". A very basic, non-medical dictionary is going to be good enough for that.
  What we ultimately want is accurate, up-to-date information in the article. Using reviews and textbooks and other top-quality sources is nothing more than a means for getting accurate, up-to-date information into the article. If you can get accurate, up-to-date information into the article while using a relatively weak (but still adequately reliable) source, then that's actually okay: perfectly good material shouldn't be removed merely because it's followed by a merely adequate source instead of a stellar one—or, indeed, by no source at all. It must be possible to supply a reliable source for all material in articles; the source need not actually be named at the time of the original edit. WhatamIdoing (talk) 18:18, 27 February 2012 (UTC)

• Yep. I regret that someone launched an RFC here before we had a chance to hammer out how the page would have to be formulated to give it teeth akin to BLP, addressing concerns such as those raised by WhatamIdoing. For example, the BLP page clarifies when certain types of sources can be used in BLPs. Sometimes we can use a primary source for basic info; sometimes not. SandyGeorgia (Talk) 18:21, 27 February 2012 (UTC)

And there we go ... a premature RFC before those knowledgeable have weighed in, and we even have an RFC proposal. Surefire way to crater anything useful. [3] Thanks. Do folks ever consider that before launching an RFC, you should have formulated what the RFC is about? And why do folks start "voting" before there's even something to "vote" on? SandyGeorgia (Talk) 18:24, 27 February 2012 (UTC)

Thank you ASCII, much appreciated. Now we can hammer something out more carefully, and then hopefully launch an RFC. Colin is out for the day, not feeling well, so I hope we can all put our heads together and come up with something after he's back on board. SandyGeorgia (Talk) 18:29, 27 February 2012 (UTC)

Yeah, the RfC tag is definitely premature. This needs thoughtful input from the experts in this area before it gets put to the wider community. Nathan ^T 18:38, 27 February 2012 (UTC)

(groan) yes I think this page needs to become policy, but we must ensure all permutations and exceptions are covered. I really need to read it through when I have time (hopefully later today) and think it through. Casliber (talk · contribs) 19:30, 27 February 2012 (UTC)

One thing that strikes me is the different tones taken by the two documents (MEDRS and BLP). MEDRS reads like advice (appropriate for a guideline), while BLP is far more stern. As just an example of the tone difference, I've reproduced the intro to both below. There are benefits and drawbacks to both approaches, but it seems like MEDRS could be worded more forcefully and that this might provide better cover and support for people trying to enforce it. That's possible even without changing MEDRS to a policy, I think. Nathan ^T 19:32, 27 February 2012 (UTC)

Yes. After consensus has been approached here, we should invite SlimVirgin and other BLP champions for advice.  Kiefer.Wolfowitz 19:45, 27 February 2012 (UTC)

That will be interesting: Wikipedia:Featured article candidates/Female genital mutilation/archive1 revealed several instances of non-compliance with MEDRS, which resulted in POV. SandyGeorgia (Talk) 20:02, 27 February 2012 (UTC)

(ec) As I recall, BLP was an act-of-Jimbo, hence it was possible for it to be strongly worded from the beginning. Most other policies and guidelines lack that particular luxury, and have to adapt over time, with a gradual cycle of editors being slightly more strict than required by policy, then policy being updated to reflect that, and so on until a steady state emerges. I suspect that there would be more resistance to making MEDRS policy now if it were strongly worded; it would be easier, I suspect, to do that on an incremental basis once it had become accepted policy.

I'm a little concerned by the above remarks that we might be "letting the perfect be the enemy of the good" (to paraphrase Voltaire). MEDRS doesn't have to be perfect before it becomes policy, and it doesn't have to cover every conceivable scenario. It just needs to enjoy strong support in the situations it does cover. MEDRS is a fantastic document: it helps tremendously in implementing and clarifying many other policies in the context of medical articles, adding extremely useful guidance of its own. It deserves to be policy, and that shouldn't in any way stop it from improving further. Jakew (talk) 19:46, 27 February 2012 (UTC)

Yes, but almost all of us who have to apply this guideline in our every day editing still acknowledge that we have some niggles to work out (along the lines of WhatamIdoing's post) before it's ready to be upgraded to a place where it's useful to shoot badstuff on sight as we can with BLP. She gives the example of a dictionary definition of a condition that wouldn't necessarily require sourcing to a medical journal secondary review. That is sometimes true. And it is sometimes not true (the Merck Manual for years had an incorrect definition of Tourette syndrome, and they can still be found many places, so we get into territory of when better quality sources disagree). It gets even better: the New England Journal of Medicine (a most prestigious journal) once printed a faulty definition of Tourette syndrome (damagingly so, claiming that everyone with TS also has ADHD, OCD and disruptive behaviors, utterly untrue), later addressed via a letter to the editor, but they never retracted the article (the faulty part was only one Venn diagram). We've got to nail down the spectrum of possibilities so that we end up with something that allows us to shoot poorly sourced info on site, while not demanding too much of sources. There are also situations where it is OK to use a primary source: I was obliged, unwillingly, to work on klazomania after a group of students adopted it. There are no reviews on that topic, so it was OK to judiciously use primary sources to describe symptoms. I can think of lots of work we need to do here to cover all bases, but I believe if we do that, we can have it ready for a discussion of promotion to policy. We need to end up with something where we know when we can strongly enforce the removal of inaccurate medical text when there are secondary reviews that give better peer-reviewed info. SandyGeorgia (Talk) 19:59, 27 February 2012 (UTC)

Okay, those are fair points. But please remember that an awful lot of good can come of it simply becoming policy, so let's get the right balance between getting it just right and making it policy soon. Jakew (talk) 21:57, 27 February 2012 (UTC)

While I note Klazomania isn't in "Category:Rare diseases", would no results from a PubMed search "review[pt] Klazomania" or Category membership be a reasonable criterion for allowing primary sources? Something like the DOIBot could walk the Medical articles to flag citations with a PMID that don't have "publication type" review (probably), if useful. RDBrown (talk) 02:17, 28 February 2012 (UTC)

• Support after helping out here I fully support MEDRS and I think It would be better for it to have some teeth --Guerillero | My Talk 21:11, 27 February 2012 (UTC)

• Support appreciated, but let's not !vote on anything, please, until some issues are hammered out. We're not ready for policy prime-time yet :) SandyGeorgia (Talk) 21:27, 27 February 2012 (UTC)

Focus of goals

User:ImperfectlyInformed changed the above thread title from "Focus of goals" to "Preliminary discussion about promoting to policy" for transparency.

And I changed it back, since your new title is even more misleading. No regular editor here believes we are ready to promote to policy. The discussion is about our goals. If we get to the point of having something ready to be policy, then the title can suggest that. SandyGeorgia (Talk) 16:03, 28 February 2012 (UTC)

• Hi. Not up to big thoughts today :-(. I'm very encouraged to see folk wanting to get serious about medical topics in the same way as BLP. I share some of the concerns raised. I'm not totally sure than "upgrading" MEDRS to policy is what we're after. Perhaps we should have a discussion about what we really are after, since WhatAmIDoing is quite right that sourcing is a means to an end -- and also that there is a hierarchy of sourcing quality that needs to be matched with the claim being made (extraordinary claims / extraordinary evidence). The gold standard randomised controlled trial, or meta analysis of such, tends to only address efficacy questions, which is really a small part of health related information. When dealing with a dispute I've often found citing WP:WEIGHT to be more helpful than MEDRS. Because weight is a reason to need secondary sources. I also find the "reliable" aspect to cause confusion sometimes. People naturally don't understand why we might say a paper published in a respected peer-reviewed journal might be "unreliable". WhatAmIDoing frequently says that "primary" is not another word for "unreliable" and that is true. A primary research paper can be an extremely reliable document about the research (except if Wakefield wrote it). But we generally shouldn't be writing directly about research but rather about the knowledge that research brought about. And the clinical application and generalisation of that research requires a secondary source. That isn't an issue of "reliability" but of NOR and WEIGHT.

Oh, this is a bit of a wandering brain dump. Sorry. So I think we should discuss the nutshell of our goals and principles. The issue of scope is also important. BLP has a fairly specific scope that is easy to address. The issue of power is also important. BLP gives editors absolute rights to remove text on sight. Are we after that and if so, which sorts of medical facts would be subject to such? What would be the consequence of that on our articles, which may not be well sourced at present. Would someone take the new policy and start gutting medical articles of information that actually was ok just because it wasn't sourced to a review in the Lancet? Colin°^Talk 21:33, 27 February 2012 (UTC)

By whatever process we get there, I believe our goals should be:

1. Better education throughout Wikipedia about what a MEDRS-compliant source is, how to find reviews, when they should be used, why not all medical journal reviews are the best sources (per Colin above), etc-- general education. I am often shocked to realize that MEDRS isn't well known at GAN, DYK, Articles for creation, New page patrol-- so many places that could be the first place to educate offenders. Many editors see a journal citation, or a New York Times citation, or a book written by a lay advocate, and think the article is well sourced.
2. A page that will better allow us to shoot the really bad non-compliant poorly sourced text on site, and not have to spend weeks arguing over it.

SandyGeorgia (Talk) 21:57, 27 February 2012 (UTC)

One benefit to "upgrading" it is the follow-on benefits in other fora - policy status and prescriptive language avoids the "it's just a guideline" or "it only says you should try to do X, not that you have to" objections. The question on the consequence to articles that Colin raises is a good one. "Harm" in the BLP sense is comparatively easy for a layperson to identify, and scaling the severity of harm is somewhat straightforward. How would someone not practiced in applying MEDRS calibrate how aggressive to be in addressing questionable claims? Judging the potential for harm in the MEDRS umbra of articles is likely to be difficult for a non-expert. Nathan ^T 22:42, 27 February 2012 (UTC)

Related to the point about "it's just a guideline", you might like to read WP:The difference between policies, guidelines, and essays if you aren't familiar with it. WhatamIdoing (talk) 04:35, 29 February 2012 (UTC)

I am indeed. I would say that just like there is a vast gulf between how we are supposed to see administrators and how they are seen in practice, there's a real difference between how policies, guidelines and essays are supposed to interact and how they are actually used day to day. Policies and guidelines are both supposed to represent the Wikipedia consensus, and both can be enforced, and policies are descriptive not prescriptive, etc. etc. but in practice, a policy is a much bigger club than either a guideline or essay. Nathan ^T 15:10, 29 February 2012 (UTC)

Background and opposing view

Oppose Is this a serious proposal? If so, the title of this thread should be changed or a subthread should be added that clearly states the proposal, e.g. "Proposal to upgrade MEDRS to a policy". Someone should also probably drop a note in the village pump and probably at some point at the top of the watchlist. I had to fix the RfC ta.

I think this policy already gets abused enough as a guideline. That's not to say I don't think it should be a guideline, but it really gets abused. For example, I've had a perfectly good secondary source published by the American Diabetes Association blocked (Talk:Statin/Archive 1#Doc News, MEDRS, and doctors' dismissal of complaints); in the archives, the most recent (Wikipedia talk:Identifying reliable sources (medicine)/Archive 5) shows comments about it being used to remove research sections (Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#MEDRS and research sections) and about editors overlooking the finer details of the guideline and its neglect of research article value(Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#The value of peer-reviewed studies reflecting developments since the last review is greatly overlooked.). There's a lot of need for editorial discretion in editing in order to make a balance, highly-valuable. MEDRS tends to be interpeted as "reviews are required". However, reviews are a very small fraction of the research out there. In many lines of research, they don't exist. In many others, there's none that are good or up-to-date. Back in Wikipedia talk:Identifying reliable sources (medicine)/Archive 1, I was involved in some spirited (probably much too spirited) debate. As I recall it User:Eubulides drafted much of the original language. In Archive 2 based on Wikipedia talk:Identifying reliable sources (medicine)/Archive 2#Should we make this a guideline.3F this was promoted to a guideline, although it was opposed by 3 of the 17 or 18 voters (User:Paul gene, User:Una Smith, and User:Mihai cartoaje; I abstained). The promotion over 3 led to a controversy when one of the opposers, User:Paul gene (a self-professed pharmaceutical researcher), said that it had been raised to a guideline inappropriately (Wikipedia talk:Identifying reliable sources (medicine)/Archive 2#Rfc Promotion of MEDRS to guideline). Eventually, when User:Paul gene stopped protesting, everything went on as usual, but there's frequent discussion and concern about the ability to use MEDRS to keep out perfectly encyclopedic information, and to make Wikipedia more susceptible to biases such as the pharmaceutical lobby. II | (t - c) 07:40, 28 February 2012 (UTC)

II, the RFC was agreed to be a mistake and was retracted. You didn't "fix" it, you've just buggered things up. We absolutely don't need a pile-on of support/oppose comments at this stage. We are at the very early stages of working out what should be proposed to the community. That might be a change to MEDRS status but it might be a new policy page.

The above linked discussions don't really help your case, I'm afraid. For example, Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#MEDRS and research_sections contains a friendly discussion where everyone agreed that the information could and should come from secondary sources in order to determine the appropriate weight to give to the new research. I agree that some editors may be over-zealous in their application of MEDRS but that is an editor-behaviour issue, not a content-guideline issue. For the vast majority of situations, MEDRS is appropriately used to inform editors of the standard of sourcing we require for medical claims. I do agree with you that there is more useful information than that which appears in reviews.

Paul gene was a classic example of an academic editor who didn't get Wikipedia. He believed the encyclopaedia should be written by expert editors drawing from the primary literature. He thought he could do a better job than the writers of reviews in medical journals, and should be allowed to do so on Wikipedia.

Wikipedia's guidelines are designed to keep out biases, whether from the pharmaceutical lobby or the alternative medicine lobby or the megavitamin lobby or whoever. Our secondary sources are imperfect and some may indeed be biased, but they are all we've got. Individual editors may think they are unbiased and a better arbiter of the truth than the published literature, but the evidence shows that many of the editors on Wikipedia come with their own bias and will cherry-pick the primary literature in order to push that bias on WP. Our system of deferring to the secondary literature is not perfect but it works reasonably well given our editor-base. The alternative is for Jimbo to pay experts he trusts to write articles for him. I believe he tried that and it didn't work. Colin°^Talk

Oppose I too oppose for now. Here are my concerns:

• Scope: What articles is MEDRS applicable to?
  • Is it applicable to any medicine-related article?
  • Is it applicable to the whole article?
  • Should the same stringent guidelines apply to sections on biochemistry/genetics, veterinary medicine and pharmacology? Have people from these fields been asked for their opinion?
• Too stringent on research sections.
  • Reviews are of course preferable, but stripping out primary research results without replacing with a review is, in my opinion, usually unhelpful.
  • Results in rats/model organisms should not be generalised to humans. No debate. But that doesn't mean the results are not useful or significant to non-human medicine. Yet they are unwelcome in research sections.

I feel MEDRS pushes to make Wikipedia into a human medicine textbook. By all means be stringent on human medicine material, but it also needs to be more inclusive. pgr94 (talk) 11:00, 28 February 2012 (UTC)

Pgr94, it would be really helpful if you rephrased your comments to remove the !vote. We really are not at the stage where something concrete is proposed. MEDRS does specify its scope, so I suggest you read it. There's a debate to be had about how existing text that is poorly sourced should be handled, but we need to shift the culture of editing to where editors are not adding text sourced to primary research papers nearly as often as they are. Do you agree that that text+source combination shouldn't have been added in the first place? The person who decides whether rat experimental results are of interest to humans is the expert writing a published secondary source. They are only "unwelcome in research sections" where editors have added them based on their opinion of their importantance and relevance to humans, and have failed to satisfy WP:WEIGHT by indicating which secondary sources support that view. I don't really understand your last sentence. Colin°^Talk 11:12, 28 February 2012 (UTC)

It looks like we have two participants here who didn't read the page (that's helpful), and decided to add unconstructive commentary; we know the RFC was premature, it was inititated in good faith, based on a post that wasn't a clear proposal, and that's why it was removed. If folks weighing in on discussions actually read them, it always help. I've cleaned up more of the formatting mess introduced by II, the link to Eubulides' correct name, and no-- Eubulides was not instrumental in this page, and even if he were, discuss the content, not the contributor (he was quite an excellent editor, but whatever). SandyGeorgia (Talk) 14:25, 28 February 2012 (UTC)

Eubulides wasn't involved in the initial draft, but did become a significant contributor to the text. See this contributor list. I regard that as a very good thing. Colin°^Talk 14:50, 28 February 2012 (UTC)

SandyGeorgia, I believe my comments are constructive; perhaps they just don't match your point of view. I am not in favour of crystallizing the current MEDRS page into policy at present (which is my understanding of the proposal in the opening paragraphs of this discussion). pgr94 (talk) 14:39, 28 February 2012 (UTC)

Ok. Let's move on from that. Can you deal with my response above. I don't think your "oppose" reasons are valid. We agree that it is more helpful to rewrite text with better sources, but ultimately, editors should have the right to remove contentious medical facts that aren't reliably sourced -- and policy already allows them to do that (see below). I've addressed the scope/rats comments above. Could you respond? Colin°^Talk 14:45, 28 February 2012 (UTC)

• I noticed Colin above asked Pgr94 (talk · contribs) to remove the oppose vote. Why isn't the same question being asked of everyone who has bolded support statements at the top? In any case, I don't think I'll ever sign on to promote this to policy. Guideline is just a better semantic fit. Also, I did indeed read through the thread, but there was no notice at the top whatsoever that the proposal was withdrawn by its initiator. Just because a few people think a proposal is premature doesn't mean it gets withdrawn. If reasonable processes are followed to make threads clear and transparent, then there won't be confusion. The title of this thread, for example, is clearly not a reasonable title. II | (t - c) 15:55, 28 February 2012 (UTC)
  • Because I couldn't be arsed. There's no conspiracy, II. I agree with you that as it stands, this guideline is a guideline and isn't really worded appropriately as policy. Everybody thought the RfC was premature and is best withdrawn until we've collected our thoughts. It could be quite some time before we work out what to propose to the community. It is difficult to change titles because people link to them and anyway that was the initial question, which has since matured. — Preceding unsigned comment added by Colin (talk • contribs) 16:05, February 28, 2012 (UTC)
  • II, please read more carefully; your summary is incorrect. People voting support were asked several times not to, and the discussion of the removed RFC is there. It should also have been very clear that I was unpleased that an RFC was launched by someone new to this page well before anyone who regularly participates here had said we had anything close to policy. Please avoid sidetracking the discussion of where we want to head with this page, if anyone tries to make it policy as it stands, you will find every regular medical editor here most likely opposing, the page is by no means defined by anyone yet as being close to ready for promotion to policy, thank you for not being disruptive.SandyGeorgia (Talk) 16:08, 28 February 2012 (UTC)

II, I initiated the discussion on this page, but not the RfC. It wasn't intended to be an RfC, or to go "primetime" beyond the people on this page. Nathan ^T 15:07, 29 February 2012 (UTC)

Psychology and other health-care disciplines

The policy should cover "human health-care" generally: psychology; nursing, scientific and therapeutic touchy-feely; dentistry, etc.  Kiefer.Wolfowitz 19:58, 27 February 2012 (UTC)

While that would almost certainly be a good result, the process of getting some of those adopted might lead advocates of fringe practices to undermine any progress towards such an end. On the premise that perfection is the enemy of the good, we should probably set them aside and focus on EBM. LeadSongDog come howl! 22:55, 27 February 2012 (UTC)

I agree. If it's EBM-supported, it should be covered. If it's not covered by EBM, it's a belief system--one that may well be true and functional, but not one that can support specific claims that will stand up methodologically to, say low-dose aspirin's utility vs. MI. Jclemens (talk) 07:24, 28 February 2012 (UTC)

Kiefer, I believe you mean "proposed" policy, and we've not yet got anything close to a policy proposal here. SandyGeorgia (Talk) 16:41, 28 February 2012 (UTC)

Generally, we call out specific healthcare-related disciplines as we discover a need for it, i.e., as some editor tries to claim that the failure of MEDRS to explicitly say that _____ is 'medicine-related' means that they have permission to use inappropriate sources. That happened repeatedly with alternative medicine (now named), and it recently happened with psychology (see section above). But I don't think that we've had any similar problems with nursing or dentistry, so I'd avoid adding them. WP:Nobody reads the directions anyway, and the longer they are, the even lower chance that anyone will read them. WhatamIdoing (talk) 04:43, 29 February 2012 (UTC)

Burden

The policy WP:V says:

The burden of evidence lies with the editor who adds or restores material. You may remove any material lacking an inline citation to a reliable source. Whether and how quickly removal should happen depends on the material and the overall state of the article; consider adding a citation needed tag as an interim step.¹ Editors might object if you remove material without giving them time to provide references. It has always been good practice to try to find and cite supporting sources yourself. Do not leave unsourced or poorly sourced material in an article if it might damage the reputation of living people; you should also be aware of how the BLP policy applies to groups.

¹: It may be that the article contains so few citations that it is impractical to add specific citation needed tags, in which case consider tagging a section with {{unreferencedsection}}, or the article with {{refimprove}} or {{unreferenced}}.

²: Wales, Jimmy. "Zero information is preferred to misleading or false information", WikiEN-l, May 16, 2006: "I can NOT emphasize this enough. There seems to be a terrible bias among some editors that some sort of random speculative "I heard it somewhere" pseudo information is to be tagged with a "needs a cite" tag. Wrong. It should be removed, aggressively, unless it can be sourced. This is true of all information, but it is particularly true of negative information about living persons."

it goes on to say

Content related to living people or medicine should be sourced especially carefully.

Now, I'm sure we all agree that in an ideal world, where experienced editors had limitless free time and limitless access to high quality sources, that any poorly sourced medical facts would be rewritten based on the best sources according to WP:MEDRS. That's what WP:PRESERVE encourages, though that policy also accepts there are times "when it might be more appropriate to remove information rather than to preserve it." WP:BURDEN has always placed the burden of reliable sourcing onto the editor who adds or restores the material. I disagree with WhatAmIDoing above when she says "the source need not actually be named at the time of the original edit". That's only true of material unlikely to be challenged, which is fairly rare for a health claim. We should not, of course, sanction an editor gratuitously removing text that wasn't using the best possible source. However, if an editor reasonably believes the text could be wrong or misleading, or the source fails to support the text, then they are justified to remove that text without supplying an alternative. Let's be clear that WP:V says "You may remove any material lacking an inline citation to a reliable source." All WP:BLP does, in effect, is to add some exclamation marks to that sentence, and removes the option of doing nothing. Here's what it says about sourcing:

Wikipedia's sourcing policy, Verifiability, says that all quotations and any material challenged or likely to be challenged must be attributed to a reliable, published source using an inline citation; material not meeting this standard may be removed. This policy extends that principle, adding that contentious material about living persons that is unsourced or poorly sourced should be removed immediately and without discussion. This applies whether the material is negative, positive, neutral, or just questionable, and whether it is in a biography or in some other article. Material should not be added to an article when the only sourcing is tabloid journalism. When material is both verifiable and notable, it will have appeared in more reliable sources.

A number of editors have suggested that medical facts, or health-related information, should be on the same level as facts about living people, particularly facts that may damage their reputation. So what sort of medical facts are we concerned about? How would we identify the sensitive ones from the everyday ones that can be treated like any other fact on WP? Should editor actions be similar to BLP? Will edits enforcing this policy have the same protection as BLP? --Colin°^Talk 13:17, 28 February 2012 (UTC)

I'm going to ignore your questions for now, since they will require more thought.

WP:V and WP:BLP require inline citations for exactly four types of information. But with the exception of contentious (not all) matter about BLPs, no policy puts a deadline on providing those citations. Adding non-BLP material without adding a citation is 100% 'legal' under the existing policies. You can't revert its removal if it's been removed, but WP:V does actually permit you to add the unsourced (but verifiABLE) material (once). WhatamIdoing (talk) 04:49, 29 February 2012 (UTC)

Hmm. WP:V says "material...must be attributed to a reliable, published source using an inline citation" and the responsibility for doing this is "the editor who adds or restores material". Where does it indicate that this action can be indefinitely postponed (as long as presumably you intend to do it)? Where does it say, then, that it cannot be postponed when one restores text? Both addition and restoration, to me, seem to have the same demands put upon them, and it is a demand that should be met with in a reasonable time frame (such as during an edit session). Are you saying that an editor who every day adds "liver disease" to a random drug article's adverse effects list, without supplying any source, would be editing perfectly "legally" and could not be stopped? They could simply claim that they will do it "some day". Colin°^Talk 09:28, 29 February 2012 (UTC)

Basically, yes: you can add unsourced (but not impossible-to-source) material. All you need to do is declare that it's not a direct quotation, that it's not contentious matter about a living person, and that you personally thought it was not information that was WP:LIKELY to be challenged. Assuming those three conditions, your unsourced addition is just fine unless and until it is specifically WP:CHALLENGED.

BURDEN only addresses one point: the identity of the person responsible for providing a citation. Specifically, BURDEN exists so that when I add the unsourced claim of "liver disease", and you challenge it (e.g., by removing it), then I can't tell you "Well, if you want a citation that badly, then go add one yourself."

(If you really want to make the person stop adding , you'll have to follow the general dispute resolution pathways, probably based on an allegation of disruption, not merely invoke WP:V.) WhatamIdoing (talk) 17:51, 29 February 2012 (UTC)

Please educate me if I'm wrong, but I don't believe one can do that on a BLP (or at least not continue to do it after one revert), and if I understand correctly, our aim here is to define under what circumstances similar would apply to medical statements. In other words, if someone adds an uncited birthdate to a BLP, and I have good reason to doubt it but can't cite the actual birthdate (let's say I know the person in real life, but bd is not citeable), if someone continues to add, I can cite BLP and not be subject to 3RR. SandyGeorgia (Talk) 18:15, 29 February 2012 (UTC)

Ok, to avoid talking past each other, I'm totally not interested in text that doesn't clearly require a citation. We're talking about text that says a drug has liver disease as a side effect or that eating some fruit cures cancer. So the person claiming they didn't think it was "likely to be challenged" is either lying or lacks a clue. Does policy make that action "legal". I'd say no. They've added some text to Wikipedia that breaks our policies. We all make mistakes but if they keep doing after being asked not to, and asked to fix things, then we'll get round to stopping them at some point.

The problem with the WP:V text is that it is a passive requirement of the text rather than an active requirement of the editor. And as you interpret it, even though BURDEN places a requirement on the text-adding-person, it doesn't explicitly say they should have done it in the first place. If the policy said "When adding or restoring material [that requires an inline citation] you must supply an inline citation" then we'd be all clear about where things stood. Why can't it be this clear when it uses clear language about who should or may remove the text. When my wife tells me the bins need emptied, she's not volunteering to do it herself, or expecting someone else to do it for us. Has this been discussed on WP:V before? Colin°^Talk 18:32, 29 February 2012 (UTC)

I suspect another place where we're getting into trouble here is WhatamI's concern over "five digits on a hand" getting a citation request, and I submit that is already covered at WP:V and we shouldn't get hung up on that territory. (Besides, even in that case, it looks to me like the cn tag was just applied in the wrong place-- would have been appropriate if moved a few words over, to cover the "other" conditions resulting in other than five digits.) SandyGeorgia (Talk) 18:36, 29 February 2012 (UTC)

No, I'm not worried about common knowledge here. The fact is that WP:V doesn't require the citation to be added, as Colin puts it, "in the first place", i.e., at the time of the original edit. Some editors believe it should, but it doesn't actually impose that requirement.

As for lying vs cluelessness, challenges are so rare on low-traffic pages that even the most extraordinary claims are not WP:LIKELY to be challenged, because ot's not even likely that anyone will read the page, much less tag it. LIKELY is effectively a lower standard than WP:When to cite. WhatamIdoing (talk) 19:11, 29 February 2012 (UTC)

I'd have thought "challenges" are pretty rare full stop. I mean WP is so full of unsourced crap that nobody reads that you could then statistically say WP:V isn't really required at all, and the 1% that matters to anyone is an edge-case. I've always found "challenged" problematic because it seems rather aggressive, like you're saying the editor is lying or a gullible fool if they believe that. Perhaps your essay should define "challenge" to mean "challenge the author to supply a source" rather than "challenge the author's integrity" or some other slur. I'm not happy that you say "likely to" includes the "likely to even be read" or "likely to care much" aspect. Surely a better assumption would be an editor, reading and interested in the page contents, is likely to want a source to confirm the veracity of that statement. But I suppose this discussion belongs on WP:V. I'm reluctant to go there just now because they are too busy fighting over the Truth, and because I've unwatched all the policy pages because they are a huge timesink. As for "in the first place", think we shall have to disagree on "it doesn't actually impose that requirement" as I think that is up to how someone reads the policy as currently worded (it should make it clear one way or the other if folk have agreed that much). --Colin°^Talk 19:39, 29 February 2012 (UTC)

You're right that WP:V is a mess at the moment. But if you'll remind me when (if?) that mess ever dies down, I'd be happy to propose a clarification about timing at BURDEN. Right now, it'll just get lost in the shuffle. WhatamIdoing (talk) 23:31, 1 March 2012 (UTC)

Examples

Adding in Colin's examples [4] to mine: SandyGeorgia (Talk) 15:39, 28 February 2012 (UTC)

I suggest we might focus the discussion better if we work from examples-- if we can put forward the range of possibilities, from least to most harmful, it might help others:

1. Poorly sourced information that is nonetheless correct (we need an example). Sometimes an experienced medical editor looks at an article, sees nothing incorrect, but doesn't have time to add the secondary review sources (or doesn't have journal access). What should be done in this case? Request citations, remove text to talk, etc?

   See, for example, splenectomy. The indications section appears essentially correct but is unsourced. Likewise, later sections appear to correctly outline the need for vaccination and the important risk of overwhelming post-splenectomy infection, but are also unsourced. MastCell ^Talk 16:55, 28 February 2012 (UTC)

   See this fact tag, added to the number of digits normally present on the human hand. We don't want information that is known (to the average child) to be correct to be challenged, much less to be cluttered up with a citation to a "secondary review source". You're simply not going to find a meta-analysis on the number of digits on the human hand. WhatamIdoing (talk) 04:52, 29 February 2012 (UTC)

   I don't think we should get hung up on a case like that. Common knowledge is already covered by WP:V, and I suspect that person simply put the tag in the wrong place-- move it seven words to the right and it makes sense. SandyGeorgia (Talk) 18:38, 29 February 2012 (UTC)

   Oh, I suspect the IP was having a laugh. And quite a good one really. But yes, can we just assume that this MEDRS discussion concerns something important, non-obvious, etc, otherwise it is going to be really tedious when WhatamIdoing (rightly) reminds us each time of all the exceptions to requiring a source :-) Colin°^Talk 19:11, 29 February 2012 (UTC)

2. [Added per Colin] Poorly sourced information that the editor has no idea whether correct or incorrect. For me, this is a common occurrence. This might be an inconsequential fact like the pills are blue but it might be a signficant fact like mentioning liver failure as a side-effect.
3. [Added per Colin] Poorly sourced information where the text is not fully supported by the source but may in fact be correct. An example would be an article saying that a drug is successfully used for treating a condition, and citing a drug trial. The drug trial doesn't establish clinical use, we'd need some clinical guidelines for that, or similar. What should the editor do?
4. Basic factual error, sourced to an outdated or contradicted review, or demonstrably wrong according to better or newer sources: an example of something like that would be if someone were to add, based on Jankovic's NEJM article, that the definition of Tourette syndrome includes ADHD, OCD, and disruptive behaviors. Basic factual error, only found in a Venn diagram in one journal article, contradicting hundreds of secondary reviews (and the DSM definition). Remove on sight as basic factual error that stigmatizes via mischaracterization persons with TS. Would BLP-type restrictions to reinstating something like this apply?
5. Basic factual error, poorly sourced (need an example). Remove on sight. Once removed, BLP-type provisions apply (don't reinstate, not subject to 3RR).
6. [Added per Colin] Text that violates WP:WEIGHT. Often text based only on primary research papers does this.
7. [Added per Colin] Text that violates WP:NOR. For example, stating or implying that rat research might be relevant to a human disease, citing the rate research primary research paper, rather than a review of the human disease.
8. Text that is not sourced according to MEDRS, but is not dangerous per se, although introduces POV or in one case, cultural stigma. The text from Female genital mutilation, stating a 10% death rate because one lay advocate claimed that in a book, with no citation or study or data given (based on a midwife's opinion), and not contained in any of the secondary reviews on the topic. There are, in that case, good secondary reviews available, but the article (at one point, not sure if it has since been fixed) ignored the medical reviews and the responsible and culturally sensitive tone they took to the issue in favor of lay advocate books against female circumcision. This is a case where inferior sourcing introduces POV. Do we shoot on sight, or just label the article as unbalanced?
9. Poorly sourced information that is dangerous: examples abound in PANDAS. There is an abundance of secondary journal reviews covering this unproven hypothesis, the problems with that hypothesis have led to "internet-armed parents" (terminology used in one secondary review discussing the PANDAS problem, whereby parent advocates are making faulty and dangerous choices based on misinfo spread on the internet) making poor medical decisions about the use of antibiotics, IVIG and other interventions that endanger the health of the child as well as society (over-use of antibiotics based on a controversial and unproven hypothesis). Advocates perennially chunk in primary-sourced text that is contradicted, unproven, unreplicated or otherwise unsupported by secondary reviews. Shoot on sight and apply something that needs to be developed akin to WP:BLP to dangerous poorly sourced information. The distinction I'm attempting between this (No. 9) and the previous (No. 8) is in the realm of misinformation that is dangerous to one's health (theoretically, no one will die because the incidence of death in female circumcision is overstated-- that's just a matter of POV).

I don't know if I've covered all bases, but we need to somehow define how a new guideline or policy page would be applied to different levels of severity. Better examples might be supplied. SandyGeorgia (Talk) 14:57, 28 February 2012 (UTC)

6 Poorly sourced information that the editor has no idea whether correct or incorrect. For me, this is a common occurrence. This might be an inconsequential fact like the pills are blue but it might be a signficant fact like mentioning liver failure as a side-effect.

7 Poorly sourced information where the text is not fully supported by the source but may in fact be correct. An example would be an article saying that a drug is successfully used for treating a condition, and citing a drug trial. The drug trial doesn't establish clinical use, we'd need some clinical guidelines for that, or similar. What should the editor do?

8 Text that violates WP:WEIGHT. Often text based only on primary research papers does this.

9 Text that violates WP:NOR. For example, stating or implying that rat research might be relevant to a human disease, citing the rate research primary research paper, rather than a review of the human disease. Colin°^Talk 15:11, 28 February 2012 (UTC)

Added to list above. SandyGeorgia (Talk) 15:44, 28 February 2012 (UTC)

Something like PMC 3227668 might be helpful to document why mass-media sources are unsuitable.LeadSongDog come howl! 04:34, 1 March 2012 (UTC)

On a vaguely related matter, I wonder if we could cite something like this to illustrate why advocacy sites on the web can be unreliable? Jakew (talk) 10:49, 1 March 2012 (UTC)

Policies, Guidelines, Standards, and Procedures

At one point, I wrote a large chunk of IT policies for a Fortune 100 company I'm sure you've all heard of. In that system, "policies" were broken down into: 1) Policies, a one page general statement of management's expectations, roughly analogous to our pillars, 2) Standards, specific statements of expectation on individual topics (e.g., no computer except an IT approved firewall shall be connected to both an internal and external network), and 3) Procedures, actionable means to accomplish those standards, where each procedure was hierarchically associated with a specific standard.

Maybe we need something like that here? Rather than trying to promote "identify reliable sources" to a policy on its own, we would be better served by a succinct policy that says something to the effect of "Medical articles or articles making specific medical claims must use sources according to (identifying reliable sources) to support those claims, or such claims must be reworded to make it clear that they are unsupported by scientific research". That way, we get a quick skeleton of a policy in place, which the community can all agree on in principle, and the guideline, the successor to this page as it stands, continues to evolve. I know that's not really the Wikipedia Way, which seems to be to shoehorn every last thing into a policy page and fight over changes, but maybe it would be worth trying in this case... Jclemens (talk) 15:42, 28 February 2012 (UTC)

I don't think there's any such thing as a "quick skeleton, which the community can all agree on in principle"; I can think of many editors who will object to your proposal. Slow and steady does the job, and it should be noted that no regular medical editor participating in these discussions believes that we yet have anything in shape here to be be promoted to policy-- there are issues to be resolved, and we should go about doing that slowly and systematically. The rush to RFC, or a jump to quick solutions, is detrimental to long-run solutions. SandyGeorgia (Talk) 15:50, 28 February 2012 (UTC)

I was quite surprised at how long WP:BLP was. I agree it would be helpful to start small but also agree with Sandy that there is no rush to establish something as polciy. MEDRS contains a lot of advice, as a guideline might, rather than just stating succinctly what our goals are and our main means of achieving those goals. A medical policy page could have a similar nutshell to BLP:

Material making medical or health claims must be written with the greatest care and attention to verifiability, neutrality, and avoiding original research

The same "we must get it right" and "written conservatively" attitudes would apply too. I do believe the focus on a policy page should be wider than just talking about "identifying reliable sources", which in some way is an unhelpful goal. The reliable/unreliable aspect is great for explaining why newspapers are poor sources, but not for why primary research papers in a peer reviewed journal are often "inappropriate sources" for the sort of text folk want to add. The issues we face involve all three of the above content policies, not just the appplication of WP:V to medical text. The choice and correct use of sources is a consequence of applying those policies rather than blindly driving the content. Colin°^Talk 15:59, 28 February 2012 (UTC)

By dint of comparison, I thought I'd take a look at the HONcode principles. Unless I'm misunderstanding, we should be able to meet all of them except principle 1 ("Authoritative: indicate the qualifications of the authors"), which would fall afoul of "anyone can edit" by requiring that editors out themselves. Still, it seems to me that something approaching that could help guide our policy development. We could in theory have volunteer identified reviewers put their real names on a specific revision of an article to say they found it was reasonably complete and current as of that (permalinked) revision. Such reviews wouldn't have to be either internal or external, each could be accomodated.LeadSongDog come howl! 17:05, 28 February 2012 (UTC)

HonCode is a massive failure (I know of cases where they have been contacted about websites that don't comply and they don't ever remove the HonCode), but anyway, I don't see how their principles relate to our issues? SandyGeorgia (Talk) 17:10, 28 February 2012 (UTC)

I don't know about HONcode's track record, but the principles themselves seem reasonable. I think we'll have problems with #7 ("Financial discloure"), though. Even in cases where clear financial conflicts of interest exist, we evidently have no way to address them effectively or communicate such conflicts to the reader. And worse, we're dependent on voluntary disclosure by pseudonymous editors to even learn about a financial conflict of interest in the first place. MastCell ^Talk 19:18, 28 February 2012 (UTC)

I think Sandy's right, nothing should be promoted before there's a coherent whole. But it's worth considering whether a two-tiered approach might be helpful - a shorter, to the point policy, and a longer, more detailed guideline that supports the policy and its implementation. Now, granted, knowing the way people fight over policy (i.e., by attacking it's weakest links...the supporting guidelines), the guidelines part would need just as solid consensus as the policy, prior to promotion. But it's certainly worth considering. Guettarda (talk) 18:02, 28 February 2012 (UTC)

My intention was more that we should consider the principles, not pursue their certification. Sorry I wasn't clearer. Re financial disclosure, MC's right. Because we have no way to ensure it is disclosed, we should be frank with readers that the potential of undisclosed conflict is always there.

How, precisely, would we go about "being frank with readers" on this part? A banner saying "This may have been written by drug shills?" above any pharmaceutical article? Em, I think this isn't taking us anywhere useful... Colin°^Talk 20:26, 28 February 2012 (UTC)

We already have wp:MEDICAL, it would only need a minor tweak to the wording, such as "accuracy" --> "accuracy or impartiality". LeadSongDog come howl! 20:41, 28 February 2012 (UTC)

Standards such as these require interpretation in individual cases, and interpretation will always be a matter of judgment, and in any non-trivial case, to some extent a matter of opinion. It's wrong , imo, to be exactly prescriptive in writing the standards, because there are too many individual possibilities. I do not think that this is or should be a policy; not even its parent, WP:RS is a policy, but a guideline, and this is but a specialized derivative of that. The actual policy, in the sense of something we would almost always follow, is WP:V. What happens to adopted "policies" that are not very fully supported by consensus is that they get mostly ignored,(WP:NPA is probably the prime example). the apparent agreement here is deeoptive, because the contributors here share the same general POV (one that I also share, but at least I have a skepticism about the degree to which all Wikipedians actually agree with it. ), DGG ( talk ) 18:57, 1 March 2012 (UTC)

I don't think that RS is MEDRS's "parent" page; I think that MEDRS's actual parent is WP:V, just like RS's parent page is WP:V. WhatamIdoing (talk) 23:33, 1 March 2012 (UTC)

Agree. Also I think we're losing the point here re whether page is escalated to policy. The issue was(I thought, anyway), one of enforcement, that is, can we write something that gives guidance about when text can be removed, akin to what we have in BLP. SandyGeorgia (Talk) 01:11, 2 March 2012 (UTC)

Work priority?

Could we possibly prioritize three things, before we get bogged down?

1. Work on general better education and awareness of MEDRS.
2. Look at beefing up some of the wording now here along the lines of BLP to include more of "we need to get it right" kind of wording, regardless if it's policy.
3. Define our samples so we can begin to discuss what we think should be done in each case, akin to the possibilities we have with BLP.

I submit that if we can come close to agreement on individual samples (like the nine above), it will make it easier for us to figure out which way to go with the page. And I'd love to see us come up with a general educational blurb that we can post over to DYK, GAN, Articles for creation, AN, ANI, and any new article or editor that encourages more people to look at MEDRS. I've been patrolling new medicine articles lately, and what is out there is just scary-- particularly that so few at Articles for Creation are aware of MEDRS. We should strive to educate new editors early on ! And I'm finding new articles never tagged with the Medicine WikiProject, which is another way to educate about MEDRS. [5] Let's get 'em when they come in the door! User:AlexNewArtBot/MedicineSearchResult and Wikipedia:WikiProject Medicine/Article alerts. SandyGeorgia (Talk) 20:00, 29 February 2012 (UTC)

I think better education and awareness of MEDRS is a good idea, but... If achieving policy status at some time in the near future is a goal, grabbing a lot more eyeballs at this particular point might be counterproductive. Nathan ^T 03:42, 1 March 2012 (UTC)

On point #3, perhaps a "MEDRS examples" page would be a helpful way to provide practical advice. WhatamIdoing (talk) 23:35, 1 March 2012 (UTC)

Yes, but I still thought that wrt those examples, we are looking to develop something akin to BLP that gives guidance on what can of text can be "shot on sight", and how to handle the other nine examples. SandyGeorgia (Talk) 01:12, 2 March 2012 (UTC)

I created Template:RSPlease a while ago but not sure how to get the word out. What we need really is more editors to take care of WP:MEDRS compliance and to support each other in the issues that arise. --Doc James (talk · contribs · email) 11:51, 4 March 2012 (UTC)

Sorry I dropped out of the discussion for a bit - business trip to San Diego, about as far from home as I could possibly go and still be in the lower 48. Anyway, I think we should focus on #2 and #3. Crafting stricter language will be easier, at least until we get to describing the hierarchy of problematic content. There are a few approaches we could take here, some that might be really quite novel for Wikipedia. Maybe the most familiar to people from other settings would be a staged system with criteria - i.e. "Unsourced statement of a fact that is not considered to be common knowledge, with no potential for causing harm if relied upon as fact by a reader" for a minimal harm category, up to "Unsourced statement or assertion of fact capable of resulting in significant or severe harm if relied upon by a reader." I don't think we would want the bureaucratic nightmare of people tagging these unsourced statements according to tiered categories, but it could be a useful way of distinguishing the severity of identified problems to others (and, as a result, a handy tool in enforcement). Nathan ^T 17:41, 5 March 2012 (UTC)

I think this is an interesting idea; as a general rule though, I think the BLP mirror of "contentious" material could be applicable here. We don't want to be overzealous in removing unsourced/poorly sourced information, but "contentious" material, like non-mainstream "cures" or the like should be removed on sight unless strongly supported by a good source. Yobol (talk) 16:20, 8 March 2012 (UTC)

After I posted this I thought more about it, and I don't really think its workable. It relies too much on editors being able to determine the harm or "contentiousness" associated with particular assertions. A better method is to make a sort of dummy's guide to common errors, with appropriate responses - perhaps based on the examples Sandy listed above. I've started work on a first pass attempt at User:Nathan/MEDRS. Nathan ^T 16:46, 8 March 2012 (UTC)

Related discussion

Folks here could probably contribute usefully to the discussion at Wikipedia talk:Manual of Style/Chemistry#Proposal to rename Current Events and edit the section on Resources on primary sources. WhatamIdoing (talk) 17:58, 3 March 2012 (UTC)

Primary sources

As I seem to be getting into continual conflicts over the "true" intention of MEDRS with regard to primary sources, I think adding examples of when primary sources would be allowed would be helpful in these types of discussions (i.e. if no reviews are available, etc). Thoughts? Yobol (talk) 16:22, 8 March 2012 (UTC)

From my perspective, the intent of the guideline is to prevent the abuse (not the use) of primary sources. I'm fine with primary sources being used to illustrate current medical knowledge and understanding. However, there has been a longstanding issue with the abuse of primary sources, in which a few carefully selected primary sources are editorially highlighted to rebut expert opinion, or to give undue weight to a minoritarian viewpoint. In my mind, this guideline is intended to address the abusive scenario, but not to prohibit the use of primary sources where they help illustrate the state of modern medical knowledge.

I think we should be careful about specifying "rules" for when primary sources are OK, because my experience suggests that explicit rules are gold for wikilawyers. Instead, we should clearly articulate the principles and reasoning behind the cautions about primary sources, and leave the case-by-case interpretation to a consensus of clueful editors. Just my opinion. MastCell ^Talk 18:32, 8 March 2012 (UTC)

@MastCell: I agree, hard rules is not the way to go here, as I agree they would be wikilawyered, and frankly, there would probably be too many exceptions to any small set of rules to be useful. However, I think examples of when primary sources can/should be used will help for everyone involved (as we can see in this very thread). Yobol (talk) 01:40, 9 March 2012 (UTC)

I have a feeling I've misunderstood, but should we be using primary sources when no secondary sources are available? I would have thought that's the kind of situation that we ought to avoid. Primary sources can be useful for illustrating a point made by a secondary source, but without a secondary source we've no way to judge due weight, and articles/sections based on primary sources tend to be a mess of OR. Jakew (talk) 18:44, 8 March 2012 (UTC)

I think there are instances where primary sources are OK. For example, I started total body irradiation, which I think is an important, article-worthy topic. But secondary sources dealing specifically with TBI are limited or non-existent, so the article is mostly based on a summary of primary sources. I think this is OK because it's not a particularly controversial subject, I think there is little dispute about how to use or interpret those primary sources, and I think our article uses the primary sources to illustrate (rather than rebut or question) current medical opinion. But I think we do need to be very careful on more controversial articles or topics; I'm just wary of being too prescriptive or dogmatic since I think there are examples where we can write decent capsule articles using primary sources. MastCell ^Talk 18:57, 8 March 2012 (UTC)

I'm not convinced, MastCell. A quick Google Book search turned up Chapter 18 "Total Body Irradiation" in "The Physics of Radiation Therapy" by Kahn F. ISBN 0781788560. Also, Chapter 14 "Total Body and Hemibody Irradiation" of "Perez and Brady's Principles and Practice of Radiation Oncology" ISBN 078176369X. PubMed finds 39 reviews with "Total Body Irradiation" in the title. In addition, there are a number of patient-information booklets online that (despite being normally discouraged by MEDRS) could be used to flesh out the patient-experience aspect of this topic. So the best secondary sources are available for this topic, though of course individual editors may be restricted in their access to them.

Also, although the article cites some primary research papers, I suspect it isn't using all of them as primary sources. Rather, it is picking some material from the introductory discussion within the papers. It isn't clear that all of the text is sourced and the final paragraph on Ewing sarcoma uses dated sources and one wonders if it is even worth mentioning any more.

So, although a talented knowledgeable writer can piece together a short article worth reading after browsing some primary research papers, I wouldn't hold this up as an example of a good exception to MEDRS typical advice. Colin°^Talk 20:08, 8 March 2012 (UTC)

The reason I had added to policy that "material" based purely on primary sources should be avoided was to not only address the viewpoint of reliability but also subsequent notability as a matter of speaking. If a particular sub-topic within a larger article (or for that matter the article itself) hasn't been addressed by reliable, secondary sources, then there is a prevalent question on whether the sub-topic being reported has notability worthiness. Clearly, avoiding primary sources doesn't mean never use. But editors have to give a strong consideration to finding secondary sources when both reliability and notability of an article or material (however non-controversial) is dependent purely on primary sources. Wifione ^Message 03:26, 9 March 2012 (UTC)

One appropriate use of primary sources is to fill in the blanks. Any given subtopic might be so trivial from the perspective of high-quality secondary sources that they don't bother to mention it, but still might be important or expected by our readers. For example, our biographies include the birth year for the subject (when known) and the person's location/nationality per WP:OPENPARA, even though many secondary sources about people who are merely semi-famous won't mention these things. WhatamIdoing (talk) 05:52, 9 March 2012 (UTC)

I'll offer up an example for discussion. Klazomania was a stub when a class project took it on. There is one statement about it in one review, but it meets notability. The students expanded the stub based on primary sources. I cleaned it up, trying to leave the use of primary sources only for descriptions of symptoms. I've no attachment to the article, had to clean up after students who are likely no longer editing-- it might provide a sample for discussion. SandyGeorgia 03:41, 9 March 2012 (UTC)

That's a great article. Of the six sources, we have two reviews, one diagnostic manual, one medical dictionary and two examples of a "case report and review of the literature". Since there are only a few case reports, you've described those cases using the reports. We've also got a few hypotheses presented and they are attributed to their author's primary-source texts with both in-text attribution and footnote citations. That's the extent of primary-source usage and one that fits with policy. This is also a good example of when it is misleading to categorise a whole document as a "primary source". This is a topic that has few good texts to draw on and where editors have done a good job with what is available. Colin°^Talk 09:33, 9 March 2012 (UTC)

• Just as a matter of the actual policy, nobody should be saying that MEDRS prohibits using primary sources (as it sounds like Jakew was thinking). That's just untrue. If people are saying such things even after being reminded that it's untrue, that's a pretty serious problem. And before anyone jumps up to edit this page (or WP:PSTS) to change this, please let people know with a thread topic and probably a RfC. These pages are longstanding and have been in place for years, with a lot of debate. Discussions should be substantive rather than cursory link to WP:MEDRS; that would seem more like wikilawyering. We also need to remember the old adage that the perfect is the enemy of the good. Not all of us are right next to world-class medical research libraries. Medical review journals tend to be locked up pretty tight behind paywalls. That doesn't mean that people are prohibited from contributing. This is not Scholarpedia; it is the encyclopedia anyone can edit and everyone knows it and the limitations which that involves. II | (t - c) 05:52, 9 March 2012 (UTC)

Just as a matter of the actual policy, let's quote it:

Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources. Secondary or tertiary sources are needed to establish the topic's notability and to avoid novel interpretations of primary sources, though primary sources are permitted if used carefully. Material based purely on primary sources should be avoided. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than to the original analysis of the primary-source material by Wikipedia editors.

Ok, so there's a difference between "prohibited" and "generally unwise" but let me tell you that every time I've seen someone argue that "... says you can use primary sources" they have been trying to do something wrong. To me, it is like driving on the correct side of the road. There are times when you need to drive on the other side, but you don't base your driving style on it. If you find yourself getting into an argument over using primary sources to present some fact or cover some topic, then it is likely you are on the wrong side. Like WhatAmIDoing's favourite "challeged" aspect to sourcing, nobody is challenging Sandy's text at Klazomania and I'm quite sure she's more than capable of gleaning something good from inferior material. There is a spectrum of source qualities and few sources are absolutely prohibited. We don't encourage the use of primary research papers, per policy, just like we don't encourage the use of abstracts or newspapers or press releases. There are times when these source types are helpful, mostly where the topic isn't well documented. I agree with WhatAmIDoing's "fill in the blanks" point. For example, in the total body irradiation article, look at this patient information leaflet. It is full of useful information that I bet nobody would ever mention in a review or a chapter in a medical text. I wouldn't use it to list the indications or effectivenes for various conditions, but we can use it to fill in the blanks. For the big article topics like obesity, cancer, epilepsy one should be easily capable of writing all that needs to be said using secondary sources. Colin°^Talk 09:03, 9 March 2012 (UTC)

I believe that the patient information leaflet would be considered a secondary source for information about TBI. Secondary is not another way of spelling "high-quality".

I am happy to use such "low-quality" (properly, non-academic) sources to support very general or basic information. I think it's good for readers to find a variety of types of sources, so that people who don't have a scholarly bent can find sources that are appropriate to them. WhatamIdoing (talk) 16:05, 9 March 2012 (UTC)

Sorry if that was confusing. I wasn't saying the PI leaflet was a primary source, merely that it was one of the types of sources we wouldn't recommend folk base an article on, like abstracts, and press releases, etc. Sometimes, you have to make do with what you can get. And sometimes, an atypical source can be exactly what you need: it is very difficult to get academic sources that discuss "what the patient goes through". Colin°^Talk 19:25, 9 March 2012 (UTC)

WebMD as a reliable source

The current guidelines explicitly name WebMD, stating that WebMD and other sources "are usually acceptable sources in themselves". I'd like to bring this into question, and propose removing WebMD from the list or changing the wording here. Lately I've been delving into more depth in material on which I am fairly knowledgeable, and finding serious problems with WebMD's research. Let me give two examples:

• The following page on WebMD, [6], says "One study has found regular use of products containing tea tree oil may cause gynecomastia", and then cites the following study: [7], which, if you read the source, provides insufficient evidence to support the claim made by WebMD. The way WebMD has cited the source suggests they merely read the headline of the article or possibly the abstract, rather than actually reading the article.

• The following page on WebMD: [8] makes the claim "black tea has the highest caffeine content", without citing any source. This is a topic I have researched extensively, as I maintain a page on RateTea about the caffeine content of tea. In particular, the following sources: [9] and [10] draw this into question. Also, although it's less relevant because it's not really a medical topic, the page also contains information about tea that is sloppy, i.e. that Pu-erh is "Considered a black tea", and that Wuyi is "One variety of oolong" (it is a region that produces oolong and other teas, not usually considered a variety).

Furthermore, I have contacted WebMD about the caffeine point, and I did not receive any response, nor has it been corrected. These may be only two examples, but they're examples of topics that I have researched the most thoroughly and know the most about. I suspect that if I really delved into depth, I would be able to find more examples of how the quality of scholarship on this site is lacking and is thus unsuitable for citing as a reliable source.

My questions are: do you think WebMD's quality is lacking relative to the other sources given? If so, we could remove it from the list. Or do you think that the quality of all these sources are relatively similar? In this case, it might be better to change the wording to "may be acceptable..." rather than "are usually acceptable..." Cazort (talk) 16:23, 3 April 2012 (UTC)

I have seen at least two instances of poor analysis and conclusions published in Cochrane Review and it is considered one of the highest standard sources. WebMD I imagine given it's size and scope has lots of health-care professionals writing for it, so some mistakes are inevitable. From my use of WebMD, they seem fairly comprehensive and accurate. If you see a clear error in WebMD, find a different good source to add a differing viewpoint or possibly replace the WebMD with it if it is a clear error on WebMD's part. I think removing WebMD as a recommended source is like basically throwing the baby out with the bathwater and will create more problems than it solves. Others may have a differing viewpoint though.--Literaturegeek | T@1k? 16:48, 3 April 2012 (UTC)

Hmm, you're right that even the best sources have problems. My impression of WebMD is an overall impression, not just based on these specific examples. My impression of WebMD is that it tends to present material that fits with societal "common knowledge", sometimes perpetuating myths without questioning, and that it does not show enough evidence of critical research for me to be comfortable with it being mentioned in the guidelines as it is here. I personally have a more favorable impression of Mayo Clinic, but I don't feel confident with my impression, it just is an intuition, and I have no opinion on the other sites mentioned. "Throwing the baby out with the bathwater" is less of a concern for me in this case because I don't find much unique material on WebMD, i.e. material that can't be found in higher-quality sources. If we leave WebMD and the other sources, I'd be more comfortable if we weakened the language to "may be acceptable..." or something similar. Cazort (talk) 18:19, 3 April 2012 (UTC)

I pretty much agree with LG: all of these are "usually acceptable", by which we mean that they are "sometimes not". Like any source, if they disagree with the general run of sources, or with sources that are higher quality and/or more appropriate (e.g., a source entirely about tea-growing regions for your Wuyi example), then they should be avoided. WhatamIdoing (talk) 19:22, 3 April 2012 (UTC)

I rarely use WebMD. There are concerns of conflict of interest per [11] but it is not the only source with one. While there are better sources I would not argue for an outright ban. Doc James (talk · contribs · email) 01:53, 4 April 2012 (UTC)

Hmm. I also would not propose an outright ban yet, but after reading this article, I'm feeling stronger about changing how we mention WebMD, relative to, say, Mayo Clinic (I haven't found any overt misinformation on Mayo Clinic, whereas I've pretty consistently found misinformation on WebMD when I've delved deep enough to assess the truth of the material there). I've had an impression that Mayo was more reliable and it's seeming an increasingly poor choice to leave them in a list that implies they're roughly on the same level in terms of reliability. I have not had much experience with the other sites in the list so I don't really have any opinion on them at the moment. Regarding wording, I think there's a huge difference in connotation between saying something is "usually acceptable" or "sometimes acceptable". My personal inclination would be to consider WebMD to be "sometimes acceptable". For instance, I tend to be very skeptical of its recommendations about drugs too. For example, it has a ton of pages on Benzodiazepines which implies they are suitable for long-term or general use, like this page that provides it as an option for bipolar disorder: [12], yet there seems to be a growing medical consensus that these drugs are only suitable for short term use for acute anxiety. Mayo Clinic expresses this: [13] "Benzodiazepines are generally only used for relieving acute anxiety on a short-term basis". The potential conflict of interest, which I did not know about until reading that article, would make me more concerned particularly because of this sort of discrepancy. Cazort (talk) 14:15, 6 April 2012 (UTC)

"Unless the primary source itself directly makes such a claim"

The sentence "Individual primary sources should not be cited or juxtaposed so as to "debunk" or contradict the conclusions of reliable secondary sources" is wonderfully clear, and I've been wondering whether to incorporate it in WP:PSTS (on grounds that it's a logical consequence of existing policy, and hence applies to all articles). However, the final part of the sentence ("unless the primary source itself directly makes such a claim") doesn't make sense to me. If a source debunks, contradicts, or otherwise comments on the conclusions of a secondary source then it isn't a primary source. It's a secondary (or tertiary) source. What is it intended to mean, and how can it be rephrased to better express that meaning? Jakew (talk) 09:58, 24 April 2012 (UTC)

Agree we should get rid of it. --Doc James (talk · contribs · email) 10:37, 24 April 2012 (UTC)

(outdent) It means, if a primary source directly criticises a secondary source, for example, a primary source, may point out undeclared conflicts of interests or perceived flawed methodology used by reviewers or confounding variables etc. The primary source would basically be saying the review by such and such group of researchers is flawed because,,,,. The primary source would have to specifically mention a secondary source by name. It is NOT the same as using a primary source such as an individual study to debunk a review of studies. I don't think the line should be got rid of but perhaps it is possible to better word it?--Literaturegeek | T@1k? 23:55, 24 April 2012 (UTC)

But many of these comments are not WP:DUE. For example we have this report by the AHRQ [14] which was criticised by one of the leaders of the religious movement in question [15] --Doc James (talk · contribs · email) 06:10, 25 April 2012 (UTC)

Ah. As I see it, whether a source is primary or secondary depends on how it is used. If we cite a source for its criticism of another source, then we aren't using it as a primary source. We're using it as a secondary source. That's why I find it really confusing to see it described as a "primary" source. I think we've got used to thinking "primary source = anything in a primary research study, secondary source = anything in a review", which is often convenient, but it's an oversimplification. As Doc James points out, though, there are frequently undue weight issues associated with citing primary research studies.

How about rephrasing: Individual primary sources should not be cited or juxtaposed so as to "debunk" or contradict the conclusions of reliable secondary sources. Explicit criticism of secondary sources, however, may be included if appropriate, but be sure to assess due weight. Jakew (talk) 07:32, 25 April 2012 (UTC)

I don't agree with the suggested "Explicit criticism of secondary sources, however, may be included if appropriate". We really should try to mostly not discuss the sources in article text at all. Sometimes it is useful to explicitly mention research studies and meta analyses and even literature reviews, in text, but generally we should be in the business of describing facts, not the means by which folk discovered or write about those facts. So I don't want to encourage editors to argue/prove a point in front of the readers.

Note that the guideline text does not say the primary source debunks the text of the secondary source, or the methodology or even the moral character of the authors (that would, indeed, make it a secondary source on the review it was debunking. It says it explicitly debunks or contradicts the conclusions in the secondary source. I think what we're trying to avoid is something like this garbage: "A Cochrane review in 2006 concluded that drinking XXX fruit juice had no protective benefits on YYY cancer. However, recent research shows that XXX fruit juice contains aaa compound [link to research study on fruit juice composition], which is protective against cancer [link to research study injecting aaa compound into rats]. Neither primary research paper directly or explicitly contradicts the conclusions of the review. However, if somone did a long-term study of people who regularly drink XXX fruit juice and those who didn't, and found and stated that it did indeed appear to have protective benefits on YYY cancer.... That recent research could potentially be used to debunk the old review. It would be explicitly debunking/contradicting the conclusions, but doesn't even need to mention previous reviews, so doesn't become a secondary source. Colin°^Talk 12:04, 25 April 2012 (UTC)

When we talk about directly making the claim that it de-bunks older works, I think we are looking for a paper that says something like, "The received wisdom in the field (as seen in every textbook for the last 100 years) says that patients should not be permitted to eat anything for 24 hours after major surgery. However, as far as we can make out, this popular old idea is based on zero evidence, and we've actually done a proper randomized, controlled trial, which we outline here, and the data produced says that the old surgeons' ideas about low diet is a bunch of bunk."

Our approach to using them (the textbooks and the RCT paper) would then look something like "Keeping the patient on a low diet after major surgery has been widely recommended,^[1][2] but newer research suggests that it may be unnecessary.^[3]"

If, on the other hand, we had the same study, but the authors made no direct mention of how their study supersedes the older works, then we would probably ignore it (for now): "Keeping the patient on a low diet after major surgery has been widely recommended.^[1][2]" WhatamIdoing (talk) 13:51, 25 April 2012 (UTC)

I disagree. I don't think there is any requirement for the new source text to explicitly name the debunked sources or to imply them with a remark like "every textbook for the last 100 years". The "de-bunks older works" comment falls into the same trap of confusing source text/work with the facts it presents. Criticism of older "works" might happen in a field like history, say. But research in medicine would tend to overturn previous "facts" rather than "works". Also, there's nothing the research authors can say or not say that makes any difference to the WP:WEIGHT we give to their findings. So the argument that, depending on what they say, we could ignore them or not, doesn't work. The guideline we are discussing is preventing original research based on primary sources. It is a separate issue to consider whether that primary research paper's conclusions have enough weight to warrant mention. Colin°^Talk 15:22, 25 April 2012 (UTC)

I have to say, I'm not comfortable with citing primary sources in either situation.

In the case that the authors assert that their study is more significant than the sum of previous work, their assessment is unlikely to be wholly objective. They may well be correct, but it wouldn't be surprising if their view of the importance of their work was a little out of proportion. We're all prone to thinking our work is more important than it really is. I would generally think it best to wait until a secondary source cites their paper and judges the significance of their work from a less biased perspective.

In the case that they make no particular assertion, I'd be very concerned if editors were to make that assessment themselves. True, much of the time the decision will be rational and in accordance with principles of evidence-based medicine, but many controversial topics are subject to widespread POV pushing, and it would be best to avoid long-winded debates full of original research about the relative importance of various studies. Again, why not just wait until a secondary source becomes available? It's unlikely to be that urgent, after all... Jakew (talk) 09:35, 27 April 2012 (UTC)

MEDRS

I think the guidelines has become too complicated for its own good, specifically about primary sources. For example this vital article http://news.harvard.edu/gazette/2006/12.07/11-dairy.html would be valid in most places but not here, this really concerns me and shows the flaws of Wikipedia and the urgent need for revision of the guidelines in general of any field. Helios solaris 16:11, 26 April 2012 (UTC)

It is complex because the problem is complex. The article you link is a newspaper/magazine type of article based on a talk someone gave in their lunch hour. Why do you think a serious encyclopaedia should base its articles on that kind of "I've got a radical new idea and have cherry-picked a bunch of studies that support it" kind of journalism. -- Colin°^Talk 16:36, 26 April 2012 (UTC)

Because if not it will fall behind. Helios solaris 17:33, 26 April 2012 (UTC)

I don't mind "falling behind" speculation and cherry-picking. The researcher in your source says that her hypothesis is unproven and not ready for prime time.

That said, this source is usable, for certain limited purposes. You could use it, for example, to support a claim that research is being conducted on the relationship between dairy consumption and hormone-sensitive cancers. You can't use it for a claim that any such link has been proven, or that readers should change their diets. WhatamIdoing (talk) 19:04, 26 April 2012 (UTC)

There is insufficient evidence of WP:DUE. We have much better sources discussing what research is ongoing. --Doc James (talk · contribs · email) 10:19, 27 April 2012 (UTC)

When you realise how hard it is to find secondary sources in the way that pleases Wikipedia and add it, even then it's suppressed, it becomes obvious that they are suspiciously superfluous. They exist only to subdue true evidence and strenghten the corporate establishment, just remember the tobacco industry do I need to say more. Critical thinking is clearly not welcomed here. --Helios solaris 16:25, 27 April 2012 (UTC)

There's nothing wrong with critical thinking, we just don't let it affect our edits. WP:NOR is long established policy for one very good reason: when "anyone can edit", some anonymous/pseudonymous editors will be incapable of critical thinking. If we allow it for you, we have to allow it for everyone, including those who can't think at all. Instead, we insist on wp:verifiability and secondary reliable sources. It works, though we have to keep explaining why we do it. LeadSongDog come howl! 19:34, 27 April 2012 (UTC)