Wikipedia talk:Identifying reliable sources (medicine)

Uptodate.com

There has been mention previously about the us of Uptodate.com as a secondary source. Wondering what people thought are about structuring references to this source? I have started siting both Uptodate and the primary research article so people without access can look at the primary source.

Doc James (talk) 15:31, 16 July 2008 (UTC)

UpToDate is a good source, but the best sections on UpToDate heavily reference the actual medical literature. As much as possible, we should use UpToDate as a guide to which literature is relevant and how to weight it,but citing UpToDate directly is a bit iffier since it's essentially one expert's opinion. That said, the site has an excellent reputation for accuracy and is a widely used professional resource, so it's not a matter of accuracy, more of how we approach things. MastCell ^Talk 15:55, 16 July 2008 (UTC)

I think UpToDate is a good resource, but heavily restricted even to many editors here, making it hard to access.

Jmh649 has started this discussion specifically because UpToDate is one of the secondary sources that cites PMID 814892. Now if we could find another secondary source that cites that paper, then I think this source would be preferable to UpToDate. JFW | T@lk 17:16, 16 July 2008 (UTC)

Or you could just use that primary source in good faith, and refer anyone who questions you to uptodate.

Should we set up a template, like for {{eMedicine}} ? --Steven Fruitsmaak (Reply) 17:25, 16 July 2008 (UTC)

Just one data point. I'd never looked into uptodate.com, so I thought I'd check it by looking at its coverage of the epidemiology of autism, an area I've worked on in Wikipedia. Uptodate.com's first sentence on the subject is "The prevalence of ASD in the United States and other countries has increased since the 1970s and particularly since the late 1990s"[1]. But this claim isn't true, and the source they cite (Williams et al. 2006, PMID 15863467) doesn't make the claim. What the source says is that the reported prevalence has increased, but it is not known whether the actual prevalence has increased; it could be that the changes are due to changes in diagnostic criteria or other factors. Admittedly this is just one data point, but still, wow, that's a pretty basic error. Accordingly:

• I'd like to second MastCell's suggestion to use uptodate.com as a source of plausible citations rather than to cite uptodate.com directly.
• If there is some dispute about whether to cite PMID 814892 (a 30-year-old letter to the editor?) I wouldn't rely on uptodate.com to resolve it.

Eubulides (talk) 17:46, 16 July 2008 (UTC)

Following up on my own comment: I see that later text in uptodate.com does make the point that autism's bureaucratic prevalence is not the same as its true prevalence, so I was too critical: it's more a major editorial error in uptodate.com than an actual content error. And it is just one data point. Still…. Eubulides (talk) 17:51, 16 July 2008 (UTC)

That actual reference from Uptodate JFW is referring to is: Lee L, Kumar S, Leong LC (1994). "The impact of five-month basic military training on the body weight and body fat of 197 moderately to severely obese Singaporean males aged 17 to 19 years". Int. J. Obes. Relat. Metab. Disord. 18 (2): 105–9. PMID 8148923. {{cite journal}}: Unknown parameter |month= ignored (help)CS1 maint: multiple names: authors list (link) Doc James (talk) 20:31, 16 July 2008 (UTC)

Mea culpa, dropped a digit. I have refactored your comment to include the citation rather than the entire abstract. A primary research study in 197 subjects remains problematic. JFW | T@lk 21:12, 16 July 2008 (UTC)

In follow up I would like to mention that the article from Uptodate was used to emphasis a point made by the systematic review. That the amount of exercise has an effect on the degree of weight lose. It was not used to "debunk" the systematic review. See page on obesity.

Doc James (talk) 15:25, 17 July 2008 (UTC)

uptodate is peer reviewed - here is what they say about there process:

Editorial policy

Authors

All topics in UpToDate are written by the listed authors in conjunction with a deputy editor. Authors are identified as experts by the Editors-in-Chief, our editorial staff, and the participating societies. Exceptions are guidelines from major societies, which are added to UpToDate in their original form. All material is originally prepared by the contributing author(s) whose name(s) and affiliation(s) appear in the upper left corner of each topic. This material is reviewed extensively by our physician editors and peer reviewers for accuracy and completeness of the literature search, and for consistency with all aspects of the editorial policy.

Doc James (talk) 17:53, 20 July 2008 (UTC)

There is no doubt that UpToDate is peer-reviewed, but it remains expert opinion in review form with an uncertain selection process. The content is also not permanent and the same URL may lead to content that is liable to change. If there is a recent review available in a "real" journal this is vastly preferable over UpToDate.

What does the panel think about using UpToDate as a "hidden" supportive source. For instance, a primary research study is cited on UpToDate, thereby confirming its relevance, so the editor who includes that reference makes a comment in the reference code (not visible to the general reader) that the reference is cited by UpToDate? Thereby we satisfy the demands of this policy without overburdening the reader with a resource they are unlikely to be able to access even by interlibrary loan? JFW | T@lk 07:47, 21 July 2008 (UTC)

Hidden sources give me the willies. Wikipedia should be open about its sources, even if the sources themselves are closed. Eubulides (talk) 07:58, 21 July 2008 (UTC)

I don't understand how this is really "hidden". We are actually citing the primary source, and justify its importance under the "reviews preferred" aspect of WP:MEDRS by saying that it was cited by UpToDate. So long as you then actually read the primary source, to meet WP:CITE#SAYWHEREYOUGOTIT, then I have no problem with this. WhatamIdoing (talk) 20:48, 22 July 2008 (UTC)

Sources the ordinary reader can't see are hidden to the reader. If there's a reliable review, and the reason we're citing the primary source is that the reliable review cited it, then we typically should be citing the reliable review and not the primary source. This is true regardless of whether the review is published in uptodate.com or elsewhere. There are some exceptions to this (for example, notable primary sources worthy of discussion directly in the article text), but the exceptions shouldn't be common; and even then, it's better to cite recent reviews in addition to citing the primary source directly. Eubulides (talk) 23:03, 22 July 2008 (UTC)

RSN that may interest some editors here

I have revived a question at RSN on pseudonymous authors of "letters to the editor" in otherwise reliable sources. It may interest some editors here, and as the only editor who supports this source is demanding responses from additional editors to demonstrate "consensus", I am looking for editors who are willing to read it and respond.

Note that I don't have a vested interest in the outcome; I just want it very thoroughly settled one way or the other. WhatamIdoing (talk) 18:06, 4 August 2008 (UTC)

Process for guideline status?

Does anyone know what the proper procedure is to have this adopted as a guideline? WhatamIdoing (talk) 17:27, 5 August 2008 (UTC)

Huge can of worms and pet peeve extraordinaire alert. There isn't one, there should be, there is no coordination, but some editors are vehemently opposed to any efforts to centralize and coordinate MoS. In terms of history, we (MEDMOS) were actually the only WikiProject to be subjected to a process. In the olden days, Radiant! tagged our page (historical, I think), and we had to jump through hoops to get added to MoS. We posted to several dozen other Projects, the Village Pump, etc. to garner widespread consensus for our guidelines. To my knowledge, no other MoS page has ever had to do that. When I proposed something similar at MoS, it was rejected. Summary: we have a mess at MoS where a few editors can put together a page and call it a guideline with no mechanism for gathering broad consensus. And I'm supposed to make sure FAs are MoS-compliant, per WP:WIAFA crit 2, even though guidelines often contradict each other. Back when I proposed this, someone started a MoS WikiProject, which was promptly ignored. I could find all of these links if I had to, but it would take me a very long time. Ugh. SandyGeorgia (Talk) 17:32, 5 August 2008 (UTC)

By the way, the place to get our reliable sources text included would be at WP:RS, but it was already moved out of there, back during the old WP:ATT debacle, so that is yet another can of worms alert. SandyGeorgia (Talk) 17:34, 5 August 2008 (UTC)

I'm sorry things are such a mess. So, Sandy, what would make you (and therefore the FA process) happy? Would you like me to declare this a guideline on the grounds that the editors of this page largely agree that other people ought to follow our excellent advice? Would you like me to pretend that such a thought never crossed my mind? Would you like me to invent an "RfG" process for you? WhatamIdoing (talk) 17:51, 5 August 2008 (UTC)

Should we make this a guideline?

It may not be perfect, but this page is in good shape and doesn't appear to conflict with any policies or other guidelines. Should this be an official guideline for Wikipedia? WhatamIdoing (talk) 22:03, 10 August 2008 (UTC)

• Support guideline status. --Arcadian (talk) 18:15, 5 August 2008 (UTC)
• Support guideline status, but awaiting Sandy's reply on consensus vs guidelines. JFW | T@lk 20:25, 5 August 2008 (UTC)

I don't know the answer to that; I wasn't one of the roadblocks in the past discussions, I'm not sure how you can overcome them, and if I had my druthers, our text here would still be part of the WP:RS page :-) One point, though, is that reliable sources are dealt with at WP:RS, while the template at WP:MOS is about style guidelines, so I'm not sure how we make a reliable source guideline part of manual of style guidelines, particularly considering certain parties fought to get reliable scientific sources removed from sourcing pages. Ummm ... and I've already been to ArbCom twice, so count me out when it comes to sorting out this little hot potato :-)) So far, it hasn't been a problem at FAC, but if it ever is, I may run screaming from the building. SandyGeorgia (Talk) 22:07, 5 August 2008 (UTC)

New info. Considering that Wikipedia:Scientific citation guidelines is part of the style guidelines, perhaps I'm wrong? Who knows the history of that page, and what they had to do? That would be a starting place. SandyGeorgia (Talk) 22:26, 5 August 2008 (UTC)

• Support: This should be a guideline simply because it encapsulates current best practices. There are still some tweaks that need to be made, but no deal-breakers. This should supplement (not replace) WP:V, WP:RS, WP:NOR, WP:MOS, etc, on medical articles, of course. Oh, as to process - just mark it a guideline. If someone strongly objects, we'll hear about it soon enough. MastCell ^Talk 22:15, 5 August 2008 (UTC)

Sandy, you're right: This is really a content guideline, not a style guideline. (We should probably put a link to WP:SCG somewhere on the WPMED pages.) Perhaps if there are no objections by tomorrow, then we'll make this change. WhatamIdoing (talk) 17:09, 6 August 2008 (UTC)

Yes, it's troubling that content and sourcing guidelines are being added to style guidelines, but considering the chaos that is MoS and the lack of a method for sorting MoS issues, I don't know how to go about solving that. Someone really needs to take on WP:MOSCO and invigorate the entire process of fixing MoS, but I don't have time. SandyGeorgia (Talk) 17:15, 6 August 2008 (UTC)

So with a bit of looking about, I've found that the closest thing we have to a proper process is {{proposed}} or an RfC. I've chosen the RfC, because it seems more appropriate for a page that is widely used and not really "under construction". Mostly I don't want someone to be able to say later, "Well, it's only a guideline because you say it is, and I disagree, so therefore it's not a guideline." WhatamIdoing (talk) 22:07, 10 August 2008 (UTC)

• Support and should be linked to from WP:RS from a new sub-section in the "Reliability in specific contexts" section. I note that the Wikipedia:Scientific citation guidelines is mostly style with a little bit of "when to cite" and doesn't venture into the source-selection business. We should keep content/sourcing guidelines separate from style guidelines. This doesn't belong in MoS. Colin°^Talk 12:50, 10 August 2008 (UTC)

• Against. The idea that reviews are somehow more reliable than original publications promoted in this paragraph: "In general, Wikipedia's medical articles should be based upon published, reliable secondary sources whenever possible. Reliable primary sources should be used with great care because of the potential for misuse. For that reason, edits that rely on primary sources should only make descriptive claims that can be checked by anyone without specialist knowledge. Where primary sources are cited, they should be presented in a manner which hews closely to the interpretation given by the authors or by published, reliable secondary sources. Primary sources should not be cited in support of a conclusion which is not clearly adduced by the authors or by reliable secondary sources, as defined above (see Wikipedia:No original research)." - is batty. The reviews are written by the same people who do the original research, and they are subjects of the same personal prejudices and biases. Every warning and rule applicable to the primary sources equally applies to the secondary sources: "Reliable secondary sources should be used with great care because of the potential for misuse. For that reason, edits that rely on secondary sources should only make descriptive claims that can be checked by anyone without specialist knowledge. Where secondary sources are cited, they should be presented in a manner which hews closely to the interpretation given by the authors. Secondary sources should not be cited in support of a conclusion which is not clearly adduced by the authors (see Wikipedia:No original research)." Paul Gene (talk) 20:42, 10 August 2008 (UTC)

Paul, the discussion about how to apply WP:PRIMARY to medicine-related articles is higher up on the page. WhatamIdoing (talk) 21:09, 10 August 2008 (UTC)

Review articles are much to be preferred because they provide the context often lacking in primary studies. They weed out primary studies that have not been replicated. And yes, in a small minority of cases they propagate bias. But the number of biased reviews pales in comparison to the number of primary studies that are poorly designed, do not prove or disprove the null hypothesis. have significant methodological problems etc. We have, above, arrived at a widely supported version by Eubulides that takes all these factors into account. It is still possible to cite primary studies, but preferably backed up by a secondary study that proves its relevance and acceptance as a representative piece of scientific enquiry.

For those who are surprised by Paul's vote, this thread on WT:MEDMOS started it all. JFW | T@lk 21:26, 10 August 2008 (UTC)

The argument that secondary sources are no better or less risky than primary sources, and so should not be preferred, goes against WP policy in WP:NOR. This guideline is unable to deviate from such policy. Colin°^Talk 09:46, 21 August 2008 (UTC)

Your blind insistence that secondary sources are better goes against WP policy in WP:NOR, which states Deciding whether primary or secondary sources are more suitable on any given occasion is a matter of common sense and good editorial judgment, and should be discussed on article talk pages. Paul Gene (talk) 09:03, 24 August 2008 (UTC)

Selective quoting and exaggeration do your case no favours. You know fine well that WP:NOR prefers secondary sources and that WP:MEDRS is merely repeating the same guidance. Primary sources can be used "with care", which is exactly what "common sense and good editorial judgement" are for. Calling me blind is rude. I haven't said they "are better" in the sense you imply -- that they are always better in all circumstances. Please can you point out where this proposed guideline deviates from policy, or fails to accurately reflect that policy. One could add a "use common sense and good editorial judgement" caveat after every guideline in WP, but it is already enshrined in policy. Colin°^Talk 10:00, 24 August 2008 (UTC)

• Support Great article that deserves wider use. --Ronz (talk) 18:58, 20 August 2008 (UTC)

• Support. Would make a great addition to RS. -- Fyslee / talk 19:34, 20 August 2008 (UTC)

• Oppose. It is unclear what are independent, or third party, sources for the purposes of Wikipedia. --Mihai cartoaje (talk) 07:17, 21 August 2008 (UTC)

It is unclear whether this is a criticism of a supposed requirement in MEDRS for "independent, or third party, sources" (and such a requirement doesn't exist since those words don't appear in the guideline) or of the lack of such a requirement in MEDRS. Can you make your opposition clearer, so that any issues can be resolved. Thanks. Colin°^Talk 09:46, 21 August 2008 (UTC)

Unclear? WP:PSTS doesn't cover it in enough detail? --Ronz (talk) 21:43, 21 August 2008 (UTC)

A look at the comments on this user's talk page suggests that Wikipedia content policies in general are unclear to this user. WhatamIdoing (talk) 00:48, 22 August 2008 (UTC)

Strongly suggest that this users vote is pretty much ignored, almost certain a vandal who has already tried to get Jfdwolff desysoped. —[[::User:Cyclonenim|Cyclonenim]] ([[::User talk:Cyclonenim|talk]] · [[::Special:Contributions/Cyclonenim|contribs]]) 22:09, 22 August 2008 (UTC)

• Support as per MastCell and Colin. Eubulides (talk) 09:10, 21 August 2008 (UTC)

• Strong support, I've already been using this as a guideline, anyway. —[[::User:Cyclonenim|Cyclonenim]] ([[::User talk:Cyclonenim|talk]] · [[::Special:Contributions/Cyclonenim|contribs]]) 22:03, 22 August 2008 (UTC)

• cautious/weak Support-ish - I know where Paul Gene is comng from, and (especially in psychiatry) seen quite a bit of schism in views depending on which secondary source one looks at. I do like the idea of acknowledging that the same is possible for secondary sources, though generally less likely. And that if possible, try and get an expert in the area. Cheers, Casliber (talk · contribs) 22:13, 22 August 2008 (UTC)
• Support. Any real issues can easily be addressed on a case-by-case basis and the scholarly consensus is usually well-represented in any well-reviewed reliable source, including lit reviews and whatnot. WLU (talk) 22:37, 22 August 2008 (UTC)

• Oppose. I understand and share the desire to have a guideline such as this, but I cannot support it. I have read far too many mediocre review articles that are a giant step backward from the primary sources they cite. (Also, I take issue with the Wikipedia idea that synthesis is a form of original research. Both synthesis and original research are original work, and therein lies the confusion.) --Una Smith (talk) 05:01, 23 August 2008 (UTC)

  You're opposing a guideline because it reflects Wikipedia policies, which you state you disagree with ? There is mediocrity in every area of publication; that doesn't mean we engage in synthesis and original research on Wiki. SandyGeorgia (Talk) 05:12, 23 August 2008 (UTC)

Certainly not. I oppose this guideline because in my opinion it is not an improvement on Wikipedia:Reliable sources. --Una Smith (talk) 18:45, 24 August 2008 (UTC)

• Support. We've worked on this for years; it's ready. SandyGeorgia (Talk) 05:12, 23 August 2008 (UTC)

You barely worked on it for a month. It is not ready. There is still an active discussion going on. Paul Gene (talk) 09:03, 24 August 2008 (UTC)

• Support. The document has been discussed at depth and at times, er, robustly. I don't see any active discussions going on, and am confident that we can handle them as and when they arise. Nmg20 (talk) 17:35, 24 August 2008 (UTC)

Systematic reviews and primary sources

This edit removed the following sentence from Wikipedia:WikiProject Medicine/Reliable sources #Article type without discussion:

A systematic review is both a primary and secondary source—it summarizes other papers but it does so in order to research the field and possibly come to a novel conclusion.

with the comment "this is really not true -- a good systematic review basically summarizes previous findings only. Looser reviews do the more primary work." Possibly-controversial edits like this should be discussed on the talk page first, so I took the liberty of reverting the change and moving the text here, for discussion. Eubulides (talk) 02:20, 15 August 2008 (UTC)

I actually support removal of this sentence. In the vast majority of cases, systematic reviews make no new observations at all. Non-systematic (or "narrative") reviews occasionally reflect expert experience without an evidence base, but they remain reviews in their editorial sense. JFW | T@lk 02:22, 15 August 2008 (UTC)

I honestly didn't think that removing this sentence could possibly be controversial, as it seemed to be superflous in addition to being false. I thought about simply cutting the word systematic from the sentence, but I figured that might be controversial! II | (t - c) 02:42, 15 August 2008 (UTC)

I also support the removal of that sentence since it differs from the definition of the term in the article at systematic review and from the use of the term in the lead paragraph of this page. --Jack-A-Roe (talk) 02:57, 15 August 2008 (UTC)

The sentence in question has been present in nearly its current form since the talk page was created in 2006. It was added in the very first change to the article, by Colin, the original author of the page, here. I'll leave a note on Colin's talk page to ask him to weigh in on this issue. Eubulides (talk) 05:31, 15 August 2008 (UTC)

Feel free to correct my mistakes -- I'm just an amateur. I bow to the folk with training in this subject! I wrote that a while ago and might have worded it differently now. I've always hoped, in fact, that someone with training/qualifications in this area would rewrite much of MEDRS, while keeping those aspects that are unique to WP. Re-reading Greenhalgh's How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses), I can see how I thought the "novel conclusion = primary" might occur with such papers. If you guys think that is rare then clearly the sentence isn't useful. Here's my impression: A narrative review by an expert might be likely to have a conclusion that the expert already "knew" before he even started working on the paper. He's set out to describe rather than systematically discover the current situation. A systematic review is an objective exercise in examining the published research and the results might surprise the author(s). Such a review might gain access to the raw data from published trials, and other less accessible input (see the "Checklist of data sources for a systematic review" in Greenhalgh's paper). If the review contains a meta-analysis or other synthesis of results, then arguably it produces its own conclusion (even if this is similar to and adds weight to what was "known" already). The publication of a Cochrane review can make the newspapers, so it must be saying something we haven't heard before or with as much conviction. Still, I agree that it may be simpler to just consider such reviews to be excellent secondary sources. Colin°^Talk 18:33, 15 August 2008 (UTC)

Thanks for following up. In rereading the article it seems clear to me that something needs to be said here about systematic reviews, as systematic reviews are mentioned in the lead and therefore need to be in the body as well. How about if we remove the phrase about primary and secondary source, along with a newly-redundant phrase, as follows:

A systematic review is both a primary and secondary source—it summarizes other papers but it does so in order to research the field and possibly come to a novel conclusion.

Eubulides (talk) 20:09, 15 August 2008 (UTC)

Eubulides, edit away and expand if you can. Please, the more you can add to this proto-guideline the better IMO. Colin°^Talk 20:21, 15 August 2008 (UTC)

OK, thanks. For now the consensus seems to be that this change is OK, so I put it in. Expanding will take more work.... Eubulides (talk) 18:57, 22 August 2008 (UTC)

I support the direction you seem to be taking with your edits above. In my mind systematic reviews are not given their due. They should not be lumped in with primary sources, or with summary reviews of the literature that frequently precede the presenting of original research in a journal article. The latter is often self-selected to support the hypothesis the researcher is evaluating, but I frequently see these reviews (often in abstract form) used as a secondary source. —Mattisse (Talk) 20:32, 15 August 2008 (UTC)

Hmm. I don't oppose the addition of that sentence, but I think it is bit superflous. Systematic review is wikilinked in the lead. We don't really need define it in MEDRS. Incidentally, Mattisse, I actually previously supported emphasizing systematic reviews, but if I recall correctly JFW wasn't really happy about that(not sure what Eubulides thoughts on the matter were). Then again, the page on systematic review already states that these are the standard in evidence-based medicine, which works just as well. Actual medicine guidelines > MEDRS in my mind. I actually think modularity in these articles is best -- keep the things which are strictly MEDRS here, and fork off tangents and definitions elsewhere. And, no offense, but that sentence is a "duh" sentence. It's a "duh" statement that in most cases, reviews will synthesize the literature and come to their own, possibly novel, probably not novel, conclusion. If you want to add that sentence, you're going to have to at least strike the word systematic from it, Eubulides. II | (t - c) 19:07, 16 August 2008 (UTC)

I have never opposed the use of systematic reviews, and indeed I feel that we should place much emphasis on them. But they tend to be relatively narrow in scope and broad in methodological thoroughness. Hence, "narrative reviews" are great for broad sketches, while systematic reviews are better at answering specific questions. JFW | T@lk 21:56, 17 August 2008 (UTC)

• There seems to be some confusion here, as the proposal is to remove some text, not to add it.
• Removing the entire systematic-review sentence would be unwise, as the containing paragraph discusses all sorts of article types, and discussing other article types while omitting mention of systematic reviews (a preferred source) would be very odd.
• It's not a "duh" statement to define a systematic review; we can't expect all Wikipedia editors to know what it is. The paragraph already defines general reviews, and it's useful for it to make a point about systematic reviews and what makes them different. If you prefer a different definition for "systematic review", please propose something.

Eubulides (talk) 23:12, 16 August 2008 (UTC)

I don't think it needs to be defined, and I don't think they come to novel conclusions. Generally a systematic review will have one of two conclusions: 1) Not enough evidence, 2) Enough evidence, some consistent result. The paper pointed by Colin has good summary of them, and I think it would be nice to actually reference that paper in this article. A good definition would note that a systematic review uses a predefined, replicable methodology to select all articles from a certain time period, with predefined inclusion criteria. II | (t - c) 01:15, 18 August 2008 (UTC)

WP:MEDRS #Article type defines/discusses general reviews (which are mentioned in the lead); for consistency shouldn't it also define/discuss systematic reviews (also mentioned in the lead)? It would be odd for the lead to mention a topic not discussed in the body. The current (and proposed) wording doesn't say that systematic reviews always come to novel conclusions, only that they "possibly" come to novel conclusions. Again, if you could propose specific wording to replace the proposal above, that would help drive the discussion forward more quickly. Eubulides (talk) 19:42, 18 August 2008 (UTC)

Although general (journalistic or narrative) reviews are a good introduction to a topic, they should be used with a consideration of the sources cited by the author. When citing information on the effectiveness of certain therapies, consider looking for a systematic review. [Alternative: Systematic reviews are the considered to be the gold standard in evidence-based medicine in regard to complex medical questions.] A systematic review uses a predefined, replicable methodology to select all articles from a certain time period in order to answer a specific question. This it thought to reduce bias and increase thoroughness, since articles which do not support the researcher's preconceived conclusion must be included if they fit the inclusion criteria.

— [2]

II | (t - c) 20:06, 18 August 2008 (UTC)

I assume this is intended to replace the last 3 sentences of WP:MEDRS #Article type? If so, the proposed replacement has two major problems. First, it disses general reviews far too much. General reviews, when done well, are better sources for Wikipedia than systematic reviews are, since they match Wikipedia's encyclopedic aims better. Second, WP:MEDRS #Article type is descriptive: it does not (and is not intended to) give advice. Advice is in WP:MEDRS #Some definitions and basics and proposed changes to advice should be made there. Eubulides (talk) 20:13, 18 August 2008 (UTC)

Sorry, it is intended to replace the last sentence only. I tweaked it slightly. General reviews are only as good as the sources (or perhaps the credentials) of the author. If the review cites only one primary study for a claim, its reliability is not much better than that primary study. I don't understand how general reviews are more encyclopedic. They are certainly not more reliable than systematic reviews. As far as "advice", I think that's your opinion. This page is a guide, and I think it should read as such. But if you prefer, the second sentence can be changed to "" Any other advice? II | (t - c) 20:28, 18 August 2008 (UTC)

(Undent.) As discussed previously, editorial judgement is advised when selecting sources and citing their contents. But secondary sources will generally trump primary sources for most citations, and I don't think we should try to codify for exceptions. I dispute the claim that systematic reviews can answer complex medical questions; in the vast majority of cases, the data is so insufficient or contradictory as to allow for the drawing of definite conclusions about complex medical questions. Back to you, ImperfectlyInformed. JFW | T@lk 20:50, 18 August 2008 (UTC)

Yes, and here we go again. I'm not codifying for exceptions. I'm just pointing out that some attention should be paid to the sources cited by a review. That's probably the most important part of editorially analyzing reviews. As far as systematic reviews actually answering the questions, perhaps you're right. But they are considered to have the best chance at answering these types of contentious questions, and they are the best at putting the question into proper context, since they attempt to include all context. II | (t - c) 21:17, 18 August 2008 (UTC)

I think we're actually much in agreement regarding the place of various types of publications. In a world full of evidence, systematic reviews (especially of the Cochrane flavour) would answer any question. JFW | T@lk 21:31, 18 August 2008 (UTC)

agreed. Systematic reviews of this sort are in fact the goldstandard and the very model of an authoritative secondary/tertiary source. ; the Cochrane series are not the only ones--there are also other formal consensus statements issued by appropriate groups. When they don'ta gree, which is not unheard of, w need only present the different ones equally. I wish other fields were as well organized for such things. DGG (talk) 04:14, 19 August 2008 (UTC)

I agree with Eubulides and JFW that the proposed text disses general reviews too much. I don't really know what the "should be used with a consideration of the sources cited by the author" is supposed to add. Many editors won't have the ability/experience/training to judge whether the review's sources are balanced/good (and so just have to trust that a review published in a good journal is probably good). Those that are capable of judging the quality of a review probably don't need to take advice on this matter from Wikipedia! I wonder if II you are thinking too much about arguing about the efficacy of therapies rather than trying to write a rounded encyclopaedic article on a subject. A good general review provides far more than just an "introduction to a topic". In the ketogenic diet, I was able to source whole paragraphs to narrative review sources such as Clinical Aspects of the Ketogenic Diet. I used two systematic reviews: Ketogenic Diet for the Treatment of Refractory Epilepsy in Children: A Systematic Review of Efficacy and Ketogenic diet for epilepsy (the former funded by US insurance companies in 2000 and the latter from the Cochrane Database of Systematic Reviews 2003). Both answered only one question, "does it work", and both said "weakly yes, but there are no randomised controlled trials" and are dated since such a high quality trial was finally published in 2008 (PMID 18456557). Both those systematic reviews read much like a primary research paper (background, methods, results, discussion, conclusion) and are so focussed that they aren't any use for the rest of the WP article.

Guidelines

I agree with DGG about the "other formal consensus statements issued by appropriate groups". Currently, MEDRS doesn't mention clinical guidelines. Two excellent sources I have used are the guidelines for Scotland and England and Wales. I know less about the US but I have seen consensus statements begin published (self or in journals) by various medical professional bodies and by some of the more respectable charities. Could we add some mention of this to MEDRS? Colin°^Talk 08:28, 19 August 2008 (UTC)

Guidelines tend to be limited in geographical scope. NICE does not pretend that its guidelines are valid outside England and Wales, and therefore not everything recommended by that body is therefore necessarily representative of practice in the rest of the world. An example would be the risk stratification algorithm for warfarin in people with atrial fibrillation; this is a novel system not used elsewhere. Similarly, its guidelines on computed tomography of the brain after head injury may differ substantially from similar guidelines elsewhere. As a result, I prefer to use reviews in preference over guidelines; that said, they can be excellent sources of information (e.g. the British Society of Gastroenterology guideline at coeliac disease). JFW | T@lk 18:49, 19 August 2008 (UTC)

• I like the idea of mentioning guidelines, but agree with JFW that the area's a bit tricky. In Chiropractic I've had to deal with the problem that there are three competing groups of guidelines for treatments used by chiropractors: one by the majority "mixer" chiropractors, another by the minority "straight" chiropractors, and another being the mainstream medical guidelines. The three groups differ markedly in their results. Even if we restrict ourself to mainstream medical guidelines developed by national governmental organizations, guidelines in different countries disagree markedly; see Murphy et al. 2006 (PMID 16949948).
• My experience is that guidelines tend to be less useful than reviews as sources to Wikipedia articles than reviews are, not necessarily because they are lower quality, but because their goal (advice about treatment) is more distant from Wikipedia's goal (encyclopedic coverage) than the goal of reviews (broad and balanced coverage of a topic). Though of course guidelines often have good reviews within them.

Eubulides (talk) 19:19, 19 August 2008 (UTC)

Guidelines must be carefully evaluated. An institution like the Mayo Clinic may offer "guidelines" meant for the general public without any attempt to source or valid the claims. —Mattisse (Talk) 20:03, 19 August 2008 (UTC)

Again, editorial judgement is advised. The NICE guidelines, too, are sometimes expert/experience based rather than supported by evidence because such evidence is often lacking. Also, Matisse, we are talking about professional guidelines here, adherence to which is often mandatory and a yardstick by which professional performance is meausured, as opposed to resources for the general public that cannot legitimately be called "guidelines" but rather "advice" or "tips". JFW | T@lk 20:19, 19 August 2008 (UTC)

Well, psychology articles are notoriously awful, but these sorts of references are often used in articles on topics in psychology. I would like to see a firm way of discouraging it. —Mattisse (Talk) 20:23, 19 August 2008 (UTC)

Sounds reasonable. JFW | T@lk 20:35, 19 August 2008 (UTC)

I have to agree with matisse, some of the guidelines I have seen put out by otherwise reputable bodies (stocked with professionals) have been pretty poor at times. Maybe a good caveat WRT secondary sources is be especially careful if one finds two or more sec. sources putting forth substantially variant positions/findings/information etc. Cheers, Casliber (talk · contribs) 22:29, 22 August 2008 (UTC)

Funny one I found - in australia (and apparently America), screening tools are fait accompli these days and promoted as such, yet the data seems to tell a different story..actually there was another BMJ paper which was even more cautious. Cheers, Casliber (talk · contribs) 22:34, 22 August 2008 (UTC)