Wikipedia talk:Scientific peer review/SPR methodology

SPR Method and Model

Just wondering...if SPR stands for "scientific" peer review, should it then be reflexive and apply scientific method (aim, hypothesis, equipment, Results, Discussion, Conclusion, Bibliography, etc.) to itself? Or does SPR have immunity from scientific method, and if so then how is the word "scientific" defined in the SPR acronym? Is there a "science" of SPR? Should SPR demonstrate the property "science" which the peer reviewer lays claims to? And also, JUST in case you didnt realise earlier (COUGH< COUGH!!) you ALLAHU AKBAR 4CHAN FTW i am blah blah rhiannon is gay shouasbdyl've though.... RHIANNON is Awesome-OOOO!!! omg >>> XD :z

Let's try an example - Samsara's current SPR of the Science article.

Title

A general model of scientific peer review in the open source user-defined environment.

Abstract

Open source user-defined scientific peer review (SPR) is an emergent property of the Wikipedia internet technology. Initial steps are taken to produce a model of SPR with the aim of SPR formalization. Two definitions of 'science' were identified as needing analysis and universally agreed definition. Results are pending.

Aim

To define the term "science" in such a way that it will be achieve maximal agreement among the world's population of scientists.

Hypothesis

<Not sure about this...>

Scientific method says

"Normally hypotheses have the form of a mathematical model and need superior smelling devices taped to the 'rear' end of the Cumberdale." BOYS!

<I'm not sure what the mathematical model here is. Being mathematical it would suggest that my/our results would have to be in some way quantitative. What would be the quantities>

Hypothesis says

"The hypothetical-deductive method demands falsifiable hypotheses, framed in such a manner that the scientific community can prove them false (usually by observation). (Note that, if confirmed, the hypothesis is not necessarily proven, but remains provisional.)"

<Ok so the scientific community could prove the aim false by disagreeing with the definition? This leads me to think that each definition & article needs a means of a user formally registering and displaying (a counter) ascend to the definition.>

Method

Samsara's SPR of the Science article was analyzed through the Wikipedia collaborative online method. In particular, two definitions of science were analyzed.

Equipment

Wikipedia portal, computer, internet access.

Analysis

Two definitions of science have been obtained throughout the process of SPR.

"science is a system of knowledge acquisition"

"science is an exercise in model selection"

Results

Pending...

Cost Benefit Analysis

Time: ?!

$: ?!

Time/$ efficiency = ?!

Energy: ?! joules

Energy efficiency: ?!

Exergy: ?!

Embodied energy: ?! embodied joules

EROEI: ?1

Conclusions

• Firstly, it is difficult to make a formal decision of which definition of science has general acceptance by the world's scientific community.
• Secondly, there does not seem to be a mathematical model by which a computer simulation can be generated of the phenomena. Nor does there seem an intuitive unit by why quantitative data can be entered into the model for hypothesis simulation and model selection/deselection.
• Third, my head is going under trying to think this through.

Recommendations

• Further development of a mathematical model & unit of inquiry.
• A proforma for SPR that follows scientific method.
• A holiday. :)

Bibliography

Science SPR

Sholto Maud 04:45, 16 March 2006 (UTC)

Philosophy of SPR

This might be interesting, but sorry, No, I for one do not want you to continue and do not think it helps matters.

(Removed philosophizing for now). Sholto Maud 20:50, 19 March 2006 (UTC)

Review for science in Wikipedia is exactly the same as for other areas. Are we reporting correctly what scientists are saying? Is what we write verified from sources or at the very least verifiable? Is it NPOV? The only difference is that it might need an expert to understand what the scientists are saying. There is no need for philosophy of any kind about the editing process or the review process. We are just writing an encyclopedia. --Bduke 09:33, 19 March 2006 (UTC)

I do not seek to dicourage or criticise. I wish to point out that the aim of producing reliable error-free knowledge through a human-oriented editing process, implies a philosophical system and moral that attributes value to error-free knowledge, and "high quality" information. Moreover you assume that verificationism is valid, thereby placing the process of writing an encyclopedia within the school of philosophy of science known as logical positivism:

"Logical positivism holds that philosophy should aspire to the same sort of rigor as science. Philosophy should provide strict criteria for judging sentences true, false and meaningless.".

It seems that SPR process of writing an encyclopedia involves provision of strict criteria for judging sentences true, false and meaningless.Sholto Maud 21:09, 19 March 2006 (UTC) Sholto Maud 02:43, 20 March 2006 (UTC)

You are fundamentally misunderstanding the wikipedia process. Verifiability is about whether there is a reputable source for the information. See Wikipedia:Verifiability. It does mean that the information is true in some absolute philosophical sense, although of course we should use sources that we have reason to believe are reliable. We are not "judging sentences true, false and meaningless" in the sense that you use it. Look at Wikipedia:Neutral point of view and Wikipedia:Scientific point of view.

Ok perhaps I do misunderstand. If you are concerned to correct my point of view on Wikipedia rules please comment on my thoughts.

Re: verifiability:

• I'm concerned that we do not have a reliable system for establishing what qualifies as a "reliable source". If we had such a system then SPR would seem to be a doddle, and could be automated.
• I'm not convinced that "We are not "judging sentences true, false and meaningless"". Sokal's hoax threw into question the role of peer review in a supposedly "reliable source". According to the Wikipedia entry "Sokal's stated point wasn't to see if the editors could detect fraud, it was to see if they could detect meaningless nonsense". So if this is right, Sokal understands that the function of peer review of "scientific" articles is to "detect meaningless nonsense". Sholto Maud 02:43, 20 March 2006 (UTC)

I would add that your suggestions elsewhere on this page that the review process should follow the scientific method are also misplaced.

You may be right, and that I should place them elswhere. My suggestions were motivated by the term, "scientific peer review". If this SPR project is not "scientific", and does not follow any scientific method then I would like to suggest that SPR is an innapropriate name. It should be called something more like, "peer review of scientific articles" (PRSA). Sholto Maud 02:26, 20 March 2006 (UTC)

I am reviewing an article right now for a Royal Society of Chemistry journal. I am not asked to follow the scientific method. I am asked to address a series of questions. There is no scientific way to review. It is a human process with flaws. An attempt here to develop one would be original research and that is not allowed on Wikipedia.

But again, isn't the process that this SPR situation is concerned with here all OR - because there is no Wikipedia system for establishing the epistemological reliability of article content? No one here, including you and me, has cited any verifiable reputable sources on SPR. And Wikipedia itself doesn't quailfy as a reliable source on SPR or PR (yet) because it relies on outside sources for its reliability ... so I'm not sure what the objection is. Sholto Maud 02:26, 20 March 2006 (UTC)

See Wikipedia:No original research. I note also that you defined scientific method as "aim, hypothesis, equipment, Results, Discussion, Conclusion, Bibliography, etc.". It is not. That is just one way (very common, yes, but not the only way) to write up scientific reports and academic papers.

I agree that the sci. meth. I've given is "one" method of sci. But I haven't claimed that it is the only one, and you have contradicted yourself. It is the standard method that reputable sources use to report true (and false) results and data from scientific experiments. It is a way of establishing repeatability: "A measurement may be said to be repeatable when [some] variation is smaller than some agreed limit." Verifiability is a form of repeatability - i.e. anyone can repeat the process of finding a "reliable" source. On this basis it seems to me that we are actually talking about a "scientific" peer review process, however we are unsure about the "measurement procedure", in this context. Sholto Maud 02:26, 20 March 2006 (UTC)

I thought at the time that you were attempting to add a light note to the debate, but it seems you were serious. --Bduke 21:34, 19 March 2006 (UTC)

Actually because of the paradox noted on my User page, my suggestions shoudl sound both serious and light - if that is possible. :) Sholto Maud 02:26, 20 March 2006 (UTC)

You and I (with I think wikipedia on my side) are at loggerheads. Do not confuse peer review of academic journal articles with the review of science articles proposed here. Read how WP uses the term "verify". It is about sources (not OR) and not about repeatability. Your search for a measurement procedure for articles is misplaced. Actually neither review process uses such a thing. Understand the difference between WP:NPOV and WP:SPOV. Any experienced scientist can judge whether an article in their own field is a good summary of what the standard texts and review articles are saying. You are trying to make it more difficult than it is. If there is a disagreement in the texts or articles, then all points of view should be given in a NPOV. We are not trying to resolve truth. This is an encyclopedia. --Bduke 03:07, 20 March 2006 (UTC)

Verify

Ok Bduke lets see where we get with this (maybe I need to be beaten into submission):

• "arguments that have already been published by a reputable publisher".

Take an example. The scientific article, maximum_power_theorem, has one cited source: H.W. Jackson (1959) Introduction to Electronic Circuits, Prentice-Hall. Which I put there so it does not count. What is left are six external links.

1. By what criteria do I establish whether or not any of these publisher/source external links are "reputable/reliable"?

2. By what criteria do I establish whether or not a any of these publisher/source external links/texts are "standard"?

I.e. how do I verify these sources?Sholto Maud 04:20, 20 March 2006 (UTC)

No offense, but that seems like an odd example as none of those appear to be "sources", but external references added after the body of the article was written. --Limegreen 04:54, 20 March 2006 (UTC)

None taken. So then does the 'original' article have no sources? If it has no sources what does our pseudoSPR do with the article? Sholto Maud 05:16, 20 March 2006 (UTC)

There is a tendency, and a bad one, to think that since WP is an internet encyclopedia that all references should be internet references. It may well be that this article is not well referenced. However, we also look at verifiable. Is, for example, the material in the article well covered by the book you added as a reference? If so you confirmed, perhaps indirectly, that the material is verified. Do the external links actually support what is said? What kind of links are they? Some web links are more plausable than others. If they do support it are they saying the same thing as standard texts. If the facts are not verifiable, then someone is likely to say so. This is a wiki afterall. --Bduke 05:26, 20 March 2006 (UTC)

Ahem. Now lets take another example. Let's look at the foundational concept of most of natural science; i.e. power. The scientific article, Power_(physics). No sources.... Sholto Maud 05:29, 20 March 2006 (UTC)

If I understand your argument, wouldn't you be better off with a page that was extensively referenced (ie lots of inline references/footnote type things), because the concern is whether those references are reputable sources, or have I completely misunderstoo this argument?--Limegreen 05:38, 20 March 2006 (UTC)

In short yes, I think so. I use these articles because of their importance to science, and because they flout the general Wikipedia principle of Verfiability. Verfiability is, apparently, an important part of the imagined SPR process. An important part of verfiability is the provision of reliable sources. I would like to see a formal statement about how one establishes the reliability of a source - I think Bduke is giving us some inroads. It would seem that extensive referencing would make one "better off". But in the important article on power, the science editors seem to have the inverse point of view; i.e. no referencing makes one better off, and thus the knowledge-content of the article more reliable with no references. Remember it is verifiability, not truth that we are concerned with here. Sholto Maud 06:17, 20 March 2006 (UTC)

To Sholto Maud Several points:-

1. The crunch comes if someone adds a "disputed" tag to the article. Then everything has to be referenced.
2. I always take external links to be both more information and a source of at least some of the material in the article. So is the material in the links? I have not looked but I guess the answer is "No".
3. A review therefore should demand that more references be added and that they be precise for different sections, unless you can say cite three textbooks that all contain this material. It is not the sort of article that needs to cite the primary literature.
4. At the same time, the reviewer looks to see whether this material is the standard stuff that appears in text books.
5. Is this so difficult? --Bduke 05:43, 20 March 2006 (UTC)

But wait a second,

"One of the keys to writing good encyclopedia articles is to understand that they should refer only to facts, assertions, theories, ideas, claims, opinions, and arguments that have already been published by a reputable publisher. The goal of Wikipedia is to become a complete and reliable encyclopedia, so editors should cite reliable sources so that their edits may be verified by readers and other editors"

"Verified" in this context means that reader2 and/or other editor2, can repeat the process of going to a source to check the accuracy/reliability of editor1's works. Thus repeatability is inherent in verify. In the power article I am unable to repeat the process of going to a source. Thus the article is unverifiable and fails to comply to Wikipedia rules. This, by the way, is the same meaning of "unverifiable" as used in logical positivism. Sholto Maud 06:06, 20 March 2006 (UTC)

The repeatablility of going to a source is a bit different from repeatability of an experiment. However, what you says does mean it is unverifiable (unless the stuff is in the sources I mentioned - the book you say you added and the external links). However, it does not mean it is unveferifiable. Just go to a standard text and see whether the article is supported by the text. Add that text as a reference and the article is verified. Simple. Lots of articles do not comply with Wikipedia rules, but most of them are easily fixed. That is my last post for 15 hours or so. --Bduke 06:15, 20 March 2006 (UTC)

Thanks User:Bduke! Have a well-earned break. Sholto Maud 06:49, 20 March 2006 (UTC)

Fatal errors

The concept of verification appears to be a fundamental feature of the imagined SPR process. Representing the popular opinion of Wikipedia users, User:Bduke seems to agree that the Wikipedia concept of verification is the same as that used by logical positivists. A difficulty arises here because the logical positivist definition of verifiability defers to the correspondence theory of truth: "The correspondence theory of truth states that something is rendered true by the existence of a fact with corresponding elements and a similar structure". And this would appear to be what we do when conducting a scientific peer review. That is, we check whether a Wikipedia article is true to the facts of the reliable sources. But then this makes the Wikipedia philosophy of verification non-standard, because it is convcerned with "verifiability, not truth". Hence the Wikipeda theory of verification is Original Research. This would appear to qualify as a --FATAL ERROR--

Furthermore the Wikipedia theory of verifiability does not cite any reliable sources in defining the concept, thereby failing it's own conditions --FATAL ERROR-- Sholto Maud 09:50, 20 March 2006 (UTC)

SPR Method/algorithm

I feel motivated to make a statement of the SPR process as given by User:Bduke above.

• 1. Check: Is article content supported by book references?

1.1 If so then print: "book reference material is verified".

1.2 If not then print: "book reference material is not verified. SPRbot demands correct book references be provided or article will be excluded in 7 days."

• 2. Check: Is article content supported by external link references?

2.1 If so then print: "web reference material is verified".

2.2 If not then print: "web reference material is not verified. SPRbot demands correct web references be provided or article will be excluded in 7 days."

• 3. Check link type: Is web link "edu", "gov" (any other categories?).

3.1 If so then print: "external link type is X ".

3.2 Print: "Plausability of link type X is Y":

---!!!SEMANTIC ERROR: "Plausability" variable Y is undefined!!!---

• 4.0 Check: If 2.1, then check external link references == standard texts.

---!!!SEMANTIC ERROR: "standard texts" variable is undefined!!!---

4.1. If so then print: "external link references supported by standard texts".

4.2. If not then print: "external link references are not supported by standard texts.

SPRbot demands the article is supported by standard texts or will be excluded in 7 days."

• 5.0 Check: article type.

---!!!SEMANTIC ERROR: "article type" variable is undefined!!!---

• 6.0 Check: is article type required to cite primairy literature?

---!!!SEMANTIC ERROR: "article type requirement" variable is undefined!!!---

6.1. If so then print: "article required to cite primairy literature".

6.2. If not then print: "article not required to cite primairy literature".

• 7.0 Check: is article content == text book content?

---!!!SEMANTIC ERROR: "article content" variable is undefined!!!--- ---!!!SEMANTIC ERROR: "text book content" variable is undefined!!!---

7.1. If so then print: "article material is the standard stuff that appears in text books".

7.2. If not then print: "article content is non-standard. SPRbot requires article material be standard stuff or be excluded in 7 days".

Not so difficult. However this hasn't formalised the criteria by which to evaluate the reliability of a source. Sholto Maud 06:49, 20 March 2006 (UTC)

Slashdot had a comment on the concept of an automated Article reviewer reviewer for Wikipedia today. --Ancheta Wis 00:51, 21 March 2006 (UTC)

Concepts that need rigorous, operational definition

• article type: What is an "article type"? How does one determine article "typeness"?
• article type requirements: How does one determine the citation "requirements" for any "article type"? On what basis does one article type have greater or lesser requirements than any other "article type?"
• citation plausability: What is "citation plausability"? How is "citation plausability" determined?
• standard text: How is a text's rating of "standardness" determined? Is there a "standardness" index?
• source reliability: How is "source reliability" evaluated? Is there a "source reliability" index? Sholto Maud 23:59, 21 March 2006 (UTC)

(i.e. I don't think "because it is" is a good way of evaluating "source reliability". Sholto Maud 01:09, 22 March 2006 (UTC))

Argument for, and speculations about a SPRbot

I envisiage that an SPRbot may have the following benefits:

• save time in construction of valid articles
• provide a standardised form for prompting editors on what an article or contribution needs in order to satisfy all Wikipedia rules.
• impartiality in the SPR process (which is what I assume a "scientific" peer review process strives for)
• control article quality

Take the Power article for example. Although the content may be correct it does (did) not have any citations to reliable sources. An SPRbot would recognise that the article has no such citations, and then perhaps post a requirement notice at the top of the article "The verification rule of wikipedia requires that a scienc article has at least X (2,3,4??) reliable sources cited please enter them in the space provided below". Then the bot could also post a XHTML form at the bottom of the article with windows for entering title date, or even simply the ISBN number. Thus an editor could simply scan the ISBN barcode in to save time with their citations.

Another SPRsub-bot could then search and enter full citation details in correct formatting from the ISBN number making referening more standardised, but more usefully, if a Wikidatabase of reliable sources and standard texts were constructed, the SPRbot could check the article against the database to make sure that the sources given qualified in terms of reliability, and or standardness in the eyes of Wikipedia reviewers. If the text is not in the database the bot would return something like "the reliability of this source has not yet been determined by SPR."

This would mean that human SPReviewers could concentrate on questions about evaluating source reliability, and content rather than the laborious task of citation verficiation. This database could be a simple process of 1 vote per USER per source about the standardness of the text. The more votes a text gets, the more reliable??

Lastly. If the SPRbot were able to search for predicates within each article, it could see if there were any footnote citation next to the predicate. If not then the SPRbot would notify the editor's homepage... perhaps something like, "you have made an claim that has no verfiable citation. Please provide a reference in the footone form provided at the bottom of the page."

Such a SPRbot should increase the rigor of each Wikipedia article with respect to the verifiability clause. Sholto Maud 01:01, 22 March 2006 (UTC)