Wikipedia talk:Source grading

Review

A sort of review/note/demo/log of source grading is at Wikipedia talk:Source grading/reviews. In no way is this part of the source grading toolbox; there is no requirement to list here. This is simply offered as a quick peek/example of source grading in progress. See the actual sites so graded to see what was done. John Reid ° 06:59, 21 November 2006 (UTC)

Complaints

Too Hard

I had seen this proposal somewhere before (but can't remember where), anyway currently I come here from wikipedia talk:reliable sources

Some points to consider:

• Proposal is adding yet another layer of complexity to something that is already difficult to understand to the average Wikipedia contributor. Don't know whether all the trouble is worth it: it has massive overhead, would need almost a separate help desk division to answer the expected questions, etc - don't know whether eventually there would be a net result of article improvement, compared to using the same amount of energy for hitting the "random article" link, and attempt to improve the citations in the article you get in front of you.
• Proposal seems like an unneeded exception to Wikipedia:Avoid self-references to me. I mean, all these letters, colour-code and categorisation is supposed to be applied in article namespace, while this codification has no meaning outside Wikipedia. That is stuff we generally try to avoid. OK if people from the fact and reference check project would use such codes on the associated talk page, but please keep it out of main namespace. We already have more than enough problem-tags-with-project-categorisation. The fact that we have so many (in fact personally I would say too many) should not be used as an excuse to add yet another of such systems.

So, unless this project is reformulated as a talk page endeavour (so, outside main namespace), I'd reject it in advance. If this is made into something on talk pages, it should be self-evident (yeah, that's another problem with the current proposal), and rather reduce overhead and dispute potential ("this source is A grade, no it isn't it's worse than C" - oops, yet another problem) than increase it. --Francis Schonken 14:15, 15 November 2006 (UTC)

• I tentatively agree with Francis. Please elaborate on the need for the added complexity proposed here. (Radiant) 16:36, 16 November 2006 (UTC)

This is not a policy proposal or even a process. It's just a collection of tools. Use them if you like; if you don't, don't. If you see anything on the page that seems to take a position of authority, remove it.

No, it's not another layer of complexity. It's simplicity itself: A, B, C. This is a concept that the dullest, least-educated person in the room can understand. You, of course, are much better off than this so you see complexity where there is none. A-B-C grades are well understood by almost everyone and are used in more fields than anyone can count.

There are no self-references here; I can't see where that idea came from. Each template simply sticks the letter in front of the citation and categorizes. No clever links, no parameters, no parser functions.

No overhead is involved; you simply stick the grade on the source. You don't tag stuff to do later or ask why. Definitions of A, B, and C level sources are purposely not given, so there's nothing to fight over. There's already plenty of "dispute potential" over what is and is not a reliable source; this collection of tools eliminates a good deal of this because it is possible to say a citation is fair, rather than only OK or BAD. Try not to see everything in terms of black and white; you'll enjoy a more colorful world.

These are not "problem" tags, like {{weasel}}. Those indicate problems to be fixed. No citation needs to be fixed because it has a C grade. Perhaps the article can be improved by adding more, better citations; perhaps not. If the citation is unacceptable, it doesn't get a C grade; it gets removed.

Talk page is wrong for this. Our readers deserve to see what quality of citations we offer. Besides, copying citations back and forth would be massive overhead. Do not do this.

I'm sure there will be opposition to this idea from people who fear to see just how it shakes out; we may be ashamed to see how few A citations we really have. Well, sorry. Honesty is the best policy. This is stupid simple. Just do it. John Reid ° 22:59, 16 November 2006 (UTC)

• Obviously any kind of tagging is more complex than not tagging in the first place. If your aim is to have a lot sources tagged like this, then you need to demonstrate to the community the need for this. Otherwise you're just givng scope for revert warring over "this is an A source!" "no, it's B!" especially because no clear definitions exist on that. It would also help if you didn't make implications that people who disagree with you might be undeducated, or assuming they have the ulterior motive of being ashamed of Wikipedia's current state. (Radiant) 10:47, 17 November 2006 (UTC)

I don't "make implications"; I suggest it outright. I didn't say you were too stupid to understand this; you may be too smart. If you are afraid to see what kind of grades will be assigned, you fear shame. If you're not afraid, let it be.

There are already plenty of squabbles over sources, references, and citations. I say this will reduce the level of rancor. We'll need to try it to find out. Nobody needs to "demonstrate" that this will work -- nobody can. Nor can anybody prove it won't. We have to try it to find out. And if that doesn't work, well, we'll try something else. John Reid ° 02:07, 18 November 2006 (UTC)

Who would do this grading?

I agree that in principle sources could be graded, but I don't see the point in making it explicit, at least not yet. My reasoning is simple: (1) Those who know enough to assess a source should be able to provide a better source themselves; (2) Those who don't know enough to assess a source will just get it wrong.

What might be useful is a way to allow internal workgroups (ie. WikiProjects) mark references and sources this way, so they know which references they have checked, and which ones still need checking. Really though, this sort of thing is best done in a fact-checking, clean-up stage when an article is finished, not during the creation and expansion of an article. Theoretically, every featured article should get its sources intensively checked, with each reference being considered and weighed in this way. Carcharoth 02:45, 18 November 2006 (UTC)

You do -- we all do. Articles really don't go through a straightforward process from "new" to "working it up" to "finished". More importantly, our key principle is that as an article wobbles about, pursuing some upward trend, it is at every stage open to the entire world's inspection and edit. Everybody gets to participate. If citations are important to an article, then the status of those citations is important to our readers. It's just a matter of honesty and transparency. Objections leave me baffled. John Reid ° 10:59, 20 November 2006 (UTC)

Why don't you take an article as an example and show us what you mean? The we can all, um, discuss how we disagree agree with your grading of the sources. Carcharoth 11:32, 20 November 2006 (UTC)

Well, it doesn't really matter -- to the tool -- if you disagree or not. If you disagree, raise or lower the grade -- or rm the citation if it sucks. If I revert, I'm a dick.

I dunno, I think you just want me to go and actually grade a citation or two? I think I lack the specialist knowledge to make render an accurate opinion on most citations -- you have to understand, in some ways I'm as useless as tits on a bull around here. Hmm, let me dig through my mainspace contribs, see if I can go to actual work for a bit. Okay? John Reid ° 15:06, 20 November 2006 (UTC)

Okay, check out Wikipedia talk:Source grading/reviews. I can't stress enough that that page is not any kind or part of the regular toolbox; it's just there for you to see how the thing goes. I don't expect anybody to log their grades there, as I have -- although of course you're welcome to do so. I hit a couple of articles; perhaps I'll hit a couple more. I can't stress enough that I lack the specialist knowledge for this task, myself. I do have a pretty good university library in town and a card but parking is insane there; you'd have to pay me to do research there on anything except my personal obsessions. Sorry. John Reid ° 16:10, 20 November 2006 (UTC)

I think what you've written over there is demonstration that references and their sources should never be taken at face value. After only a little bit of digging, you were able to find problems. This is why I agree that a more rigorous system of checking references is needed. I'm just not sure that this is the solution. I would prefer a talk subpage dedicated to more extensive annotation and/or discussion of the references. ie. Use this book for xyz reasons, but don't use this webpage for abc reasons. This sometimes gets discussed on talk pages, but lost in the noise. Carcharoth 14:55, 21 November 2006 (UTC)

Most people will never make such an exhaustive survey and it's unrealistic to hold the project to such a standard. As it is, I've only skimmed the surface. I think it says a great deal for good common sense that most articles in Wikipedia are basically factual, even though poorly sourced. EB spends a vast amount of professional time nailing down sources and doesn't come out much better than we do.

I think I've answered the question. I don't expect anybody to grade a source as seriously as I've done; I don't think it's reasonable for anybody to do that much work, let alone to annotate that work in detail. I just expect ordinary editors to make a judgement -- A, B, C -- about one source after another and let the rest of us -- and our readers -- know what that is. John Reid ° 03:32, 22 November 2006 (UTC)

My problems with this.

One is that the grading is subjective, thus instantly violating WP:NPOV. We're supposed to be providing objective information on a subject, not telling which sources are best. So unless we come up with an objective method of grading sources (impossible), it won't work.

Second, it is simply not necessary. If a source is so unreliable that we don't believe it, it's not included, and if it is reliable enough, why does it matter HOW reliable it is? It might if we were providing subjective analysis, but just providing the facts, it is entirely irrelevant.

-Amarkov ^blah_edits 17:04, 18 November 2006 (UTC)

I agree with this. The minute we assess sources, we're making judgements that should be left to the reader. Any doubts about a source can usually be rectified by trying to verify it or following WP:RS closely. Daniel Case 05:52, 19 November 2006 (UTC)

Well, this is about the silliest objection I've read today. Our articles are supposed to be neutral; but every time we add or subtract from an article, we express a bias. The whole is more than the sum of its parts. For that matter, neutrality is a goal that can never be achieved. We are wise to pursue it but stupid and arrogant to imagine we can grasp it.

If I remove a citation entirely, I'm making a judgement on its merits. You may disagree with me; that's your bias. It is utterly impossible to say objectively whether a given citation is or is not "correct". If you think otherwise, you simply haven't been to grad school or read heavily in metaphysics. I'm sorry to put it that way but that's how it is. No source is definitive. Sources exists on a continuum from good to bad and every reviewer has a subjective opinion on each one.

Objections like these come from a background in Western culture, steeped in positivism and materialism. I share that background and those values, too. But I also know what when you get right down to the roots of the gigantic structures of science, mathematics, and logic -- all scholarly thought -- there is nothing at all holding up that groaning weight except a large number of personal, subjective opinions. There are not really any facts at all, when you get right down to it. Sorry. John Reid ° 10:54, 20 November 2006 (UTC)

Um... what? It is objective whether or not a concrete fact is true. If my pants are grey, they are grey for EVERYONE, not just certain people. If a source reports that my pants are actually green, then it is wrong, and anyone who protests against its removal is stupid. Since we should be reporting nothing but the objective facts, it doesn't matter. -Amarkov ^blah_edits 15:42, 20 November 2006 (UTC)

How very Western. I sympathize, I really do. I once saw the world that clearly myself. But I say your pants are not grey and if you give me a day or two, I can show you refs to demonstrate it. What's more, I'll bet you a dollar to a doughnut that I can bring you refs you have not the time, inclination, nor skill to refute.

Have you -- you, personally -- ever tried to verify a citation to academic standards? I'm not trying to run you down or make you feel bad but it would be helpful for me to know who you are. Have you ever spent a weekend -- or a year -- in a library, shuttling from periodical stacks to bookshelves to microfilm? I don't ask if you've done this to finish a term paper; I ask if you've ever done this to prop up a thesis you know will be torn apart by the department committee?

You can, of course, try to play it safe. Use only the most reputable sources -- what here we would call (A) sources. You might cut corners a wee bit in the claims you make and the way you connect them to your citations but you stick with the real good peer-reviewed journals you know your dean likes.

If you're actually trying to do work, though -- to bolster claims that not everyone has already made -- then you dig and dig. You come up with some obscure paper published by the University of Outer Somewhere; if you're lucky, in your own language. Who is this guy? Is he any good? See what else he's done. Did anybody else ever read his paper? Comment on it? He's got plenty of refs himself. Are you ready to check them out? All of them? Do you understand exactly what claims he is making based on his own research and which he's drawn from his own refs? Are you ready to insert a claim into your own paper based on this guy's stuff?

If you have ever been in that cramped little acoustic tile room, staring at text on thermal paper that costs a quarter a sheet yet disappears before your eyes, trying to decide is this guy bullshit or is this guy for real then you will know, for certain, that a True Fact is a rare gem. For the rest, we rely on probabilities. That's all. John Reid ° 16:27, 20 November 2006 (UTC)

Stay COOL; I haven't set fire to your dog. It took me an hour to evaluate those sources, 30 seconds for you to rm them. I'm not going to get ahead of you. If you disagree with a grade I've made, you're welcome to change, even remove it. If somebody else disagrees with you, you can discuss it with that person. I won't be back to enforce anything. No policy is proposed; it's just a toolbox.

As for your points, I suggest you read the prior objections, which I've addressed in considerable detail.

You do not even want to contemplate the reign of terror that will erupt when some academic with a jones on who knows what a reliable source constitutes starts stripping out all the B's and C's from this project -- without the courtesy of grading them. Trust me; I can't do this without your page-by-page consent -- literally. John Reid ° 14:25, 22 November 2006 (UTC)

I agree that we cannot make a statement of the quality of a source in an encyclopedic article unless this statement of quality is backed up by a reliable source. If a citation is bad, remove it. If different sources have different known levels of reputation, say so in the article, but cite this claim. HighInBC ^{(Need help? Ask me)} 16:51, 24 November 2006 (UTC)

a thought - with example

Is that of WP:NPOV being that by assessing the reference we are giving weight or unbalanced authority to one particular source. Under the current unassessed system when multiple sources support the article contents they just get included where as editors will become lazy rating a source as "A" and not including other supporting sources. Then the problem arises in articles like Indigenous Australians where a source like the Walter Roth study is exceptional and would be rated "A" but at another part of the same article same source the information would have be rated "A" as it a direct quote from the written work of the individual. Yet immediately following is another quote refuting this specific claim of Roth again as it a direct quote it would also be rated "A". That anloe is a big problem as most people will see the two sides and make their own assessment but... Then from this theres the political side of Roths work outside Wikipedia where one political group that expresses intolerance would use the Wikipedia article citing that we view Roth statement as authorative and strongly supporting the claim Roth made. The end result of all this would do more damage to Wikipedia than a knowing cite being used only vaguely supports a statement. Gnangarra 11:43, 22 November 2006 (UTC)

You have a point; a source can be cited more than once in an article. It may well support one claim but not another. This is irrelevant to the value of the source itself. Thus far, I have only treated sources that are cited once in an article. I'm tempted to say that the higher (or lower) grade should apply to the entire source; why complicate things? Or split the citation -- duplicate it, grade each copy individually, depending on its merits in context? Of course, if a good source poorly supports a given claim and no other supports it at all, best to remove that claim altogether, retaining other citations to the same good source.

Why don't you think it over and tell me how to improve the source grading system so as to address your concern? I'll be happy to help. John Reid ° 14:32, 22 November 2006 (UTC)

Inline citations

Okay, hold the phone. I envisioned this to apply to citations in References sections and didn't think through the issue of inline cites. Of course, it's distracting madness to stick visible grades right inline with the cites. I never considered that. Until recently, there was always a token in the Refs section for every inline cite -- or such could be made; now we've got cite.php to wrestle. Let me stick my thinking cap on. John Reid ° 06:06, 21 November 2006 (UTC)

No problem after all. I exercised {{sa}} by putting it, inline, before a citation within cite.php-style ref tags. It does not display inline and displays perfectly in the Refs/Notes section. John Reid ° 06:59, 21 November 2006 (UTC)

Still don't like them

The source grades look weird. It's not clear what they mean from looking at the page, and they reduce the readability of the source. If they must be there (and I suspect the idea will be squashed soon), they should go after the reference, not before. Carcharoth 16:40, 24 November 2006 (UTC)

One place where it *would* work!

Medical articles. Position papers and guidelines grade sources all the time (ranging from randomized control trials to anecdotal reports). But you need objective definitions of the quality of the source -- Samir धर्म 05:06, 29 November 2006 (UTC)