User:Parc wiki researcher/survey1 results

Wikipedia Conflict Survey

[prelimary results: please do not distribute]

We conducted a survey of Wikipedia users expert in conflict identification and resolution in order to better characterize conflict processes and ascertain “ground truth” for a representative group of articles. The results of this survey are being used in a larger academic article about conflict in Wikipedia which will be submitted to a peer-reviewed journal. Typically this means it will become freely available, and at that point a link to it will be provided from our web site (http://www.parc.com/istl/projects/uir/). Meanwhile we are posting preliminary results from the survey in order to provide information for and get feedback from the Wikipedia community on our research.

Method

Participants. 54 Wikipedia users were targeted. All users were members of the Wikipedia “Mediation Cabal” (http://en.wikipedia.org/wiki/Wikipedia:Mediation_Cabal), an informal group of experienced users whose purpose is to provide mediation for situations that have escalated into conflict. Users were solicited by placing descriptive text and a link to the survey on their discussion pages. The survey was completely voluntary with no monetary reward offered, though participants could opt-in to receive a “research star” image on their user pages. 27 users completed the survey.

Materials. 10 articles were selected for inclusion in the survey. These articles were chosen to be representative of the following article types: Neutral Point Of View disputes (NPOV), Accuracy disputes, Controversial articles, Featured articles, and articles with none of the above tags. The selection process for articles involved ordering all articles within a type by the number of article revisions (according to our February 2006 database), then for each type selecting one article within the top 8-15 and one closest to the median which were still of that type in the live Wikipedia system (as of 7/21/06).

Design. Users were asked to rate the overall level of conflict in the article, the current remaining level of conflict, and the quality of the article. They were also asked to judge the extant of various types of conflict occurring, including NPOV, accuracy, user, importance of information, and point of procedure disputes, as well as the level of edit warring. Fields for free-form comments were available both for each page and for general feedback at the end of the survey. A final debriefing question asked users to describe the process by which they made their decisions about ratings and categories.

Results: Ratings

The below graphs show the aggregate ratings of participants for the conflict and quality questions. The scale is from 1-5, with 5 being highest conflict or quality, respectively.

Results: Free-form answers

A number of questions allowed for free-form user input. Below are results analyzing these inputs.

Evaluating Conflict. 27 users responded to the question “Please describe the process by which you made your decisions about the ratings and categories in this survey.” Of these, 21 provided detailed process information. Responses were remarkably similar, with most users indicating they reached their decisions by examining the edit history of the article page (100%), the discussion page (86%), and the article page itself (57%). A few users also mentioned examining the protection logs (2), featured and good article discussions (1), the edit history of the talk page (1), and the user pages of article editors (1). 8 users also indicated the order in which they evaluated each of these resources. Their responses are included in Table 1. A clear order effect is evident, with article pages evaluated before edit histories, which are evaluated before discussion pages. Only one user was inconsistent with this pattern.

Table 1. Order of evaluation of listed resources. Each row correspond to a single user. Only users who included order information are included.

Evaluation order

• Article Edit History Discussion page
• Article Edit History Discussion page
• Article Edit History Discussion page
• Article Edit History
• Edit History Discussion page
• Article Edit History Discussion page
• Edit History Discussion page
• Protection logs Discussion page Edit history Article

Users also elaborated on the metrics they used in evaluating conflict and quality, which are shown in Table 2.

Table 2. Metrics of evaluation listed by users.

• Conflict
• Discussion
  • POV
  • user dispute
  • policy issues
  • # comments
  • heat level
  • issues raised
  • # archives
  • presence of FAQs for previously discussed issues
  • edit war tag
  • protection tag
  • npov tags
  • # headings
  • long sections
  • length
  • newness of disputing users
  • lack of citations
• Edit history
  • 3rr violations + source
  • # reverts
  • summaries (+ aggressiveness)
  • edit wars
  • newness of disputing users
  • antivandalbots
  • historical vs. new reverts
• Article
  • Read article
  • Wording
  • Current debates outside wikipedia
• Featured article discussion pages
• Good article discussion pages
• Quality
  • Comprehensiveness
  • Style
  • Referencing
  • Length
  • 1st Paragraph readability
  • Featured article status

Uncaptured Conflict: Types. For each article, in addition to evaluating the experimenter-specified types of conflicts users were also given the opportunity to list any conflict types that were not listed. The conflict types listed more than once were vandalism, unreferenced/unverified material, accuracy of sources, and factual vs. popular views. The last type of conflict occurred only for the article “ninja”, which a user characterized as “between people who actually know something about ninjas, and people who think ninjas are cool and want to write about them.”

Uncaptured Conflict: Pages. A number of users suggested other pages which are frequent sources of conflicts. These pages are shown in Table 3.

Table 3. Frequent sources of conflict not represented in the survey.

• Nationalist battles on central european and balkan pages
• Middle eastern subjects (e.g., "New anti-semitism" and "Israeli apartheid")
• American political bios
• Policy + guideline pages
• Nominations for article deletion
• Administrator noticeboard pages
• Requests for comment

Interesting Quotes

There were a number of interesting quotes captured in the free-form feedback portion of the survey. A few of these are listed below, minus their identifying information. Feedback from the community on these topics (or others) would be welcome.

“A lot of disputes arise because, in spite of WP:OWN policy, people generally take a *lot* of offense to others reverting or substantially editing their work. Wikipedia is kind of this great community project, and when someone deletes your input, well, I think it's human nature to get your feelings hurt.”

“It seems from your (very small) sample that the articles with more conflict are the better developed. Of course I could give you countless examples where this is not the case and I think it might be better to look at the main contributers to the articles in question. For example in the Creationism/ID/Evolution articles, they was a lot of conflict, but the editors coped with it well, largely due to a good understanding of the subject.”

“There needs to be a ratio of conflict per new editors/users. The majority of users in dispute with one another, have not read and or understand Wikipedia policies and guidelines, have strong feelings about certain topics, and cannot remain neutral in writing. Whereas users who have been around longer have a better understanding of this.”

“I found it interesting that you only chose articles for your study. Stick around for a while and you will soon discover that an equal (if not greater) amount of conflict occurs on policy and guideline pages, nominations for article deletion, administrator noticeboard pages, requests for comment, and so on. The degree of success that one meets in dealing with conflicts (especially conflicts with experiences editors) often depends on the efficiency with which one can quote policy and precedent.”

“Although this highlights on content and article quality, it lacks questions on how the community tries to address these problems and lead to improvements.”

“I think this survey is a good idea. I was particularly interested to find that only one of the articles Youth Culture, was of notably low quality. The Wikipedia dispute resolution process seems to work better than I thought.”

“Thank you for the oppotunity, and the experience! It was a great way not only to help out, but also look at wikipedia from a new light, and see new articles!”

Thank You

Thanks again to all participants for the hard work and generous efforts. For all those reading this, we hope to keep contributing back to the community in an effort to better understand and deal with conflict. We are very interested in your feedback and comments on conflict in Wikipedia as well; further discussion on the discussion page is encouraged. Any comments on what type of research or tools the community could use to help deal with conflict would also be useful. Thank you all!