Wikipedia:India Education Program/Analysis/Independent Report from Tory Read

This report is also available as a PDF.

This report, prepared in January 2012 by Tory Read of Tory Read Studio for the Wikimedia Foundation, presents findings and recommendations from a qualitative review of the India Education Program Pune Pilot Project, which is part of the Wikimedia Foundation's Wikipedia Education Program.

The report includes:

• An overview of the Wikipedia Education Program and how it fits in to the Foundation's global development work,
• A description of what happened in the Pune Pilot Project, which took place in India and on English Wikipedia, and
• Recommendations for the India Education Program going forward.

The information presented herein is derived from a review of relevant talk pages and email lists, interviews with 43 people who played key roles in the pilot project, and statistical data compiled and analyzed by the Foundation. People interviewed include: Wikipedia administrators and editors; university directors, professors and students in Pune, India; Campus Ambassadors and Online Ambassadors, and Foundation staff and consultants in San Francisco and Delhi.

Overview

The goal of the Wikipedia Education Program is to engage students and professors across disciplines, universities and countries in using Wikipedia as a teaching tool. This goal dovetails with that of the Foundation's Global Development unit, which is to grow the Wikipedia editor community in the Global South, including India. This objective is discussed in the Wikipedia strategic plan, which was created in 2009 and 2010 through a wiki-based collaboration between the Foundation and more than 1,000 Wikimedia project volunteers.

The goal of the India Education Program is to create a "sustainable and long-term channel to recruit new editors" by integrating Wikipedia assignments into university courses in India. The Foundation's India Programs also include:

• Other types of outreach to attract new editors to English and Indic language Wikipedias,
• Developing mobile platforms, and
• Forging partnerships to support the growth of Wikipedia in India.

As Foundation Consultant for India Programs, Hisham Mundol leads the Foundation's work in India. His assignment includes overseeing planning and implementation of the India Education Program in consultation with the Foundation's San Francisco-based Wikipedia Education Program team, which is led by Frank Schulenburg. Schulenburg reports to Barry Newstead, who leads the Global Development team and oversees Mundol's consulting contract.

The Pune Pilot Project was the first effort of the India Education Program. The Foundation based its design on that of the U.S. Public Policy Initiative (PPI), which was the Foundation's first university-based Wikipedia program. According to a November 2011 report from the Wikipedia Education Program, PPI achieved positive results. In its first 18 months, 40 professors at 24 universities taught the Wikipedia assignment in 47 classes, and more than 800 students contributed the equivalent of 5,800 printed pages of content to Wikipedia. According to the report, this content was high quality‚ public policy articles that students worked on improved an average of 64 percent.

Campus Ambassadors (CAs) played a lynchpin role in both the PPI and the India Education Program Pune Pilot Project. In both efforts, CAs were volunteers who served as the "face" of Wikipedia in participating classrooms. They helped professors develop and introduce the Wikipedia assignment and taught students the basics of Wikipedia (including creating a username and making edits). The majority of CAs did not have deep experience as Wikipedians, and in the U.S. program, this did not pose a problem.

The Pune Pilot Project: A Recap

Between February and November 2011, Foundation staff and consultants planned and implemented the Pune Pilot Project. It took place at three universities in Pune, India, and included 1,014 students spanning 24 courses.

The Foundation Plans the Pilot

In initial conversations in February and March, Mundol, Newstead and Schulenburg discussed the pilot design, drawing on the PPI model. PPI had commissioned a review and evaluation of the program, but the report was not yet completed, so the information shared was largely anecdotal.

In the same months, Mundol also traveled around India to introduce himself to Wikipedians and to think about the best location for the pilot project. After conversations with Wikipedians, university directors and professors, he selected Pune as the pilot location, because Pune is home to more than 100 colleges and universities and has more than 200,000 students from across the country. A summary of what he learned and the resulting plan for all India programs appears here. Subsequent visits to Pune in March and April included conversations with the Wikipedia community there. A summary of his April visit to Pune appears here.

Mundol assumed that the global Wikipedia community would review and comment on his plans, which were both posted on Wikimedia Meta-Wiki. The former page, containing the overall India strategic plan, did attract some discussion in mid-March and early April, mostly from Wikipedians in India. See the conversation here.

Mundol Selects English Wikipedia

Mundol selected English Wikipedia for the project because English is the dominant language of instruction at many universities and colleges in India, including in Pune. However, Foundation staff and consultants did not expressly alert editors or administrators on the English Wikipedia to this decision.

"The moment the Foundation selected English Wikipedia, it should have asked the community for input," said one global editor. "Given that Foundation staff and consultants are legally obligated not to do content work, any new work on the encyclopedia falls to us, the volunteers."

"We could have given a realistic estimate of what is possible with existing resources," said another editor.

According to LiAnna Davis, Wikipedia Education Program communications manager, writing by email, "Foundation staff believed the Pune Pilot to be a natural extension of the well-received Public Policy Initiative pilot in the United States. ... Communications on PPI pages talked frequently of its extension to India as well, although no effort was made to publicize the program beyond existing education program information channels."

She noted that Foundation staff mentioned the expansion of the program to India in the Wikipedia Education Program newsletter here and her blog here. The Wikipedia Signpost mentioned it here, as well. "For us it seemed like we had publicized in channels people interested in Wikipedia's use in education were paying attention to," she wrote.

Mundol Selects Campus Ambassadors and Host Schools

In April and May, Mundol advertised the Campus Ambassador (CA) position, and 700 candidates submitted applications. As noted above, the CA role involved being the face of Wikipedia in the classroom, liaising with professors to ensure smooth implementation in their classrooms and teaching students the basics of editing.

After a multi-stage review process that included interviews with short-listed candidates, Mundol selected 20 CAs, based on their understanding of the Wikipedia mission and on their ability to learn and teach. Many had never before contributed to Wikipedia, but the Foundation did not flag this as a cause for concern, because CAs in the U.S. program had done fine without having had extensive Wikipedia experience.

In May and June, Mundol worked with Schulenburg and his team to finalize plans for and launch the Pune Pilot. Schulenburg's team included Wikipedia Education Program Manager Annie Lin and Wikimedia Fellow P.J. Tabit, who had served as a CA in the U.S. program. Together, they planned the training for the first cohort of India CAs.

Meanwhile, Mundol conducted preliminary meetings with directors and professors at 14 Pune universities and colleges to find the best host schools for the pilot project. He also selected Srikeit Tadepalli, an Indian university student who was an experienced Wikipedian, to become a Wikimedia Fellow and work with Tabit to oversee CAs as the pilot became operational.

Although planning happened quickly, over a matter of months, Mundol was confident that all of the pieces were coming together to support a successful pilot project. Newstead supported his approach and timeline, because he felt Mundol had the skills to lead the effort and because the Foundation wanted the global and India Wikipedia communities to see the Foundation's investment yielding action. "Mundol hit the ground running," said Newstead. "His planning phase was measured in weeks, not months. It probably made a shockingly fast impression in India, but I wanted people to see that when we pay staff, things happen."

"I also had confidence in Schulenburg and his team," said Newstead.

Schulenburg, Lin and Tabit went to Pune in June to lead the CA training with Mundol. They also met with the school directors at the colleges and universities that, in Mundol's estimation, had the greatest potential to succeed as host sites. His criteria included: A commitment to innovative teaching approaches, strong academic heritage, solid awareness of the Wikipedia mission and enthusiastic support from the director and a core group of professors. Mundol also sought a diversity of subject areas. The Foundation team homed in on six universities that best matched these criteria.

The Foundation Trains the CAs

The CA training itself lasted two days and included: An introduction to the basics of Wikipedia editing, an overview of the CA role, panel presentations by Wikipedians telling personal stories about their experience editing the encyclopedia, team-building games, teach-backs of key Wikipedia content by CAs and role-plays of how to interact effectively with college professors and directors.

"The training was designed for the U.S. program, where Online Ambassadors (described later in this report) handle most on-wiki communication," said Davis in an email.

She added that the India team asked all CAs to do homework, which included creating a user account, making five article namespace edits and leaving a message on a talk page, but many of the India CAs didn't do the assignment.

Three members of the Wikipedia community in Pune participated in the CA training as presenters, but they reported that overall they felt excluded from the pilot planning process. "They're not making good use of us," said one Wikipedian. "We're capable of contributing so much more."

At the time of the training, they commented that it did not go deep enough in the "how to" of Wikipedia and did not include enough hands-on Wikipedia editing. They said that if they had participated in planning, they would have urged greater emphasis on teaching these skills to the CAs. These concerns were posted by one Pune community member on a Wikimedia Meta-Wiki page in late June. Mundol responded quickly. See their exchange here.

At the end of the CA training in mid-June, CAs asked for specific next steps. Mundol had not yet finalized schools, so there was a waiting period between the time of the CA training and when work got going in classrooms in mid-August.

After the CA training, Schulenburg and Lin left India, and Tabit stayed on in Pune. Bob Cummings, a U.S. professor who had participated in PPI, joined him. Mundol, Tabit and Cummings led workshops for professors at the six finalist universities. Cummings departed after the workshops, and Tabit stayed on with Tadepalli to help launch classroom activities later in August. A summary of the Foundation's June Pune activities appears here.

Mundol Hires Tandon to Manage the Pilot Project

In July, Mundol selected three host universities for the pilot: College of Engineering Pune (COEP), Symbiosis School of Economics (SSE) and SNDT Women's College. In the same month, he recruited and hired Nitika Tandon to oversee participation, which includes managing the India Education Program. It had been part of the plan since March to hire a consultant to run the India Education Program, because Mundol's role includes other major responsibilities in India, such as setting up a legal entity for the Foundation, supporting and designing other outreach efforts, forging partnerships and developing mobile platforms.

Tandon began her work in early August by attending Wikimania in Haifa, where she got a crash course in all things Wikipedia. Back in India after Wikimania, she began traveling back and forth to Pune each week as the pilot project got under way. Tandon is based with Mundol in Delhi, a two-hour flight from Pune. She said it was initially challenging to get up to speed on all of the moving parts of the India Education Program.

"The importance of communicating with the global Wikipedia community on talk pages was not emphasized," she said. One factor was that Mundol had been actively recruiting a communications person, but he had not yet found the right candidate, so no one was expressly tasked with talk-page communication.

The Foundation Determines the Size of the Pilot Project

Early on, Mundol and the San Francisco team had discussed including between 150 and 200 students and a handful of classrooms in the pilot, yielding a CA-to-student ratio of roughly 1-to-10. During planning meetings between Mundol and the colleges, the numbers grew, reaching more than 1,000 participants in 24 courses by the time the Foundation and the colleges launched the Wikipedia assignment. As a point of reference, the U.S. program had 200 students in the first semester and 600 students in the second.

It's unclear why the number grew so large. From Mundol's perspective, the school directors and professors were enthusiastic and confident and pushed to have more classrooms participate. From the beginning, Mundol wanted to maximize results per investment dollar, and he wanted to design programs that had potential for scale. "The Foundation has limited resources," he said. "The question is, how do we make sure we have the biggest impact? We have to design programs so they have measurable, substantial impact, and we have to do them at some kind of scale, even in the pilot phase."

When he discussed the number with Schulenburg and Newstead, he assumed they would tell him if they thought it was too large. Schulenburg said he had questions about the total number, but he didn't want to step in and take control of key decisions away from the India team. "It's an issue of balance, between micro-managing and being hands-off with the local team," said Schulenburg. "My role is not to say, No. It's to consult and step back."

Newstead said he, too, had some misgivings, and he raised them with Schulenburg and Mundol, but they appeared to be aligned and confident, and he let the plan go forward. "I provided a reality check, but in hindsight I should have made a decision that we were creating too much risk for ourselves," said Newstead.

"I didn't know what I didn't know," said Mundol. "They had knowledge based on the U.S. experience, but they didn't get in my face with it. I wish they had."

"We had not done a large scale pilot, so we didn't know how it would work," wrote Davis in an email. "We just knew it was a much larger undertaking than we'd done with PPI."

College directors and professors gave other points of view on this topic. One director said that professors volunteered to participate because they thought it might bring them prestige, and the resulting internal competition among professors yielded a snowballing number of classrooms. Some professors said that their director pressured them to participate. According to directors and professors, when the number of participants grew, the Foundation didn't discourage it.

The Foundation Makes the Wikipedia Assignment Mandatory

Directors and professors interviewed said that they requested that the Wikipedia assignment be voluntary, because they knew that only some of their students would have the ability to succeed on it. They reported that Schulenburg, Mundol and Tabit required that the assignment be mandatory and that all students participate. "I had doubts about making all 180 of my students do the assignment, and I suggested selecting just 30 or 40, but the Foundation said, No, it had to be the whole class," said one professor. "I succumbed to the pressure."

"I knew that some of my students didn't have the writing skills, but doing a subset of my students was not an option," said another.

"If the directors and professors had really pushed back, we would have responded," said Mundol.

Davis explained via email that when some professors in the first term of the U.S. program made the assignment voluntary, it became impossible for the Foundation to predict how much support a given class would need. When only a handful of students chose to do the assignment, Foundation investments in ambassador training and other supports didn't yield an acceptable return on investment.

It's worth noting that most classes that did the Wikipedia assignment in the U.S. program had fewer than 20 students, so making the assignment mandatory did not unduly burden the professor. The professor quoted above had 180 students in his class. Although the Foundation had a reasonable rationale for making the assignment mandatory from a "return on investment" perspective, applying this logic to a class with so many students in Pune was a risk.

As the India team and the CAs began to work with professors to introduce the Wikipedia assignment in classes in August, when the semester was already underway, it became apparent that roughly half of the first cohort of CAs would not be able to participate at the required levels. Some were professionals who had jobs off campus, and others weren't prepared for the workload, so when classroom activities began in earnest, they dropped out.

Mundol and Tandon Recruit and Train More CAs

As a result, Tandon, Mundol and existing CAs recruited and trained a second cohort of 15 CAs from students on the two main campuses where the pilot project was unfolding: SSE and COEP. (SNDT had just 11 students participating.) Many of these students were simultaneously participating in the Wikipedia assignment. Around this time, Tadepalli realized that, for a variety of reasons, he would not be able to fulfill his duties as a Fellow on the project, and he withdrew.

Based on her own experience teaching herself to edit on Wikipedia, Tandon asked why the CA training didn't include more material on editing skills, Wikipedia policies and how to interact with the community on talk pages, and she integrated more of this kind of knowledge in the training for the second cohort.

Out of a total of 37 CAs in both cohorts, 12 fully engaged in the role and stuck with it through the end of the pilot, yielding an average CA-to-student ratio of roughly 1-to-85. Mundol and Tandon recognized that this ratio spread the CAs thin, compared to the ratio of 1-to-10 planned at the outset, and in early September they reached out to the India Wikipedia community and asked people to volunteer to serve as Online Ambassadors (OAs).

Mundol and Tandon Recruit Online Ambassadors from India

OAs were part of the U.S. program, but according to the PPI review published in November 2011, the role was never perfected. "We were dissatisfied with some aspects of the OA role in the U.S., and we wanted to see if we could do without it in India," said Annie Lin.

In the U.S. program, OAs were online mentors for students working on the Wikipedia assignment, and the India team advertised the OA role in keeping with the way the role had been defined in the U.S. According to Tandon, 27 people volunteered, and roughly 15 of them actually worked with Pune students in some capacity. However, some of them came from Indic language Wikipedia projects and couldn't assist very well with English Wikipedia. Others who were English Wikipedia contributors did not specialize in article creation and so were unable to assist students who were having trouble with this task. OAs interviewed said the expectations for the OA role in Pune were unclear.

Students Begin to Post Copyvios

The first students began to add material to Wikipedia in mid-August. They were in a class in which the professor set an early, 250-word deadline, and many of these contributions contained copyright violations (copyvios) and were deleted by experienced global Wikipedia editors. Most students responded by re-posting copyvio content to the encyclopedia, and this surprised Foundation team members in both San Francisco and India.

According to Mundol, most students re-posted copyvio content because they were not checking their talk pages and had no idea how the content that they had added earlier suddenly got deleted. For the subset of students who were checking their talk pages, they did not understand what a copyvio was, and they thought they were fixing the problem by simply rewording some of the content. The vast majority of students were not intentionally defying global editors and violating the rules of Wikipedia.

"For most of us, Wikipedia was a black box," said one student. "Our contributions were magically reverted, magically blocked, with no explanation. We were left trying to guess what we did wrong."

Global Editors Detect an Increase in Copyvios from India

Global editors who patrol new articles on English Wikipedia noticed an increase in copyvio content in new pages and traced the articles to IP addresses in Pune. "I had never heard of the Pune Pilot Project, nor had any of the other patrollers," said one Wikipedia administrator.

A number of global editors began to voice concerns on talk pages on both English Wikipedia and Wikimedia Meta-Wiki (see an example here). It was unclear to them who was in charge of the project, because this information was not posted in predictable locations on either Wiki. For Wikipedians, predictable locations include village pumps, Wikiprojects and notice boards.

"When I finally figured out who it was, I discovered that the only way I could contact him (Mundol) and get a quick response was to email him, and that's not how Wikipedians communicate," said one administrator.

Heading into September, an increasing amount of problematic new material began to appear on English Wikipedia. Problems included copyvios, plagiarized passages, content mistakes and English language errors. In addition, many of the contributions were on topics in the areas of computer science and engineering, which were already well covered by Wikipedia's native editing community.

India has copyright and plagiarism laws on the books, but they aren't generally enforced, and many professors accept copy-pasted and plagiarized content from students. "Piracy is rampant here," said one professor. "Students copy directly from books and get away with it."

"Before the Wikipedia assignment, copy-pasting from the Internet was standard practice for all of us," said one student.

Regarding plagiarism, one student said, "We routinely take other people's work and pass it on as our own. This idea of illegality was alien."

Global Wikipedia editors said that even before the Pune Pilot Project, a disproportionate amount of copyvio content that appeared on English Wikipedia was coming from IP addresses on the subcontinent. "We already had concerns for the past few years about a lot of work coming out of India being loaded with copyvios," said a Wikipedia arbitrator. "Of course, there are excellent editors from India, too. I don't want to paint everyone with the same brush."

Some people interviewed questioned why the India team did not anticipate that students would be prone to copyvio, given the prevalence of copy-paste practices in the Indian education system. "I expected some copyvios, but I thought they would be teething problems," said Mundol.

Mundol and Tandon Mount an Intervention

Based on the early evidence, Mundol and Tandon recognized that they were facing a problem that was bound to grow, so they mounted an intensive effort to intervene before the copyvio problem got larger. They led 20 in-class sessions for professors and students at COEP, SSE and SNDT about copyvio, and they met with the remaining professors and asked them to talk to their students about Wikipedia copyvio policy.

In addition, they reviewed Wikipedia's copyvio and plagiarism policies with the CAs, who talked one-on-one with students who had made illegal contributions. CAs began to go desk-to-desk to answer specific queries, and they asked students to do all future work in sandboxes and have it approved by professors or CAs before moving it to the article namespace.

Meanwhile, as more and more Pune students faced assignment deadlines in September, they continued to post offending material on the encyclopedia at an escalating rate. An increasing number of global Wikipedia editors got involved in reviewing and deleting illegal material submitted by pilot project students, and they began to try to interact with students on their talk pages. Many students did not respond.

"Students were afraid to interact on talk pages," said Tandon.

New page patrollers and editors working on copyright investigations were already taxed before the Pune Pilot Project. According to one administrator, new pages appear on Wikipedia at an average rate of one new page every five seconds at peak times of day, and there was already a backlog of work in the Contributor Copyright Investigations project.

Of 10 global editors and administrators interviewed for this report, none were aware that the India Education Program was running a pilot project, and the appearance of student work on English Wikipedia was unexpected.

Many Students Continue to Copyvio

In all, close to half of participating students posted copyvio material at some point during the Pune Pilot Project. The efforts of the India team to course-correct in classrooms do appear to have had some impact. A subset of students who posted copyvios‚ roughly 24 percent at SSE and 2 percent at COEP, according to Tandon's calculations‚ began to understand Wikipedia's copyright and plagiarism rules, and they adapted their behavior. "Once I understood the rules, I pulled up my socks and started again," said one student.

However, many students continued to post illegal material. "My observations from looking at what students were doing very closely at this point in time was that they struggled mostly with close paraphrasing," wrote Davis in an email. "They were trying, they just hadn't been asked to do it before and didn't understand how to."

When global editors posted warnings on their talk pages, many students either didn't respond, or they asked for leniency until their professors could grade their assignment. Students frequently used slang terms when they did respond on talk pages, and this offended global editors.

Most students seemed to have no idea that their actions were creating unexpected work for people they had never met. "I never imagined that anyone would know that I was copying and doing sloppy work, or that someone else would have to clean it up," said one student.

According to Wikipedia statistics, there are an average of 3,500 very active editors working on Wikipedia. Very active Wikipedians are those who contribute more than 100 edits per month. You can see the statistics here. A pilot project with more than 1,000 participants‚ posting a lot of substandard new material in concentrated bursts linked to assignment deadlines‚ introduced a significant amount of unexpected work for the subset of very active editors who devoted time to the task of cleanup on the encyclopedia.

"If the Foundation had asked anybody who does new page patrolling, 'What impact would 1,000 new editors from India, uploading assignments from classes, have on English Wikipedia,' we would all have said, 'That's awful, don't do that,'" said one administrator.

Global Editors Have Extended Difficulty Getting Information from the Foundation

Global editors quickly got overwhelmed and sought redress from the Foundation, but comments and questions posted on talk pages on both Wikipedia and Wikimedia-Meta went largely unanswered for many days.

For example, a note from an editor regarding copyvio and the pilot project was posted on the English Wikipedia India Education Program talk page on 21 September. The first response from a Foundation team member was from Tandon on 6 October, followed by input from CA Ram Shankar Yadev on 7 October and Mundol on 10 October.

On 11 October, LiAnna Davis posted an announcement for an IRC office hour to discuss pilot project issues. Frank Schulenburg began to participate in the conversation on this talk page on 5 November, although he began to contribute to the discussion almost three weeks earlier on the Wikimedia Meta-Wiki India Education Program talk page.

In between these communications by Foundation staff and consultants, many Wikipedians posted comments about ongoing, growing problems with content posted by pilot project participants.

The communication lag was due to a number of factors. The India team chose to give top priority to on-the-ground interventions with students and professors. At the time, the team consisted of just two people‚ Mundol and Tandon. Mundol was actively searching for a communications person for the team, but there was not yet a team member with dedicated responsibility for overseeing talk page conversations with the global Wikipedia community.

As soon as they felt the situation was under control, Mundol and Tandon engaged on the talk pages with a recap of the actions they had taken. On the San Francisco end, the Wikipedia Education Program team saw the escalating comments from global Wikipedia editors, but team members hung back from engaging because they didn't want to usurp the authority of the India team. "We were trying to work out who needs to step up to talk about which issues," said Lin.

Furthermore, Wikipedians operate online on a 24/7 schedule. Two days in the physical world is an eternity on Wikipedia, and issues can quickly snowball.

Still another factor was that the India team was communicating with CAs primarily through a closed Google group, so global Wikipedians had no idea that the India team was taking aggressive steps with people on the ground in Pune.

"We began to think than no one was driving the bus," said one global editor.

When Foundation staff and consultants did respond, their posts tended to be long. "Talk page posts need to be short and respond directly to points above," said one administrator. "In our world, we say, TLDR, which means, Too Long, Didn't Read."

An Administrator Blocks One School's IP Address

In early October, a global administrator blocked the IP address of one of the participating universities to stem the tide of copyvios. Administrators also blocked individual students. This got students' attention, and some of them stopped posting illegal content. "Getting blocked and used as a bad example was hard," said one student. "I felt bad, but I used it positively, and I figured out how to do it right."

Others continued to post offending material. In interviews, students cited a number of different reasons for this behavior. Some were desperate to pass their assignment. Others didn't understand the magnitude of the impact on the global editors. Some simply didn't have the English language skills to write their own content, and copy-paste was their only practical recourse.

"You have to understand, this is the first time in our lives that our work has been cross-checked, and the assignment was compulsory," said a student. "We had to keep adding material to get our grade."

A small minority succumbed to the pleasures of a tech-based, cat-and-mouse game and created sock puppets, one of the worst offenses in Wikipedia culture. "We're computer programmers," said one student. "It was fun to find a technical work-around. And besides, Indians are famous for breaking rules. If you don't want to wait in a queue with a billion other people, you find a way to go around the system. It's the only way to get things done. We know, now, that this was really bad."

The San Francisco Team Recruits OAs from the U.S. Program to Help With the Cleanup Effort

The cleanup job was by now enormous, and the India team asked the India OAs they had recruited in early September to shift their attention from mentoring students to assisting with cleanup. One India OA said that trying to mentor students had been frustrating, because the students were supposed to initiate the conversation, yet the students assigned to him either did not contact him at all, or they waited until 24 hours before their assignments were due.

"When I commented on student talk pages, it was like speaking to a wall," said one OA from India. "There was no answer."

When the India team requested assistance with clean up, the India OAs declined to participate, because this was not the task they had volunteered for. "Cleanup is a much bigger time commitment than mentoring," said an India OA. "It wasn't what I signed up for. The OA role should not derail my primary work as a Wikipedia editor."

"Cleanup is incredibly tedious," said a global editor.

Meanwhile, CAs in Pune did their best to manage the situation, putting in multiple hours per day, for weeks, while at the same time completing their own coursework in up to six classes. Some of them couldn't handle the unexpected and mounting pressure and dropped out, but 12 performed beyond the call of duty and stuck with the pilot project to the end.

Even so, the pilot project was under-resourced, and the CAs were under-trained. "In CA training, the copyvio issue was never stressed," said one CA. "We needed proper training in copyvio and how to use the tools to detect it."

To get help with cleanup, the San Francisco team reached out to OAs from the U.S. program, and 20 OAs volunteered to assist on the Pune Pilot Project. These OAs went right to work and made a major contribution to the cleanup effort, but CAs said they wished the work had been more collaborative. "The OAs did great work, but they sidelined us," said one CA. "They didn't even include us in the cleanup tables. Communication between the CAs and the OAs wasn't good. There needs to be direct communication."

Low Professor Participation Creates More Work for CAs

In the U.S. program, professors had played a hands-on role in the assignment, tracking student contributions, providing research advice and guidance, interacting with students on talk pages and evaluating and grading student work. In the Pune Pilot Project, only a handful of professors engaged to this degree, and CAs ended up taking on work that was intended for professors. "Participating professors need to be Internet-savvy, online every day and good at interacting on Wikipedia," said one professor.

"Enthusiasm is not enough," said another. "Even those of us who understand Wikipedia, we could have used some specialized training to work on this project."

Many of the CAs were students themselves, learning the "how to" of Wikipedia on their own assignments as they went, and some of them committed copyvios in the process of learning Wikipedia policies and practice. This infuriated some global editors.

CAs learned from their mistakes, and they requested multiple times to be taught how to patrol for copyvios and plagiarism. Two global editors did subsequently respond to this request, thereby enabling CAs to assist in the cleanup.

Some Wikipedians Bash CAs and Foundation Personnel on Talk Pages and Email Lists

Some Wikipedians criticized the CAs (see an example here), and CAs said they felt unfairly judged. "The Wikipedia community in India didn't help us, didn't speak up for us, didn't defend us," said one CA. "We're part of the community now, and it hurt."

"The current Wikipedia community seems resistant to change," said another CA. "We want to join. We want to work hard. They should let us in."

Some global Wikipedians posted criticisms of the Foundation, even after staff and consultants began to take concrete actions to address the Wikipedia community's legitimate concerns. On the India email list, some contributors posted personal critiques. See an example here.

Incomplete Student Data Lists Slow the Clean-Up Effort

Cleanup volunteers made multiple requests to the India team for a complete list of students, including usernames, IP addresses and article topics, but this information was not readily available, because students neglected to post their usernames to course pages, and course pages were therefore incomplete. The Foundation was as frustrated by this as the Wikipedia editors were, and the India team spent many hours prodding students to add their usernames to course pages.

Tensions in the global Wikipedia community continued to escalate. At one point, one global editor posted a note on his talk page forbidding anyone from Pune to write on the page. You can see this page here. Indians interviewed said they found this offensive. The editor who posted the banner later apologized.

The Foundation Shuts Down the Pilot

In mid-October, some editors discussed taking the extraordinary step of requesting the Arbitration Committee, which oversees major disputes regarding Wikipedia editorial content, to step in to resolve issues with the pilot project. Although editors did not ultimately do this, one arbitrator interviewed voiced serious concerns about the pilot but noted that it was not the Arbitration Committee's role to interfere in Foundation programs.

Shortly thereafter, global Wikipedians requested that the Foundation shut down the pilot project. The Foundation took the situation seriously, and when India team members flew to San Francisco for a Foundation-wide retreat, they also met with the Wikipedia Education Program team to discuss solutions. At this meeting, Newstead instructed Mundol to shut down the Pune Pilot Project.

The India team returned to Pune and met with the director of COEP, the university where the majority of the pilot students went to school, and informed him of the Foundation's decision. The program at COEP stopped on 3 November.

"It was very, very frustrating," said one student. "Serious students like me, ready to make improvements to our articles, couldn't do it."

Some Students Succeed

According to data compiled and analyzed by the Foundation, 266 students contributed content that survived cleanup. Students interviewed from this group said that they were disappointed to see the program close. Examples of successful student work include: Partial equilibrium, Human capital, Robinson Crusoe economy, Public sector banks in India, Structural model (software), Double-ended priority queue, and Object database.

When asked what factors contributed to their success, students cited the following:

• Guidance from the professor on topic selection;
• A research phase in which students read in depth about their topic before beginning to write;
• Solid writing skills in English;
• Composing in the sandbox and having their work reviewed by the professor and the CA before posting it to the article namespace;
• A professor who evaluated their work multiple times over the course of the assignment;
• A responsive CA who was knowledgeable about Wikipedia policy and practice;
• Friendly and helpful interactions with global Wikipedians on talk pages;
• Clear and detailed feedback on talk pages regarding what, exactly, was wrong with the article;
• Encouragement from the India team and CAs in the form of kittens, cupcakes and other Wikilove on their talk pages, and
• Having the opportunity to revise their work and learn from their mistakes.

Some Professors Are Frustrated

Professors whose classes had been on track with the assignment were left in a difficult position by the shut down, because the assignment was slated to play an important role in student evaluations for the semester. "How do I evaluate my class?" asked one professor. "This really messed me up, and it caused injustice to some students. It was a very bad experience."

At COEP, some professors reported that they were disillusioned with the Foundation and that they would pursue the Wikipedia assignment in future classes, independent of the India Education Program. "The Foundation was too demanding and pushy and didn't listen, and I'd rather do it on my own," said one professor who is a Wikipedian.

His course had more than 140 students. According to data compiled and analyzed by the Foundation, roughly 85 percent of his students made edits in the article namespace, but just 15 percent of this content survived cleanup, making the course one of the biggest contributors to the cleanup crew's workload. It's also worth noting that the course topic was already well covered on English Wikipedia, and students in this class said that they struggled to find suitable topics for the assignment.

At SSE, a core group of professors viewed the pilot favorably and said they would like to continue the collaboration with the Foundation.

What the Data Say

Statistical data analyzed by the Foundation indicate that the pilot yielded a high percentage of low-quality content being added to English Wikipedia. Out of 1,014 registered students, 665‚ 66 percent‚ actually made edits in the article namespace. These 665 students contributed 702,961 words to the encyclopedia, but just 21 percent of this total survived cleanup, as of mid-December.

Of these 665 students, 266 students‚ 40 percent‚ contributed content that survived the cleanup.

Overall, SSE did better than COEP. For SSE, 29 percent of total content survived cleanup, compared to just 13 percent for COEP. Factors that contributed to this include: The economics students at SSE generally had better English writing skills than the engineering students at COEP, and COEP students were writing on topics that were already well covered on English Wikipedia.

One global administrator estimated that the total impact on the community was between 2,000 and 3,000 hours. The Foundation hasn't yet found a good way of systematically measuring the impact, but it acknowledges that the impact was enormous.

Overall Assessment

The Foundation pursued a creative and bold pilot project that had problems, and it resulted in a significant amount of unexpected work for the global English Wikipedia community. It also caused stress for CAs, OAs, students and professors.

Looking back, the biggest problems on the Foundation side were: Not engaging the community as a partner in the planning process; not communicating clearly about the program in logical places on English Wikipedia; not doing sufficient due diligence in regards to understanding the Indian education system; not listening to input from Indian colleges while setting up the project, and confusion between the San Francisco and India teams over roles and responsibilities.

Although two global Wikipedians interviewed for this report called for Foundation staff and consultants to lose their jobs over mistakes they made planning and managing the Pune Pilot Project, the evidence suggests that this action would be premature.

As enumerated above, the Foundation made mistakes, and these mistakes resulted in a large burden on the English Wikipedia community. The pilot project narrative indicates that along with the mistakes, Foundation staff and consultants generally responded quickly to problems as they emerged during implementation. However, they did not report these actions in a timely manner in places where the greatest number of Wikipedians would see them, which is on Wikipedia talk pages. The result was that much of the Foundation's activity in San Francisco and in India was invisible to the global community.

To address this, the Foundation is systematizing communication about the India Education Program by posting it in logical locations on English Wikipedia and dedicating a new staff position to communicating with the Wikipedia community.

There is also evidence that Foundation personnel are actively analyzing what went wrong in Pune and applying the lessons learned as they plan and launch the next Wikipedia Education Program pilot project, which is in Cairo, Egypt, and on Arabic Wikipedia. See a comparison of the Cairo and Pune Pilot Projects here and the plan for the Cairo Pilot Project here. The India team, too, has devoted considerable time and attention to analyzing what went wrong in Pune. You can see their lessons learned here.

The Foundation also understands that it needs to include the Wikipedia community in planning and listen more to locals during planning and implementation. "We need to spend more time talking to people, listening, asking questions," said Schulenburg.

This includes taking time to develop an understanding of the education system and norms in target geographies. Examples of important dimensions of the local education context that had direct bearing on the Pune Pilot Project but that no one flagged during pilot planning include the following: Education in India emphasizes rote learning, copy-paste is a norm in student work and professors condone it, and many Indian students who learn in English as a primary language of instruction are not proficient writers in English. These realities had implications for pilot project design and for possible impacts on the native Wikipedia community, and the Foundation missed them.

The global Wikipedia community and the Wikipedia community in India must also take responsibility for harm done during the Pune Pilot Project. The main failing in both communities was that some people demonstrated a lack of civility in communicating about problems with the program, causing hurt feelings and damaging morale. Despite the monumental amount of unexpected cleanup work, there is no cause for bashing and mocking people on talk pages and email lists, particularly when so many people are working hard with good intent toward a common goal.

"The goal is to increase participation on Wikipedia," said one editor. "Everyone should come out feeling good."

Recommendations Going Forward

Following is a set of recommendations for revamping the India Education Program. These recommendations stem from suggestions from interview subjects and from the evaluator's professional opinions.

Engage the Global Wikipedia Community as a Partner in Planning

People interviewed agree that the majority of problems that emerged during implementation could have been largely avoided by engaging the Wikipedia community as a partner in the pilot project planning process. "The global Wikipedia community is an amazing resource," said one administrator. "We understand how Wikipedia functions as a society, and we have lots of diverse professional experience. We are resources that the Foundation is overlooking."

Mundol did post plans and request public comment on Wiki, but not in the most effective locations. Wikipedians recommend that in the future, announcements about programs be posted on village pumps and notice boards at relevant WikiProjects and maintenance-based projects, and that each announcement include a link to planning documents and a central communications page on English Wikipedia. "Our planning process was not adequately collaborative," said Mundol.

Examples of knowledge the global Wikipedia community could have contributed in the planning phase include:

• A pilot with more than 1,000 students will result in an unmanageable amount of new material for a finite editor community to monitor;
• Indian students will contribute copyvios and plagiarized content without in-depth, customized training;
• A high percentage of CAs and OAs will need to have significant hands-on experience in English Wikipedia policy and practice to effectively guide Indian students and professors;
• Complete student lists must be created and made available to the community to facilitate tracking and monitoring;
• Students should write on topics that fill gaps in existing Wikipedia content.

Take Advantage of New Talent Born of the Pilot

On the ground in Pune, English Wikipedia now has a cadre of experienced people in key roles, and they are ready to work. A handful of professors at SSE are actively engaged and wish to continue to work with the India Education Program. "Yes, there were bumps in the road, but this is a great idea," said one professor. "Students learned to collaborate. They had to do research, synthesize information and write it in original language. And they got their work read by a global audience. It was more work for me, but it was worth it."

A total of 266 students contributed content of acceptable quality to Wikipedia. Many students who made mistakes early on learned from them and went on to succeed. One student even went from being blocked to getting a barnstar from an administrator.

There are 10 competent CAs and 10 experienced students who want to train as CAs. These CAs and students spend their lunch hour debating Wikipedia policies and arguing over the best way to create tables and graphs. They check their talk pages and page views daily, and they joke about being "Wikiholics."

"A real community of people has grown up around Wikipedia in Pune," said one CA.

"Wikipedia should leverage this new talent," said another.

Online, English Wikipedia has a core group of global editors who want to participate in a revamp of the pilot project, in roles ranging from advisors to OAs. One administrator who lives in Asia and has experience in the Indian education system has volunteered to spend a few months in Pune working with CAs and students. In Pune, three local Wikipedians have volunteered to mentor students, and one has indicated that he would like to participate in creating course materials on plagiarism, copyright, paraphrasing and proper referencing/citations.

At the Foundation, staff and consultants have learned many lessons and are in good position to lead a successful second phase of the pilot project.

Design the Phase-Two Pilot to Test a Variety of Copyvio Interventions

For good reason, the copyvio issue is of paramount concern going forward. Based on experience accrued prior to the Pune Pilot Project, the global Wikipedia community had legitimate concerns about copyvios appearing at a regular rate in new content contributions from India.

In the Pune Pilot Project, posting of copyvio content was a serious issue. A qualitative assessment of some of the articles contributed by students by the Foundation's India team indicates that overall, 45 percent of students who participated posted copyvio content, an unacceptably high percentage. And even after their crash-course experience learning about copyvios during the Pune Pilot Project, CAs from Pune continue to have some trouble understanding the nuances of trademark infringement. In a recent example, a CA used the Wikipedia globe logo, which is trademarked, in a logo for the new Wikipedia Club Pune and posted it on Wikimedia Commons here. It has been nominated for deletion, and a conversation is in progress here regarding ways to correct the logo.

The Pune Pilot Project was not designed to evaluate whether copyvio norms in India are a deal-breaker for the Wikipedia Education Program, so it is too soon to conclude that copyvio norms in India are an insurmountable problem for the program. The next phase of the pilot should be designed to expressly investigate which parameters and interventions can result in high-quality, copyvio-free content contributions by Indian university students.

Suggestions for Heading Off Student Copyvios

The Pune Pilot Project experience and interviews with students who succeeded suggest that certain supports may contribute to heading off student copyvios and other low-quality contributions.

• Keep the phase-two pilot small.
• Establish core metrics and put systems in place to track them. Do continuous evaluation and course-correct as the pilot unfolds.
• Target advanced undergraduates.
• Target students who already write well in English. Professors and school directors suggested that students majoring in social sciences, humanities and economics are most likely to have this skill.
• Choose professors who can write well in English, have deep content expertise, want to work on Wikipedia, want to engage with students on talk pages and understand that the task involves continuous evaluation over the course of the semester.
• Give students a choice between the Wikipedia article and a more traditional assignment. Consider giving students incentive to choose the Wikipedia assignment (extra credit, a certificate).
• Attract talented students by highlighting the aspects of doing the assignment that attracted the students who succeeded in the first pilot project: Reaching a global audience, getting feedback on work from global editors, learning to collaborate and learning research and writing skills.
• Enlist CAs who are actively contributing acceptable content in the article namespace and who have good people skills.
• Line up OAs in relevant topic areas before classes begin and give them an accurate job description.
• Train CAs in Wikipedia culture and policies, creating and editing articles, interacting on talk pages and where to go to get help. Train them to identify copyvios and plagiarized content.
• Develop streamlined communication mechanisms so OAs and CAs can work together efficiently.
• Design detailed, face-to-face lessons on copyvio and plagiarism for students and professors, and teach the skills of citation, paraphrasing and the acceptable and unacceptable ways to reflect copied text and ideas in Wikipedia articles.
• Include a research phase for students so they are knowledgeable about their topic before they begin to write.
• Have students compose new articles in sandboxes.
• Have a structured, logical sequence of deliverables over the course of the semester.
• Establish benchmarks for all roles, and measure progress against them. Drop people who don't meet benchmarks. Make it clear to students from the beginning that if they copyvio, they will get dropped.
• Have reasonable CA-to-student and OA-to-student ratios.
• Design and evaluate the phase-two pilot such that it is possible to determine which interventions are most effective at heading off copyvios.

Other Suggestions for the Phase-Two Pilot

• Consider implementing a well-designed, phase-two pilot in two locations:
  • Symbiosis School of Economics (SSE) in Pune, to foster good will and build community between global Wikipedians and student Wikipedians in India, and
  • A top-tier university (such as IIT or a top humanities college) in Delhi, where the India team is based, to test the overall viability of a Wikipedia Education Program in India.
• Communicate regularly with editors in relevant WikiProjects and maintenance-based projects. Broadcast information about the pilot project on village pumps, notice boards and mailing lists, and include links to key planning documents and discussion pages. Keep the native Wikipedia community informed of deadlines and progress against goals.
• Choose classes in subjects that are underrepresented on English Wikipedia.
• Connect professors to editors in relevant WikiProjects so they can work together to select topics for student articles.
• Require that professors and CAs edit on Wikipedia as the semester progresses, because students look to them as role models.
• Recruit and train qualified OAs, and perfect the role. Bear in mind that the OA talent pool is finite. Based on interviews, OAs want to mentor students in content, Wikipedia policy and writing style. They are less interested in cleanup.
• Pilot wiki-based communication channels for CAs and OAs.
• Create an accurate, machine-readable list of students, usernames and article topics.
• Use physical course materials and online "how to" videos created for the U.S. program, evaluate their effectiveness with Indian students, and create new materials and videos as necessary to support Indian students.
• Teach students and professors about Wikipedia rules, practices and culture, including:
  • Interacting on talk pages and using Wikimarkup;
  • Avoiding copyvios and plagiarism;
  • How to paraphrase;
  • Checking and adding references;
  • Wiki syntax;
  • How to write a proper Wikipedia article;
  • How to interact with mentors and other editors online, and
  • How to respond constructively when work gets deleted or reverted.

Improve Coordination Between the San Francisco and India Teams

The India team is the first team established outside of the San Francisco office, and there has been some confusion in working out the roles and responsibilities of the different teams. The Foundation is right to empower locals to lead, but it also has to provide a safety net and step in with relevant knowledge when it's needed. "Going forward, we'll speak up more if we think we're seeing a red flag, and worry less about causing offense," said Lin.

On the India side, the India team should communicate clearly and in a timely manner about its decisions and its work on the ground. San Francisco team members aren't clear on how India team members spend their time, and vice versa. The two teams should discuss this, and create descriptions of roles and responsibilities and mutually agreed-upon milestones. The Wikipedia Education Program should post this information on English Wikipedia.

Going forward, the San Francisco team will include the India team in more Wikipedia Education Program activities, including team meetings and lessons sharing sessions. "We had weekly phone calls before, but it still felt far away," said Lin.

Take Advantage of the Break Between Pilots to Build Capacity and Relationships

During the pilot, Mundol learned that gap time is opportunity time. "We had a gap between training the CAs and starting work in the classrooms," he said. "We could have used this time constructively to build Wikipedia skills, communicate with the global community and get the data infrastructure set up."

Based on Mundol's "lesson learned," all of the human assets that are a positive byproduct of the pilot project should actively work together to hone their skills and cohere as a team between now and when the next classroom project begins later in 2012.

Suggested activities include:

• Professors get more experience editing articles, interacting on talk pages and selecting viable article topics for the next cohort of students.
• Foundation staff and consultants compile and synthesize all constructive input from relevant Wikipedia and Wikimedia communication channels and publish clear information about each team member's role and responsibilities.
• Existing and aspiring CAs in Pune work on Wikipedia, with experienced editors serving as their mentors. Topics suggested by CAs and global editors include:
  • Understanding key Wikipedia policies and procedures,
  • Editing articles,
  • Paraphrasing,
  • Creating tables and graphs,
  • Citing references properly,
  • Interacting effectively on talk pages,
  • Reviewing other people's articles,
  • Studying good articles and analyzing what makes them good, and
  • Learning to patrol for copyvios and plagiarism.

Expand the Existing Online Knowledge Portal

In India, word about the Wikipedia assignment is spreading across the university world, and it is likely that more and more professors will begin to assign Wikipedia articles to students in lieu of term papers, with or without support from the Foundation. In addition, the list of new courses on the School and University Projects page on English Wikipedia is growing at a rapid rate.

The Foundation should consider expanding the existing online knowledge portal (see it here) and making sure students and professors know about it. Link it to narrative case studies of successful efforts and a suite of practical video "how to" lessons in the voices of students from countries where the Wikipedia Education Program is working. See Khan Academy for examples of effective education videos.

Update the Wikipedia User Interface

Nerdy students can deal with the Wikipedia interface. In fact, they like it. But for most students, accustomed as they are to contributing content to the Internet via Facebook and Twitter, the Wikipedia interface is complicated. Many people interviewed in India, including professors and university directors, asked why the interface isn't more user-friendly. "We're dealing with the Facebook generation," said one CA. "It should be easier to contribute, and a little more fun."

The Foundation has been working on one tool to address this issue – a visual editor‚ that is now available to the Wikipedia community for prototype testing. See the tool and an explanation of it here.