Wikipedia:WikiProject Articles for creation/AfC Process Improvement May 2018/Background

Objective

I'm Marshall Miller -- I'm a product manager working with the Community Tech team at WMF, and I started at WMF in March 2018. I have been following along with the recent conversations around the future of Articles for Creation (AfC), and I would like to start a conversation about whether the Community Tech team can use some of our bandwidth to deliver a focused improvement to the AfC process during this quarter. Though I’m new to AfC, this page attempts a review of the background, challenges, and potential improvements to the AfC system with the objective of narrowing down to a wise decision on a single focused improvement to make to the process. Although I have read about several ideas for important major changes to AfC, we unfortunately don't currently have the bandwidth to take on major changes -- we only will have the bandwidth to work on something narrow during April and May.

You'll see that this page is not yet saying what we're going to do -- just laying out some of what appear to be the most important considerations and ideas for improvement. And I'm sure that my newness means this page doesn't do a perfect job of capturing AfC's current status -- so please help! I would love for this to be the start of a conversation over the next week or so to help narrow down to something valuable we could contribute to AfC. By Tuesday, April 10, I'm hoping we have circled around two or three potential improvements that engineers can scope, so we can decide how to proceed.

Please discuss on the Talk page.

Background

The ACTRIAL initiative showed that prohibiting newly registered users from creating articles in main namespace increases the overall rate at which high-quality articles arrive in main namespace, while reducing the workload on people patrolling the incoming new articles via the New Page Patrol process (NPP). The ACTRIAL modification, however, diverts those newly registered users to create their articles in the draft namespace via the Article Creation Wizard and then to submit them to the separate Articles for Creation (AfC) process, thereby increasing the burden on those people reviewing drafts. AfC is a critical workflow for the development of new articles, having guided thousands of new articles to the main namespace. It is important to keep the AfC process as productive as possible.

The analysis of ACTRIAL results kicked off a community discussion about potential improvements to both the NPP and AfC processes. Though the decision to implement ACTRIAL’s changes permanently is ongoing, the WMF’s Community Tech team has bandwidth during 2018Q4 to address a specific improvement to AfC, and so the team has been learning from the community about what improvement could be most valuable.

The broader discussion around AfC seems to highlight a set of main issues that I’ve attempted to summarize below. Please feel free to add summaries to the Talk page for any I’ve missed:

• There is discussion around which of these functions AfC serves:
  • AfC as a filter for inappropriate articles from newbies — in which case the bar for approval should be relatively low, so that notable topics are in the main namespace sooner rather than later.
  • AfC as a process for bringing strong articles into existence through iteration between newbie authors and experienced editors — in which case the bar for approval should be relatively high, so that once articles reach main namespace they are of strong quality.
• A potential concern is an influx of drafts caused by ACTRIAL’s process changes, but more data research is needed to fully understand the extent to which AfC’s backlog is manageable.
• There is discussion around whether AfC review is too strict, with articles being declined that would pass NPP.
• There is discussion on whether AfC should be checking notability before declining instead of putting the full burden of proving notability on the author.
• When articles are in Draft space, they are much less likely to be collaborated upon, thereby making it more difficult for those articles to improve.

Principles

From reading the discussion, it seems that these are some things that everyone might agree should be true about the AfC process. Any improvement to the AfC process should buttress one or more of these principles.

• High quality new articles should get moved to main namespace as quickly and as often as possible.
• Inappropriate new articles should get blocked from moving to main namespace.
• AfC reviewers should be able to be impactful without burning out.
• Good-faith new editors should have a positive first experience and become active editors.

Challenges

From the discussion, it seems that the following are challenges that can get in the way of fully realizing the principles listed above.

• Good articles can sit and wait for review in AfC instead of being in the main namespace.
• If the AfC process is challenging for a new editor to navigate, drafts that have the potential to become high quality articles may not progress. That could happen if:
  • The new editor does not understand the review process.
  • The new editor does not understand the review criteria.
  • The review process takes long enough that the new editor becomes disengaged.
• New editors who have a confusing or frustrating first experience may be turned off from continuing to contribute and becoming active editors.
• AfC reviewers may have a large backlog, which may cause them to burn out. This is especially true if the backlog of submitted drafts grows faster than reviewers can handle while maintaining high quality standards.
• It can be difficult for AfC reviewers to ascertain notability on topics in which they do not have expertise.
• The volume of inappropriate submissions from paid editors and others with conflicts of interest make it difficult to pay adequate attention to good-faith new contributors.

Goals and metrics

Based on the principles and problems above, I’ve listed three potential goals that we could be pursuing with this effort. They are listed below with the metric needed to measure them, and an example of how that metric might look.

Although the “Narrow” goal might improve the experience of reviewers, it could have adverse effects on content quality and new users.

The “Broad” goal may do a better job of promoting both the content of the encyclopedia and the experience of reviewers and new editors at the same time.

The “Alternative” goal may be partially accomplished by pursuing the “Broad” goal, but should probably not be the primary objective of this Community Tech initiative.

Type	Goal	Metric	Example
Narrow	Help reviewers get through the backlog faster	Length of time an article is a draft in review	Drafts submitted to AfC in May 2018 spent X days waiting for their first review and Y days of total review-backlog time over their first three reviews.
Broad	Get more quality articles in main namespace faster	Share of submitted drafts from a given cohort that remain in the main namespace without being deleted (or perhaps nominated for deletion) within a given time horizon	X% of drafts submitted to AfC in May 2018 were in the main namespace and not nominated for deletion 60 days after their submission to AfC.
Alternative	Make AfC a process that grows active editors	Share of new editors who interact with AfC that become active editors	X% of new users who submitted articles to AfC in May 2018 were active editors in July 2018.

Potential improvements

If we decide to pursue the “Broad” goal above, we’ll need to decide how to improve the software or process of AfC. The list of potential improvements in the table below is my summary from the community discussion on many wiki pages. I’ve attempted to make this list thorough, but it’s important to note that some of the improvements could be out of scope for the amount of engineering bandwidth the Community Tech team can spend. The potential improvements are tagged by which of the following five methods they take to pursue the “Broad” goal:

• A: Help newcomers understand how to develop strong drafts.
• B: Improve communication between reviewers and authors to decrease iterations.
• C: Standardize reviewing criteria across reviewers.
• D: Increase the speed/ease that reviewers can do their workflow.
• E: Help reviewers find more promising drafts sooner.

As a community, we'll want to narrow in on one of these improvements (or an unlisted one if I missed something with a lot of potential) for the Community Tech team to work on in partnership with the AfC reviewers. That decision is pending a discussion about this page.

Improvement	A	B	C	D	E
Bring notability to the forefront of the guidance in the Article Creation Wizard -- right now, the wizard does not mention notability.	X
Route drafts to AfC reviewers who are experts in the draft’s topic, and more likely to be able to detect notability.		X	X		X
Rotate reviewers on repeat submissions instead of routing repeat submissions to the same reviewer.			X
Alter template language to clarify the AfC process for new editors and to encourage them to improve their drafts.		X
Make it easy for reviewers to search for notability, potentially by implementing something similar to the “Find sources” links in Articles for Deletion.			X	X
Take steps toward a shared interface between AfC and NPP, since those processes are similar in many ways.			X	X	X
After three declines, move draft to Miscellany for Deletion.				X	X
Apply machine learning models via ORES to predict the likely quality of submitted drafts, surfacing the best or worst to reviewers to streamline their workflows.				X	X
Make it clear to autoconfirmed users that they can move their drafts to main namespace themselves without waiting for review.		X		X
Develop a way for drafts to be discoverable by editors who wish to collaborate and improve them.	X
Any of the many improvements to the Helper script listed in github: https://github.com/WPAFC/afch-rewrite/issues		X		X
Move templates and comments to be posted on Talk pages.		X
Improve intuitiveness for in-line citations.	X
Automated checks for copyright violation as part of the Helper script.				X	X
In-line commenting so that reviewers can give specific feedback.		X
Thorough and centralized reporting on the AfC backlog and throughput.			X		X
Recognition or commendations for productive reviewers				X
Prohibit very short drafts from being published or from being submitted	X				X

Next steps

This page can hopefully serve as a focus for community discussion around a potential discrete improvement, and for feedback on whether I am seeing AfC's current situation and challenges accurately. Given when engineering resources are available, we will have ideally have identified two or three potential improvements by Tuesday, April 10. Then we will scope the potential improvements to determine which makes most sense inside the bandwidth of Community Tech's engineering. I will add updates to this page as this process unfolds.

Update 2018-04-06

A good amount of discussion has taken place on the talk page this week, and we'll try to consolidate around some leading ideas next week. I have modified this page to include some points raised in the discussion:

• Added seven new potential improvements that were identified on the talk page.
• Added two additional items in the "Challenges" section that were raised on the talk page.
• Rephrased the first method for improvement to read, "Help newcomers understand how to develop strong drafts."