Wikipedia:Authority control integration proposal

This Wikipedia page has been superseded by the project FAQ, as the project has now concluded and is retained primarily for historical reference.

Video Summary of the proposal

On youtube.

Introduction

This proposed project intends to extend and systematise the use of authority control identifiers, using the {{Authority control}} template, on English Wikipedia articles. Authority control is the term-of-art in librarianship, archival practice and related fields for unique identifiers to disambiguate objects (people, places, academic subjects, etc.). These fields of study have different conceptualisations of unique identifiers form some other fields because many systems in place are backwards-compatible to pre-computerisation systems. This project aims to connect the English Wikipedia to this long tail of identifiers.

The current proposal focuses on biographies, although this may be extended in future to cover other topics, and is built around the use of data from VIAF, a composite system bringing together several major authority files. VIAF algorithmically matches and clusters entries from the individual authority files, and uses data scraped from Wikipedia to aid the process; as a result, there have already been a large number of Wikipedia-VIAF matched pairs identified and this provides a very effective springboard to work from.

The proposal was originally written up here, and discussed on the Village Pump. It has since been updated to include some of the feedback and commentary received during the discussions. While the Village Pump discussion was broadly favourable, it has been formally listed as an RFC in order to ensure clear support from the community before implementation later in 2012.

This plan is being coordinated by Max Klein, the Wikipedian in Residence at OCLC, and Andrew Gray, the Wikipedian in Residence at the British Library. OCLC are the central operating group for VIAF, and have offered to provide technical support for the matching process. If you would like to help work on it, please let us know.

Background

Authority control is a system primarily used in libraries and other metadata services, where a single entity is given a canonical unique identifier. This allows clear disambiguation between different entities with similar names, while also allowing the use of a single identifier for those with multiple variant names. On Wikipedia, this is handled with the {{authority control}} template, which places the identifiers at the end of the article and links out to library catalogues and central authority databases.

As well as these reader-visible links, the embedded data helps build infrastructure for future work, such as:

Reliable linking from external services - we can build lookup services, such as this tool for the German Wikipedia's PND files: http://toolserver.org/~apper/pd/person/pnd-redirect/de/118768581 - which takes you to the article represented by that PND. Such tools allow people to automatically generate links to Wikipedia without guessing at article titles, use the API to pull out leads from articles for reuse in other sites, etc.
Extending the scope for checking metadata - we already have methods, such as the Death anomalies project, for comparing the metadata between Wikipedia language editions and spotting inconsistencies. Including identifiers which tie into external services, with reliable APIs, give us a lot of additional data for cross-checking.
Returning metadata to the outside world - working backwards from this, once we have embedded identifiers, the curators of this metadata will find it a lot easier to incorporate information from Wikipedia, taking advantage of our fairly fast update cycle for things like death dates.
Identifying alternate names - particularly for non-standard transliterations, the alternate headings in authority files give us an extensive and curated collection of variants of names. The linkage will help the creation of redirects.
Content creation support - the presence of the identifiers allows future work on tools to, e.g., develop scripts to generate author's bibliographies for articles.

Currently, around 4,000 articles on the English Wikipedia have some form of embedded authority control identifier, and on Commons, around 45,000 articles contain authority control. On the German Wikipedia, by comparison, around 220,000 articles have embedded identifiers.

The proposal

This initial proposal focuses on identifiers in biographies; however, it is not intended to be exclusive, and the system can be extended in future to other articles if there is community support for it.

It is built around use of the Virtual International Authority File (VIAF), an international project to merge multiple national authority files into a single master system. VIAF identifiers correspond to identifiers in other systems, and can be used in parallel with, or instead of, these other identifiers.

The process will involve identifying an appropriate VIAF identifier to match to as many articles as possible, using a number of different methods ranked by probable accuracy. Following this, and testing of the data to ensure it is consistent and accurate, a VIAF identifier will be added to these articles by a bot, using an extended version of the {{Authority control}} template. This tool can later be reused to include other identifiers, such as LCCN, if desired.

Data sources

There are three available sources of data:

Articles already using {{Authority control}}. Some of these will have VIAF numbers. Where they do not, we can use the LCCN/GND numbers to match a VIAF number and include it in the existing template.
Interwikied articles with identifiers. Around 220,000 articles in the German Wikipedia have identifiers. Where an interwiki to the German Wikipedia exists, we can pull the identifier from the linked page, doing some basic metadata checks to ensure the interwiki linkage is accurate.
Around 145,000 articles on the German Wikipedia currently have VIAF identifiers; the rest use other identities, but it may be practical to match them to VIAF.
VIAF authority file links. As part of the matching process, Wikipedia is used as a source of information to help bring VIAF "clusters" together. OCLC have provided an extracted list of over 250,000 English Wikipedia articles with corresponding VIAF numbers, though these may have to be checked to ensure that pages have not been moved since the matching was carried out.
(The matching is done with this python code written by OCLC Research Scientists Thom Hickey and Jenny Toves. During the algorithmic creation of the VIAF file if a Wikipedia link is matched with ~98% accuracy then it is included in the entry. Right now there are 266,202 links from VIAF to Wikipedia. Those links are available as a tab-delimited text file.)

Implementation

The implementation will be done in stages.

Create lists of page titles and associated VIAF cluster IDs from the enwiki dataset, the dewiki dataset, and the VIAF dataset. These will then be sampled to check for accuracy.
Prior to the bot run, {{Authority control}} will be redeveloped to ensure it scales effectively to the new usage, creating sub-templates for specific identifiers. The documentation for this template, along with Wikipedia:Authority control, will be checked and updated or overhauled where necessary.
A bot will be developed and tested, then approved through the standard bot approval process to ensure there are no technical problems and that it is compliant with this proposal.
This bot will add {{authority control}} along with the VIAF codes from this list, once testing is complete.
Finally, this bot will run periodic reports in conjunction with the VIAF update schedule, to reflect any reshuffling that occurs in the file.

↓Onwiki discussion to develop proposal

↓RfC on finalised proposal

↓Creation of processes and bots; bot approval

↓Deployment of content

↓Analysis and maintenance; possible second phase (non-biographies)

Phase I

Phase II

Phase III

Phase IV

Phase V

│

June

│

July

│

August

│

September

│

October

│

November

│

December

Towards Deep Authority Control Integration

Onwiki discussion to develop proposal (by late June)
RfC on finalised proposal (by mid-July)
Creation of processes and bots; bot approval (by end of July)
Deployment of content (through August)
Future: Wikidata integration (part of phase 2 of Wikidata - entirely dependent on that schedule)
Maintenance (...ongoing...)

Template details

The template currently used to handle authority control data is {{Authority control}}; it is placed at the extreme end of the article, just above the categories, and displays a narrow box with the identifiers. These link to an external service. For an example, see Fyodor Dostoyevsky - this uses GND, LCCN, and VIAF codes, and is nested under a navigational template following the external links. It will only be used on "main" articles, and not on subpages or related bibliographies - no two articles should share an identifier.

As part of this project, we will need to rewrite {{authority control}} to form a wrapper for a number of subsidiary templates, each handling a specific identifier. This will make it easier to maintain as well as easier to develop support for other identifiers, without the need for experimentation on a template used on several hundred thousand pages. Documentation on {{authority control}}, Wikipedia:Authority control, and related pages will be updated accordingly.

Frequently asked questions

How do I add a subject's VIAF to the article about them (or mine to my user page)?
Use {{Authority control}}.
Why use VIAF and not another identifier?
VIAF is a composite of several existing authority control databases, and so includes all the content from many of the other systems. Any entity with, for example, a LCCN should have a corresponding VIAF number as well, but not every entity with a VIAF number will have an LCCN. Adding VIAF does not preclude the inclusion of other identifiers (and may indeed make it easier); this isn't aiming to impose a sole standard.
Why only people?
The authority control system does cover other things, but for the moment (written 2013) we are only planning to cover people—this is to simplify the initial program, as well as target the articles where the template is most likely to be useful.
What about errors in VIAF?
You can report apparent errors in VIAF (or its constituent catalogues) at Wikipedia:VIAF/errors. These are then available to the relevant managing body, and for linkage repair on-Wiki. For the German equivalent noticeboard, see de:WP:PND/F.
What about licensing?
VIAF is licensed as ODC-BY, which is compatible with Wikipedia licensing; the use of a VIAF URI is sufficient attribution for the terms of the license.
Will this give any control over Wikipedia content to third parties?
No. While we will be including VIAF identifiers, the content of Wikipedia and VIAF will remain entirely separate. No metadata will be imported automatically from VIAF, nor will Wikipedia need to follow VIAF naming conventions.
What if editors object to the template or the identifier?
Editors of specific pages will in all cases be free to remove the metadata where it is inaccurate or felt to be editorially inappropriate. For the purposes of Wikipedia:Sanctions, the first revert of an automated or semi-automated addition of authority control information shall not count as a revert.
What about pages covering two people?
There are many cases where a single article deals with two individuals. If two VIAF identifiers refer to the same article, this will be logged but not added to the article; if it currently contains one but not the other, or a mixture of identifiers referring to both, this will also be flagged.
What about Wikidata?
Wikidata includes authority identifiers. However, adding the template now allows us to gain the benefit of having this information available before Wikipedia transcludes it from Wikidata ; it also will simplify any future work to add these identifiers to Wikidata.
What about cases where several people have the same name?
The primary purpose of authority control records is to help distinguish between people with the same (or similar) names. As such, identifiers are usually not matched on the name alone; the software is able to take account of other information such as birth and death dates.
I wrote a new biographical article, how do find the VIAF identifier?
Thank you for contributing to Wikipedia! You can look up a subject's VIAF at http://viaf.org/ Enter their name as the "Search Terms:", and leave the other parameters at their default values. If there are two or more entries with the same name, check the listed works for a match. If you're not sure which to use, you can ask for advice at Wikipedia talk:Authority control.
I have another question
Any comments, criticisms, etc. will be gratefully received, again at Wikipedia talk:Authority control.

- Max Klein, OCLC Wikipedian in Residence, and Andrew Gray, British Library Wikipedian in Residence.

Progress

Now that RFC has passed, the work of the bot is underway. Code can be viewed at github.