User:Novem Linguae/Essays/Copyvio detectors

This is a summary of enwiki's various copyright violation detector bots and tools.

Detection via Google searches

https://copyvios.toolforge.org/
maintainer: The Earwig, Chlod
source code: https://github.com/earwig/copyvios
last commit: 3 years ago N
tech: Python
uses Google search API and the WMF eranbot Turnitin API
- Google Search API
  - WMF pays for credits
  - no discount (NPerry (WMF) used to work on Wikimedia's partnership with Google, maybe this is something worth bringing up?)
  - hard daily limit (maximum for any user of this API) of 10,000 queries per day
  - costs US$50 per day
  - makes up to 8 queries per page
  - 2,000ish checks per day (not all checks use all 8 queries)
  - as of Aug 2024, hitting the quota around hour 12 of the 24 hour day
    - AI scraping bots may be to blame for this higher than normal usage
    - to counter this, there are plans to require login / implement OAuth
  - Google has the best breadth of search coverage
    - Bing might be a reasonable backup, but not as good
    - tool used to use Yahoo until they ended their free service
    - have looked into Yandex, but English coverage isn't great
- someone had the idea of adding The Wikipedia Library / EBSCO as another search backend, but discussions with EBSCO stalled
has issues with concurrent queries
uptime report: https://stats.uptimerobot.com/BN16RUOP5/784331770
false positive handling via a community-maintained exclusion list at User:EarwigBot/Copyvios/Exclusions
previous WMF contacts: Kaldari, Runab WMF, DTankersley (WMF)

used by Earwig copyvio detector to access the Google API
https://openstack-browser.toolforge.org/project/google-api-proxy
wikitech:Nova_Resource:Google-api-proxy
maintainer: MusikAnimal
purpose: this proxy uses a static IP, and there appears to be an IP whitelist on the Google API side, so I guess this proxy increases security?

https://copypatrol.wmcloud.org/en
maintainer: WMF Community Tech team (most active recent committer: MusikAnimal)
source code: https://github.com/wikimedia/CopyPatrol
last commit: 2 months ago Y
tech: Symfony (PHP)
replaced https://copypatrol.toolforge.org/en
is mostly a viewer for an SQL database that the copyright detection bot(s) below writes to
users can mark pages/revisions as being fixed or requiring no action. (However, this information is not reflected on enwiki)
there is a "compare" feature in the CopyPatrol interface. clicking on it does an API query to the Earwig tool above

This discussion has been closed. Please do not modify it.

The following discussion has been closed. Please do not modify it.

https://copypatrol.toolforge.org/en
maintainer: WMF Community Tech team (most active recent committer: MusikAnimal)
source code: https://github.com/wikimedia/CopyPatrol/tree/569f76e113da307d3810e1333531fcfc8449dbcf
last commit: 7 months ago N
tech: PHP, Twig (wikimedia-slimapp)
is mostly a viewer for an SQL database that the copyright detection bot(s) below writes to
users can mark pages/revisions as being fixed or requiring no action. (However, this information is not reflected on enwiki)
there is a "compare" feature in the CopyPatrol interface. clicking on it does an API query to the Earwig tool above

bot name: EranBot
BRFA: Wikipedia:Bots/Requests for approval/EranBot 3
maintainer: ערן
also involved: Doc James, Ocaasi
source code: https://github.com/valhallasw/plagiabot
last commit: 2 years ago N
tech: Python, Pywikibot
writes to an SQL database that CopyPatrol uses
uses PageTriage pagetriagetagcopyvio API to mark pages/revisions as probable copyright violations
- writes this to the pagetriage-copyvio log: https://en.wikipedia.org/wiki/Special:Log?type=pagetriage-copyvio
- displays the tag in Special:NewPagesFeed and the Page Curation toolbar's info flyout
- need the copyviobot permission to use this API (not assignable by admins, probably need a bureaucrat to do it)
uses Turnitin's iThenticate API
- does WMF pay for it or if it is comped???
- what are the daily limits?
- do we hit these limits?
false positive handling via a community-maintained exclusion list at meta:User:EranBot/Copyright/Blacklist