Wikipedia talk:reFill/technical

Please use this talk page for technical discussions regarding reFill. Curb Safe Charmer (talk) 22:46, 7 February 2022 (UTC)

Upgrades

When I was still using ReFill I made some improvements to the source code, particularly regarding website title retrevial. I think I added a better way to get the website titles, and also a title blacklist (so known bad titles ). I can upload the git patch diffs on some pastebin site or on here. It could be useful to you. Rlink2 (talk) 00:38, 8 February 2022 (UTC)

@Rlink2. I think the repo is located at https://github.com/CurbSafeCharmer/refill if you want to look into submitting PRs. –Novem Linguae (talk) 02:00, 25 May 2022 (UTC)

CI/CD

@TheresNoTime and SD0001: it would be great if we could get a CI/CD pipeline working on reFill. Anyone know how it is meant to work? Curb Safe Charmer (talk) 12:31, 6 June 2023 (UTC)

GitHub Actions can be used – it would require any one maintainer's SSH key to be stored in GitHub Secrets, used to login to toolforge and run the deploy commands every time a commit is pushed to master branch. c.f. https://github.com/siddharthvp/SDZeroBot/blob/master/.github/workflows/toolforge-deploy.yml – SD0001 (talk) 14:54, 6 June 2023 (UTC)

@SD0001: how does this relate to the publishing of docker images? Curb Safe Charmer (talk) 10:30, 7 June 2023 (UTC)

@Curb Safe Charmer The docker images are just for the local dev setup, right? Toolforge deployments run on the standard python/node16 images only and the cluster doesn't allow custom-built images. CI/CD = CI (automated tests) + CD (deployment). I don't think there's much CI to be done as we haven't written any tests. CD is something that we can setup using a GitHub action. – SD0001 (talk) 11:16, 7 June 2023 (UTC)

Ah, ok, thanks for clarifying re Docker. I think there is some CI as tox is in use. I was expecting to see some remnants of automated deployment, perhaps disabled or broken, but in the absence of such, would you be willing to set up the action? Curb Safe Charmer (talk) 11:32, 7 June 2023 (UTC)

Yes. I still need to figure out some details of the setup though. I see the refill tool is running a php webservice. But the git repo doesn't contain any php code? – SD0001 (talk) 12:14, 7 June 2023 (UTC)

There's this separate repo. I think TheresNoTime found some other moving parts that weren't under source control recently, will see if I can find that too. Curb Safe Charmer (talk) 15:14, 7 June 2023 (UTC)

Worth noting that the labsconf repo has CD – it's configured with a webhook (I'm guessing - I don't have owner access to the repo to actually check). The CD script (~/public_html/hooks/commit.php) doesn't appear to be under version control - one of the reasons why I prefer GitHub actions instead. – SD0001 (talk) 17:07, 9 June 2023 (UTC)

Branches

We might also need to rebuild the venv. The venv is currently running on python 3.7. – SD0001 (talk) 15:01, 6 June 2023 (UTC)

@Curb Safe Charmer: Just FYI, refill on toolforge uses the labs-stable branch of https://github.com/CurbSafeCharmer/refill — any merges to master won't be reflected on https://refill.toolforge.org I'm sure you know that already, but just in case... — TheresNoTime (talk • they/them) 12:20, 9 June 2023 (UTC)

@TheresNoTime: no, I was unaware of that. Sadly I had a total of zero minutes of handover of the project from Zhaofenli, and anything I know about the project has had to be worked out... Curb Safe Charmer (talk) 14:16, 9 June 2023 (UTC)

@TheresNoTime At least I didn't know that. Seeing as this branch is 624 commits ahead of master, is there a reason not to mark it as the default branch? Otherwise anyone looking at GitHub is looking at the wrong code. And yep, this branch does have the PHP code which I was looking for. – SD0001 (talk) 12:32, 9 June 2023 (UTC)

@SD0001: There are probably reasons, but I'm afraid I don't know them — for the most part, the master branch was once going to be a rewrite but was never completed. — TheresNoTime (talk • they/them) 12:36, 9 June 2023 (UTC)

@SD0001, I'm shocked myself :D That's one of some 30 branches there. We should add this note to the main page here. I've wasted quite some time exploring the wrong code :( Ponor (talk) 12:38, 9 June 2023 (UTC)

It kind of looks to me like the /ng the master branch and the / version the labs-stable branch? AManWithNoPlan (talk) 13:54, 9 June 2023 (UTC)

It is all pretty confusing. At Wikipedia:Refill/technical#Hosting I started listing what I am seeing in the production environment with a view to working out what is live and what is not, working out what can be deleted, and working out what folders on toolforge map to which bits in the repo. Curb Safe Charmer (talk) 14:41, 9 June 2023 (UTC)

Tracked in Phabricator
Task T340502

Tracked in Phabricator
Task T340503

Tracked in Phabricator
Task T340504

This tool appears to have multiple issue trackers (GitHub issues and Phabricator), multiple repos, and multiple branches. Getting everything into 1 ticket tracker, 1 repo, and 1 branch should probably be a high priority goal. –Novem Linguae (talk) 16:15, 9 June 2023 (UTC)

@Novem Linguae: agreed. What's best practice for issue tracking for tools? Curb Safe Charmer (talk) 16:29, 9 June 2023 (UTC)

Could go with either issue tracker. Maybe look at which one has more open tickets and go with that one? Less bugs to migrate over. If we go with Phab, I'd recommend disabling the GitHub Issues tab, which can be done in the repo settings. If we go with GitHub issues, might need to find a Phab admin to disable the tag. –Novem Linguae (talk) 16:34, 9 June 2023 (UTC)

My 2¢ is keeping it on phab, as it'll save people having to create a GitHub account to report bugs. Pretty sure all the GitHub issues can be closed. — TheresNoTime (talk • they/them) 16:36, 9 June 2023 (UTC)

Ok, Phabricator it is. I prefer a Kanban board anyway - more visual. I will trawl through the GitHub issues and close or migrate them, and disable the tab. Will also in due course do another trawl through the Wikipedia talk:reFill to try to make sure we've captured everything that users are telling us about - there are lots of duplicate reports, and plenty in the "won't fix" category unless anyone fancies writing a load of Zotero translators. Curb Safe Charmer (talk) 16:44, 9 June 2023 (UTC)

Alright, I made a bunch of phab tickets for these items today, so we can start planning out details. phab:T340502, phab:T340503, phab:T340504. Anyone up for moving the GitHub issues tickets over to phab? If not I might boldly do it soon so we can turn off GitHub issues and centralize around phab. –Novem Linguae (talk) 01:31, 27 June 2023 (UTC)

Nov 2023 update

• I just submitted 3 PR's that will combine github.com/CurbSafeCharmer/refill#master, github.com/CurbSafeCharmer/refill#labs-stable, and github.com/refill-ng/refill-labsconf, into subfolders of github.com/CurbSafeCharmer/refill#master called undeployed-rewrite, deployed, and labs-config. Please review these PRs at: 1, 2, 3. That should close these two tickets: phab:T340503 and phab:T340504. Thanks. –Novem Linguae (talk) 09:02, 30 November 2023 (UTC)

  I think that https://refill.toolforge.org/ng is the master branch, so it is deployed after all. With that in mind, I picked these 3 folder names: version1, version2ng, and tools-config.

  The PRs numbered 2 and 3 above should be safe to merge, even though CI is failing in #2. The failure is a warning about the current version of Twig having a security vulnerability. We can address that in a follow up patch.

  The PR numbered 1 above has something broken in .github/workflows/python.yml. Assistance with diagnosing that would be appreciated. –Novem Linguae (talk) 12:58, 30 November 2023 (UTC)

• As discussed above, tonight I migrated all the https://github.com/CurbSafeCharmer/refill/issues to Phabricator. Should be everything. Sorry for all the email spam. I'd like to request that we disable and hide the Issues tab on the https://github.com/CurbSafeCharmer/refill/issues repo, so that folks use Phabricator. This will close ticket phab:T340502. Think we can do that? –Novem Linguae (talk) 11:46, 30 November 2023 (UTC)

   Done Curb Safe Charmer (talk) 11:52, 30 November 2023 (UTC)

/ng, language support

@TheresNoTime et al., I've been asking but got no answer: what is https://refill.toolforge.org/ng/ and how's it different from https://refill.toolforge.org? Why does the latter not support all languages? I've tried to find the answers in the code, but those .vue files... I really don't know what to do with them. Ponor (talk) 12:26, 9 June 2023 (UTC)

@Ponor: From memory (and I could be misremembering), /ng is just a prettier/more basic frontend tailored towards the English Wikipedia (at the very least, jobs submitted from both https://refill.toolforge.org and https://refill.toolforge.org/ng both end up in the same queue, and are both processed in the same way) — TheresNoTime (talk • they/them) 12:35, 9 June 2023 (UTC)

I wouldn't mind using https://refill.toolforge.org, but it does not let me use the language(s) I need – and I don't see why it wouldn't because they all work at /ng.

This is a great tool and I'd like to advertise it to some other communities, but I need to make sure it's working properly. Ponor (talk) 12:41, 9 June 2023 (UTC)

Which languages would you need? I can have a look at how that works (it may be as simple as adding the option..) — TheresNoTime (talk • they/them) 14:23, 9 June 2023 (UTC)

@TheresNoTime, well thank you very much! I've tracked the options down to wikioptions.html but I have no idea where the "wikis" come from (and would like to learn). I searched for "commons", "war" and some others, but github found nothing. To answer your question: adding hrwiki would be great (it works fine), but why not make it available to all? Ponor (talk) 15:48, 9 June 2023 (UTC)

Ah, yes, configs are stored in refill-ng/refill-labsconf (the config was originally in an inaccessible repo, hence it got moved to an otherwise-unused GitHub org..) — the wiki array is defined in common.php there

As for why not make it available to all, I honestly don't know — there's probably no technical reason why we couldn't.. — TheresNoTime (talk • they/them) 16:01, 9 June 2023 (UTC)

@Ponor: [EDIT CONFLICT] They are in this separate repo which TheresNoTime created recently because previously those files were not under source control. However, I do not believe that adding language support is just as simple as adding one to the list, as otherwise as you say we could just add every language version of Wikipedia. The templates that reFill emits i.e. the CS1 format must also exist on that language Wikipedia. Curb Safe Charmer (talk) 16:06, 9 June 2023 (UTC)

Ah! Forgot about the templates! — TheresNoTime (talk • they/them) 16:09, 9 June 2023 (UTC)

Oh yes, of course. But many do have them d:Q15403807. Ponor (talk) 16:22, 9 June 2023 (UTC)

Editors are invited to test reFill 2, a new version of reFill under development. It offers a redesigned interface with real-time progress reporting while the tool is running, and resolves many issues affecting the current version of the tool. Userscript and bot developers may also take advantage of the APIs exposed by reFill 2 to complete bare references programmatically.
— User:Zhaofeng Li
— Wikipedia:ReFill 22:50, 1 July 2019 (UTC+1)

Starting from June 30, 2019, all requests to fix English Wikipedia articles will be redirected to reFill 2. Both versions will remain available for Wikipedias in other languages.
— User:Zhaofeng Li
— Wikipedia talk:ReFill 19:47, 30 June 2019 (UTC+1)

@Nomopbs: It's part of the migration to the new tool, and the redirect isn't working for raw wikitext input. While I fix the issue, you can use the new interface directly at https://tools.wmflabs.org/refill/ng. The new tool will hopefully completely replace the old one when support for other Wikipedias is improved.
— User:Zhaofeng Li
— Wikipedia talk:ReFill 23:53, 13 January 2019 (UTC+1)

Oh wow. So /ng is the newer/better version. So maybe that's the way for everyone to go? I like how it works! Ponor (talk) 15:52, 9 June 2023 (UTC)

@Ponor: only if you are navigating to https://refill.toolforge.org/ng/ to call the API through that front end. Most people run reFill through the reflinks gadget which adds a reFill link to the tools sidebar. They just navigate to the page they want to expand citations on and click that link. They don't see the old or new web form. Curb Safe Charmer (talk) 16:17, 9 June 2023 (UTC)

@Curb Safe Charmer: I'm having a hard time convincing my patrollers and users to install any scripts. IDK why. So I'm thinking a nice web interface is much better when you want to show them that things work, they only need to follow a link. I thought /ng was some old version (it's well hidden), but it turned out it's the opposite. Ponor (talk) 16:46, 9 June 2023 (UTC)

Understood. I've always assumed that the developer intended ng to mean next generation. Curb Safe Charmer (talk) 16:55, 9 June 2023 (UTC)

I was going to guess "new GUI", but I like your guess better :) –Novem Linguae (talk) 17:02, 9 June 2023 (UTC)

Who is GlazerMann?

Does anyone know who is contributing to the repo as GlazerMann? They seem committed and know what they're doing, but I have no idea who they are. I'd like to ask if they'd like to become a maintainer. Curb Safe Charmer (talk) 14:31, 9 June 2023 (UTC)

Have you tried asking them on one of those PRs/issues? (If they don't want to disclose their wiki username, I see no problem with that.) – SD0001 (talk) 17:12, 9 June 2023 (UTC)

It's User:AManWithNoPlan * Pppery * _{it has begun...} 04:12, 27 June 2023 (UTC)

Phab watchers

@SD0001. Feel free to become a watcher/member of the Tool-refill tag on Phab if you're interested. –Novem Linguae (talk) 01:11, 27 June 2023 (UTC)

Any updates on pull requests and issues on github?

AManWithNoPlan (talk) 22:30, 19 September 2023 (UTC)

Merging this page with the main talk page?

I propose cutting and pasting everything on this talk page to the main talk page over at Wikipedia talk:reFill. There are no archives on this talk page to move over. We can slow down archiving on the main page by doubling min-threads. I believe it is important to consolidate everything so that things do not get too spread out. Any objections? Thanks. –Novem Linguae (talk) 09:04, 30 November 2023 (UTC)

The reason I created /technical was to document the technical architecture and centralise information about the codebase. The associated talk page became a place to talk about questions and discussion about the architecture and codebase. It makes that discussion much easier to find than if it was interspersed with the rest of the content at Wikipedia talk:ReFill which tends to be where non-technical users of reFill go to report operational problems (reFill has stopped working) or how it has mishandled a particular bare URL. Having split it for what I think are good reasons, I am not at all keen on merging it back. I don't think splitting technical and non-technical discussion means things get spread out - no more than having WP:VP/T separate from Wikipedia:Village pump (all). Curb Safe Charmer (talk) 09:29, 30 November 2023 (UTC)

Merging githubs and merging git branches

Moved from Wikipedia talk:ReFill

 – Keeping technical discussions in one place Curb Safe Charmer (talk) 14:06, 6 June 2024 (UTC)

Did we finish merging githubs and merging git branches? I think it might be a bit hard to deploy and do open source collaboration until those two housekeeping tasks are done. I have PRs open for those tasks if someone wants to review them: 261, 262, 263. –Novem Linguae (talk) 08:12, 5 June 2024 (UTC)

@Novem Linguae: That is still outstanding. I'm not comfortable doing it - needs someone who knows what they're doing! @TheresNoTime:? Curb Safe Charmer (talk) 08:22, 5 June 2024 (UTC)

I have some patches that, once merged, will put everything in one branch and one repo. There will be a couple folders in the root directory of the repo. One will be the old version, one will be the rewritten version (NG), and one will be some config stuff. Both versions appear to be deployed (the old version at https://refill.toolforge.org/ and the new version at https://refill.toolforge.org/ng). Once that's done, it will be much easier to reason about this software. I don't think we're too concerned about the user scripts. I think we're mainly concerned with the web tool itself (front end, back end, API). Sounds like the stack is PHP for the old version, and JavaScript Vue & Python for the new version.

Some open questions...

• When were the last deploys?
• How out of sync are they with the GitHub repos? In what direction are they out of sync? (e.g. are there changes that never made it into git deployed on Toolforge?)
• Having two ReFills doesn't make sense in the long run. Double the maintenance, double the patches. Which version is in better shape? We should gather info like this in preparation for picking one version and removing the other.
• Are any of us 4 familiar with Python? That is not a normal MediaWiki programming language. I am not familiar with it.

–Novem Linguae (talk) 04:18, 6 June 2024 (UTC)

@Novem Linguae: Would you mind if I moved the technical thread to Wikipedia talk:ReFill/technical? I know you'd suggested merging that talk page with this one, but the reason for keeping them separate is to avoid having significant technical discussions interspersed with more general talk (such as this thread which started off as being about a dead parameter). Would you also be willing to be added as a maintainer so that you can log in to toolforge and see for yourself what's deployed? Curb Safe Charmer (talk) 12:16, 6 June 2024 (UTC)

Sure, both are fine by me. My Toolforge account is novemlinguae. Thanks for offering :) –Novem Linguae (talk) 12:46, 6 June 2024 (UTC)

Thanks for the access. I took a look at Toolforge tonight. Here's my notes:

refill.toolforge.org
  /config - this is probably the contents of https://github.com/refill-ng/refill-labsconf as of 2022-10-10
  /Git - a small number of PHP files and folders from 2015
  /logs - 1 error log file from 2023. suggest deleting
  /old-ng - empty. suggest deleting
  /refill-ng-dist - empty. suggest deleting
  /versions
    /citoid - looks like https://github.com/CurbSafeCharmer/refill -> master branch but with more files than in the repo, deployed around 2020-01-20
    /stable - this is probably the contents of https://github.com/CurbSafeCharmer/refill -> labs-rewrite branch as of 2023-01-13
      /web - the web entry point. public_html redirects here.
        /ng - this is probably the contents of https://github.com/CurbSafeCharmer/refill -> master branch as of 2020-07-08, after undergoing a webpack build step (barely any files here)
    /test - looks like https://github.com/CurbSafeCharmer/refill -> master branch but with more files than in the repo, deployed around 2021-03-30
refill-api.toolforge.org - no access yet
refill-api-test.toolforge.org - no access yet

There's redirects in various spots, such as public_html and citoid folders, which I find a bit complicated. The nesting of /ng/ inside of the old UI also complicates things. Overall, I find this directory structure complicated to reason about. –Novem Linguae (talk) 08:55, 10 June 2024 (UTC)

@Novem Linguae: Thanks or this. I agree it is confusing. Some time back I dumped the structure of the front end and API to Wikipedia:ReFill/technical#Hosting, which you might want to use as a comparison - feel free to update/expand there. My next step would have been to download it all and then use Beyond Compare to compare each file with those in GitHub. Curb Safe Charmer (talk) 10:00, 10 June 2024 (UTC)

Whenever you have this problem, I suggest just creating a separate tool account and practicing the deploy against that account. Something like 'novemlinguae-test' (if you use refill-test, it might eternally show up in search results, which can be annoying) —TheDJ (talk • contribs) 11:12, 10 June 2024 (UTC)

Offer to help

@Curb Safe Charmer and Novem Linguae: I am willing to volunteer to help with support for this, but unfortunately I need to learn more about how to do that. I created a Toolsforge account years ago with the intention of learning how to maintain or develop tools, but I never got far in the process. Even looking those linked tasks got me confused about the branches; I'm unclear which one(s) are being hosted or used for the gadgets. In my Tools menu, I have "reFill 2 New!" and "reFill" which appear to have different URLS, but they both take me to the same "ng" URL when I use them. I also have "(o)", which opens an interface similar to the old options, and which also runs in the "ng" URL. I think I would need a tutorial or diagram to explain the branches, application structure, and support model. —Ost (talk) 19:37, 5 June 2024 (UTC)

@Ost316: thanks for being willing to help. Unfortunately, the original developer Zhaofeng Li developed the code by himself, didn't produce much documentation, and became inactive on Wikipedia before handing over the code to anyone. I stepped forward in the absence of any maintainers, but didn't get any handover and have found it difficult to get a handle on what is what. I've been loathe to change things without knowing the impact. It will need someone with more confidence in Git branching and merging to make the necessary changes and sync with the production environment. Once we are in a better place we can produce new documentation and widen the pool of maintainers. Curb Safe Charmer (talk) 14:11, 6 June 2024 (UTC)

List of maintainers

@Cyberpower678 and SD0001: while adding Novem Linguae to the list of maintainers of reFill, which includes both of you, I thought I'd drop you a note and check whether you are still willing to be involved? Curb Safe Charmer (talk) 14:16, 6 June 2024 (UTC)

I've had plans to merge reFill and IABot at some point, but I don't know when those plans will see fruition. —CYBERPOWER (Chat) 17:05, 7 June 2024 (UTC)