Replacing IFWiki's dead links -- volunteers needed

Continuing the discussion from New "volunteers-needed" tag for IFTF opportunities:

Since the IFTF took over IFWiki in January 2022 we have made some improvements, including a Software database and Events database, but as part of the Foundation’s “volunteer opportunities” scheme we would like to ask for help in updating dead links.

About 25,000 of the 30,000 external links on IFWiki are okay, but the remaining 5,000 are not. The dead links together with their HTTP response status codes are all listed on the wiki’s Status of external links page. This list is updated periodically. Please help us update these links!

We would be grateful if you only fix a single dead link, but the more the merrier! Maybe a web page is still on the internet with a new URL, or maybe it has been archived on the Internet Archive or elsewhere. Browse the list of links to see what catches your eye.

For further details see IFWiki:Replacing dead links .

7 Likes

I will be able to help after Thanksgiving. I am on the road with a small laptop. I can’t see well enough without my large screen.

I will check back when I return home.

4 Likes

How often is the list of rotten links updated? (The explanation page just says “periodically”.)
I dealt with quite a few a couple of days ago; it would save time if they weren’t still on the list if I’m minded to have another go.

4 Likes

Sorry for the delay in replying. The word “periodically” was deliberately vague because I hadn’t (and still haven’t!) set it up to run automatically. The script takes several hours and is quite primitive in that it starts from scratch with an empty list. I started running it just now (2300 GMT) and expect it’ll be finished by the morning. Thanks for the work so far!

Edit: it took 16284 seconds (just over five hours)!

7 Likes

The wiki’s dead links index no longer shows which pages reference the dead links (the “Page Usage” column is blank), making it hard to fix any dead links.
(Perhaps this is a consequence of the recent MediaWiki upgrade, although I hadn’t checked the index for ages, so it may be unrelated.)

Separately, would it be possible for this page to somehow show the date when the dead link index was last manually updated? That would help a lot when re-checking the index after fixing a load of dead links.

2 Likes

Yes I bet it’s because of the upgrade.

I’m running the script again and it seems to be populating the page usage column all right. From memory, it’ll take several hours to get through.

It doesn’t show the last date, but I could set up a cronjob to run it regularly. Maybe fortnightly or monthly – what do you think?

Also, are there any websites that could be excluded? There’s a setting for that.

2 Likes

Done this time!

Script took 23051 seconds.

Let me know how often it would be useful to do. Maybe the 1st of each month?

1 Like

Thanks for re-running this.

Once a month sounds fine, if you can spare the cycles.

Unfortunately, there are a lot of false positives now. This is probably a result of the world having changed since the script was last run, with lots of websites pulling up the drawbridge in the face of AI slop slurpers. e.g.:

Site Spurious result
web.archive.org No Response
plover.net/~davidw/ No Response
textadventures.co.uk Forbidden
kickstarter.com Forbidden
tvtropes.org Forbidden
cpcwiki.eu Forbidden
worldofspectrum.org No Response
3 Likes