I think it is worth mentioning that TALJ 2026 was a very unproblematic case with everyone getting between 11 and 16 ratings. ParserComp 2023 was a lot worse where entrants got somewhere between 5 and 18 ratings. The penalty for 5 ratings is significant. Being an Adrift 5 user I am a bit concerned for my fellow drifters 
I guess this will improve a lot when Parchment sometime in the future manages to include the Frankendrift interpreter 
I feel your pain. Adrift is a great authoring tool that allows you to create games every bit as good as those created by Inform or TADS, yet it is largely ignored by the wider community.
I don’t want to spam this thread too much, but I had some time while watching football (soccer), and so I made a spreadsheet with the game data for the jams listed in my last post, in order to see the effects of itch’s score adjustment at a glance.
This time, I included the freestyle category for ParserComp 2023, because as far as I could see while reconstructing the raw score (which doesn’t seem to be listed on the page, unlike other itch comps), the published scores can only be what they are if both categories (classic and freestyle) were considered together by itch in order to determine the median number of votes and adjust the scores using that criterion. (Which seems to be a methodical problem, if I’m not mistaken, because it means that games from both categories have an influence on the median and therefore on whether games get their scores adjusted negatively, even if those are from a different category. But that’s not relevant for this thread.)
Across the eleven jams, 132 games were entered, of which 43 suffered from the score adjustment.
(Why not half of them? Because in some jams, due to the relatively low number of votes, many games got exactly the median number, instead of less.)
A score adjustment does not necessarily result in a ranking difference, of course. There were 41 games whose ranking was affected by the adjustment.
Of these 41, 17 were negatively affected (i.e., ranked lower due to itch’s adjustment than they would have if their raw scores had been used), and 24 were positively affected (i.e., benefitted from the relegation of the other games).
I guess it’s up to individual judgment if those numbers speak for or against itch’s adjustment criterion, or are simply neutral.
The top three places (the podium) would remain unchanged in most of the jams, if we took the raw score instead of the itch method.
This could reassure us that the itch method has not been very disruptive, after all.
Only in two of the eleven jams would there be changes on the podium: in the Adventuron CaveJam, place 7 (“The Cave of Hoarding”) would soar to 3rd place if its raw score of 3.5 was used, displacing “The badly cursed treasure” which would move down to place 4. Given that “The Cave of Hoarding” had 6 ratings and therefore suffered badly from the adjustment, and that “The badly cursed treasure” had 8 ratings (so merely two more) and did not suffer, because it reached the median of 8, we might also conclude that those two ratings shouldn’t make that much of a difference, and that therefore itch’s method is quite disruptive.
The other podium-affected jam is the Adventuron Christmas Jam, where “Day of the Sleigh” would jump from 5th to 2nd place if its raw score of 4.077 was used. The current 2nd place, “Santapunk 2076”, has a score of 4.067. “Day of the Sleigh” suffered from the itch adjustment because it received 13 votes, whereas “Santapunk 2076” did not get adjusted, because it had two votes more (15), which just about allowed it to cross the median value of 14.5.
One could again argue that this result is a bit unfair, and that the added epistemic/statistical security gained by 15 versus 13 votes is not big enough to warrant such a score penalty.
On the other hand, as I said above, it’s quite likely that any system will have some counterintuitive consequences.
(To be clear, I offer no opinion on the relative merits of these particular games at all; I’m just talking about the rating system here.)
Anyway, maybe the rating queue can already address the problem to our satisfaction next year.
The spreadsheet is here: Jam Ratings - Games List.zip (36.2 KB) (unpack it to get the file in LibreOffice Calc .ods format)
Here are the two relevant excerpts from the spreadsheet as screenshots for a quick glance:
If you notice any errors, please tell me.
