I’m disappointed, too, but I’m disappointed in how hard it’s been for my team to enforce an AI ban over the last few years.
This is particularly frustrating to read in light of how difficult it is been for me and my team to do this work. (A skill issue on our part, I suppose?)
May I suggest that you redirect your disappointment? You could join me in being disappointed that AI rules are different, and act accordingly.
I’m muting this thread. Y’all are almost certainly going to get your wish to have no process in place, enforcing the rule by vibes.
If there ends up being a major controversy, I think people will say ‘maybe we should have prepared for that’.
The organizers don’t have a great track record of implementing or communicating such procedures, as demonstrated by last year’s Online Safety Act debacle. The policy was sprung on authors at the last minute, leaving people blindsided and frantically scrambling to figure out how it applied to them. We weren’t told that it would be based on our content warnings, which caused significant consternation. Nor were we warned that we would be expected to spend the three day review period liaising with organizers about whether our games would be blocked. Had this been mentioned beforehand, people could have prepared for it instead of having to interrupt their weekend plans at short notice.
Then the review period was quietly extended to two weeks, then just as quietly changed back to three days, all without any actual public announcement. There was no e-mail or forum post, someone just snuck the changes onto the website. Needless to say, this was all very confusing, and it doesn’t inspire confidence. In light of these failings, I suggest greater transparency this time. The AI situation is much more complicated and controversial than that one. There was no implication that people whose games were considered unsuitable for the UK had done anything wrong, whereas now we’re talking about entries being reviewed for potential disqualification. So people will hopefully forgive me for not assuming that everything will run smoothly without planning or consultation.
Given that the potential consequences of violating a recent and poorly-written piece of legislation could have included international legal issues and fines/fees that could have buried the IFTF and the IF Archive, I don’t think this situation is in any way more “complicated and controversial,” actually. The stakes of a couple of entries possibly (possibly!) being disqualified from a competition with, at most, a nominal cash prize aren’t even close to being higher than that.
I think there are very legitimate criticisms of how the last IFComp played out, but they were also operating on a short deadline and in the face of potential existential threats, neither of which exist here. I don’t see any reason to expect a repeat.
Yeah, I also have some feelings on how the OSA stuff went down, but considering the rule change is being announced 7 months ahead of the deadline, not three days, I think we’re in a different world. With that said talking about the process is good, though Alex, I have to confess I’m not sure what specifically you’re thinking should happen at this point. Trying to establish a non-witch-hunty norm seems good and I think most folks are agreeing with that, for example, but is there other stuff that falls into the realm of “planning or consultation” that you think is worth considering?
Trying to establish a non-witch-hunty norm seems good
It seems like we already have that norm, right? Even though a lot of people here are very vocally against AI, I haven’t seen anyone singling out people to accuse them of being bots or using AI — which is increasingly common on other forums I read.
The lack of accusations is just on the current forum, but I assume people would act similarly during events. I guess there is a bit more of an outside audience, but not entirely.
Yeah, I think we’re doing OK on that front so it’s more about articulating it and confirming we want things to continue to be fairly chill post rule-change. Which seems worthwhile, certainly, but not too a big a deal all things considered given where we’re starting from.
In my opinion, the issue was clearly given undue consideration. Is anyone worried about an IF comp game offending China or Russia or Saudi Arabia? The UK isn’t the only authoritarian country in the world. Besides, there was a period of nearly two years to prepare for the change; there was no need to announce a policy three days before the start of the competition. As it affects this community, the present issue is much more immediate.
Personally, I’d be interested in as much detail about the procedure as possible. Here are some questions off the top of my head:
Who will make the decision? Will their names be public?
What guidance will they be given?
What training or experience do they have?
Will the decision be made by majority vote or unanimity?
Will there be any ‘test runs’ of the procedure?
Will any software be used?
Will all games be scrutinized or only those that are reported?
In the former case, will all panel members test all games? If not, how will responsibilities be divided? How many will test each one?
Will suspected authors be contacted prior to disqualification? What recourse or appeals procedure will be available to them?
Will authors’ reputations or other work be taken into account?
Will authors who have used AI before or praised or defended it have this held against them?
Will there be consideration of how the policy may disproportionately affect newcomers?
Will public sentiment be taken into account? If not, how will it be avoided?
What if someone writes a game about AI? Can they include fictional AI text that they wrote themselves?
What if someone chooses to adopt AI-like style for some other artistic reason?
If a game under suspicion is found not guilty, will there be a statement clearing it or will there be announcements only in the case of disqualification?
Can games be disqualified after the end of the competition? If so, how long after?
What if the community believes a game has been unfairly punished? Can disqualifications be reversed?
Will an author found to have used AI have all of their entries disqualified or just the offending ones? Will they be eligible to participate in future competitions?
Many have suggested that the rule will be used sparingly and employed only against egregious violations. Will there be any official statement confirming this?
Will there be an explanation of the factors considered to indicate AI use and the criteria for applying them?
I generally agree that there’s stuff to critique here, but fwiw 1) Russian/Chinese/Saudi court judgments are rarely enforceable in the U.S. whereas UK ones often are, for reasons that are somewhat complex but basically intuitive if you think about it; 2) the law was enacted two years before last year’s Comp, but the regulator didn’t put out guidance on the key issues until April and May of last year, meaning that compliance preparation wasn’t possible until after that point (and in fact there are still pieces of guidance that have yet to be finalized, and ongoing litigation and other challenges that may shift things again); and 3) my understanding is that they were planning on adopting a technological solution that would have avoided having to deal with these issues, but that fell through at the last minute, requiring the close-to-the-deadline shift.
I just mention this because I read a lot of your comment as being an argument to replace the Comp’s current standards-based approach (where guidelines are applied in a case by case fashion by an adjudicator who has a reasonable degree of discretion to get at the spirit of a rule) with a codified rules-based one (where heavily-specified rules are applied more or less mechanically by an adjudicator who doesn’t have much autonomy), at least as to this rule. Often people in high-trust societies are more comfortable with standards, but understandably tend to prefer rules as trust erodes. So to the extent that’s part of what’s going on, I thought the context might be helpful - apologies if it isn’t!
As for your longer list, it’s getting late here so I can’t respond in full tonight, but would it be helpful to lay out how things have tended to work with, say, the rule against using copyrighted cover art? I feel like that might clarify areas where it might be helpful to consider a different approach for this rule. If so, I can try to write that up tomorrow.
This is, in fact, the only space I know one can have reasonable, nuanced arguments on LLMs that aren’t finger-pointing. Not all threads go this way, but the fact that it is even possible in the first place is actually impressive.
And it’s why I don’t buy into the community descending into witch-hunting. I trust the moderators, the members here, and the board enough to keep this up.
I feel like it wouldn’t be as ‘hard’ if Hosted Games had done some basic due diligence with submissions.
When I noticed two cases of games with AI art content in HG titles which had been approved for publishing, all it took me was paint, reverse image search and free AI art checking tools to find out in a day, write a detailed report with all the evidence and mention it to HG. I felt there was no reason a professional reviewer with months in the production pipeline couldn’t do what an amateur like me did in a day.
On one occasion, a writer publicly and very intentionally passed off AI art as their own hand drawn work. I sounded off to HG and was privately told that they would address it with the writer and replace the art. I was disappointed at the response and asked that my name be removed from the tester list. Eventually, the writer did replace the art, with another AI art asset which was then approved for publication. Is that the fault of the process being ‘hard’ or a fault in the way the matter was handled?
Anyway, I’ll be watching how IFcomp handles the rules from the side.
The OSA response was an area where we talked to our lawyer and followed her advice. You can reasonably object that we should have talked to our lawyer in May or June rather than the end of July, but that’s not what happened. We coped as best we could.
The AI rule is a different situation. It’s not an emergency response; it’s a response to community discussion and the survey that went out after last year’s comp.
Are AI art checking tools reliable, though? AI text checking is very much not. At my university we aren’t supposed to use those detectors on student work because the false positive rate is unacceptably high.
In case you or anyone hasn’t seen it, there’s an extensive writeup from the other perspective here:
Honestly I empathize with both of y’all in this case. It sucks to see folks break the rules in a way that’s relatively easy for an outsider like you to check, when you know the AI generated stuff is stealing air from you or the authors you care about. And I can see how spending an extra day AInvestigating every hosted game, which is supposed to be the cheap and easy option for CoG, could destroy the economics for a small business.
This all sucks! It sucks for everyone! It’s really bad out there! I think we’re going to need to be very generous with each other for the next few years.
It strikes me that the series of events that would have to occur for there to be any meaningful trouble with Ofcom is almost infinitesimally unlikely. As someone with the misfortune of living in the UK, I’m probably at far greater risk from the British government than IFTF is ever likely to be.
That’s interesting. I wasn’t aware that an intended solution had fallen through. All I heard was that it was too short notice to have it ready in time.
It’s certainly helpful to have more information about past events. In any case, I’m not sure I understand the rationale for what you call the standards-based approach. Obviously there needs to be some amount of discretion, but surely it’s best to have as much as possible handled by explicit rules?
Yes, that would indeed be helpful, thank you. And there’s no undue hurry. As you mentioned, there’s still a number of months before the competition begins.
It would be interesting to hear the specific reasoning behind this legal advice. Is there any precedent that indicates the act is likely to be applicable to an American entity?
My point was not to relitigate last year’s OSA thread (UK Online Safety Act). I’m pointing out that it was a very different situation from this year’s IFComp, and bringing it up as a comparison is not useful.
I don’t have a lot of comp-running experience, but from moderating various forums:
It’s generally best to have explicit policies for users, but not for how those policies should be enforced. When enforcement is happening, that generally means someone is already operating in bad faith and trying to cause problems. Giving those bad-faith actors an exact playbook of how enforcement will work, just encourages them to look for loopholes. “It says here that, unless I explicitly admit that I used LLMs, they can’t do anything about it? Perfect! Then I just have to lie about that, and all their rules are toothless.”
That’s why we don’t have an explicit flowchart for how we handle spam/trolls/etc on this forum. Based on the individual circumstances, we might warn someone, mute them, suspend them, or ban them; the users trust us to be reasonable and apply our discretion. If a new user shows up and posts spam, they’ll get banned right away; if someone who’s been around for a year posts spam, we’ll probably deliberate about whether the spam is IF-related, and talk to them about it instead of immediately banning.
But if we codified this into a public policy, spammers would go “oh, we just have to let our new accounts marinate for a year before we start posting, then the mods can’t ban us!”, and that would defeat the whole purpose. I don’t think any flowchart, no matter how elaborate, can ever replace trusted humans doing what they think is best. Humans are smarter than flowcharts and will always find ways to outwit them.
So personally, I think it’s best not to bind the organizers to some explicit ironclad policy—that just encourages bad actors to search for loopholes. We’re a small community that has a pretty high level of trust in each other, on the whole. I trust the IFComp organizers to have some kind of plan for this, and for them to apply it in a reasonable way; after all, they’ve generally been reasonable before. (I can’t think of any instances where an entry has been disqualified without having a proper discussion with the author, for example, except when the author’s previously been banned for serious non-game-related offenses*.)
In particular, I think Dan Fabulich’s proposal is very reasonable, and I wouldn’t complain if the IFComp organizers implemented something along those lines. But whether they do or not, I honestly think it’s better if they don’t tell me!
* Where by “serious non-game-related offenses” I mean “harassed anyone who gave them a bad review across several different sites for multiple years until those reviewers were driven out of the community altogether”, not “submitted a game with copyrighted material in it”. It’s not common, but sadly the worst case does sometimes happen.
Nothing is perfect, but Hive has been pretty accurate in all my personal testing. One can also try to spot obvious tell tale signs, such as extra fingers and so on. There are effective tools/methods for detecting AI art, not so much for AI text.
I have already seen it. Was there something specific in that post you wanted me to discuss or respond to?
I’m sure having to pull games after they are released, issuing refunds and dealing with the reputational mess is the cheaper and easier option.
Thank you for those links. As far as I’m aware, no fine has been collected from an American company unless it already had assets in the UK.
It’s an indication of general organizational ability, which undermines the ‘just trust us’ argument.
I don’t know if we can rely on that in this case. If I may try to summarize the entire problem in a nutshell: a severe policy will result in false accusations, while a lax policy will quickly be taken advantage of.
If loopholes exist, they’ll be found sooner or later. Rules may help to reduce the number of loopholes.
They haven’t had to enforce a policy this difficult before. We have no idea what may or may not happen.
I think we were getting somewhere with people suggesting various refinements to the system.
Loopholes—as in, ways to obey the letter of a rule but subvert its spirit—only exist if the committee is bound to the exact terms of some procedure. There are no loopholes in our broad, general “we’ll ban you if you post spam” policy because there’s no exact procedure we have to follow; the community trusts us to determine what is or is not spam in a fair and reasonable way, and so if someone is trying to exploit an exact dictionary definition of “spam” to get advertisements for luxury watches onto the forum, we’re allowed to just ban them.
To continue with this metaphor, I don’t see how binding the moderators to a precise definition of “spam” benefits the forum. The only people who benefit are spammers who can now slip through on technicalities. It’s not very democratic, but there haven’t been many complaints.