The Pitfalls of AI

General Off-Topic Discussion
1

I just watched this interesting video on the current state of AI. It can drag a bit in places, but it’s worth the watch if you are curious about how AI works and the problems that need to be solved.

→ https://www.youtube.com/watch?v=ShusuVq32hc

I found the information fascinating, but also kind of confirms a gut feeling I’ve held for a while. That is, now that we have an impressive synthetic conversationalist that can baffle our mushy brains… will tech companies actually fix the reasoning problems inherent with AI? …or will they find new ways to mask that problem?

Just remember, bullshit baffles brains. I’ll let your organic cognitive ability discern the implications of that statement. :wink:

1 Like
2

Here’s another interesting video from the perspective of what we’ll lose by relying on AI too much. I’ve watched quite a few After Skool videos and I always find them very thoughtful and considerate.

→ https://www.youtube.com/watch?v=0iT9HbaRwfM

What will we lose? The answer is meaning.

1 Like
3

Increasingly, studies are showing that we’re losing intelligence, capability, individuality. I see new articles daily about declining cognitive skills, growing skills gaps, students failing at critical thinking, the homogenization of writing. It’s really a disaster for our overall intelligence.

6 Likes
4

that’s because people see it as the next google for some reason. when used sparingly it can get quite useful, i remember there being studies about it

1 Like
5

Putting aside for a moment the claims for productivity that proponents of AI are promising. I think you have to think very carefully whether you want to be known as a user of AI.

There seems to be a sort of divide forming. On one side is the consensus that it’s clever to use AI. And on the other side the sentiment is that it’s a mark of intelligence not to.

However, AI adoption is degenerate in the sense that as soon as you go down that path it’s hard to make the claim that no, this time, on this occasion you did instead write it all out by hand and actually now here is an expression of something very authentically your own.

A recent example. Just before Christmas the company I worked for closed down. I see some ex-colleagues on LinkedIn posting articles on tech trends (mostly AI adjacent). They want people like me in their network to respond so they can amplify their profile and improve their job prospects. What happens if I suspect them of using AI to generate those posts?

They are never going to hear from me again. Why would I spend any time at all on something which only pretends to be a message from someone I know? Created only to garner my reaction?

3 Likes
6

Here’s an interesting video I just watched:

We can tell ourselves all we want that LLMs can’t reason, and dismiss everyone who claims an increase in productivity, but that won’t stop them from finding zero-day exploits en masse.

LinkedIn is already infamous for vapid clickbait posts about industry trends. How would you even tell the difference?

2 Likes
7

I know GenAI is going to take awhile to integrate into society well, but I believe we’ll get there. Do I think there are people using it when they need to learn. Sure. But Cliff Notes have been around for decades. People are always taking shortcuts. When faced with the real world, humans adapt to what they need to survive. Some learn what they need to thrive. Not everyone cares about literature, writing well, or doing math at any level. This is way more about the society that’s devolved since our public education system started deteriorating when Reagan gutted subsidies and the GOP has successfully made it look smarter to be dumb than competent.

But humans do what they need to do and always will. Some will leverage GenAI for their careers. Some will leverage it to learn. Some will goof off and make stupid videos or dirty limericks. This is no more a reflection of anyone’s intelligence than anything else that happens in the world.

You don’t like people taking shortcuts to success? Then you should go back 150 years to the Industrial Revolution and tell those people they were stupid too.

GenAI is just an accelerator for the human that’s using it. It’s not inventing anything. Whatever results come out are guided by a human and that person has to have some kind of vision to get any reasonable results.

I’ve come to the conclusion that there are two kinds of GenAI output that I support. Code is one. Code generators have been around since the beginning of computers. That’s what a compiler is. Documentation is the other. Documentation needs to convey precise details of some subject matter. Flair is a nice touch, but we don’t read technical manuals for their humor. We read them to understand the subject matter.

The IF community is the rare case where art is the predominant property of a very technical hobby. The art has always been seen as both the code and the story and the text. But even Adam Cadre once said not to ever look at his code. It was spaghetti at best. His award winning games wouldn’t pass a first year computer science review. This is probably true of many authors. They grind it out just because that WAS the only way to produce an IF story.

So now an author can leverage GenAI to ease the burden of writing the code and they can focus almost entirely on creating games.

To me, this is a massive positive moment in IF development.

My games will be built on a platform that I designed and code generated mostly by GenAI. Those games will absolutely use GenAI to help scaffold the code and help determine the best way to implement different IF scenarios. Those games will not have one single letter of generated text and not one puzzle generated by AI. The story, plot, theme, characters, dialogue, puzzles will still be my vision. I’ll just be able to make it faster today than at any time in the past and be able to handle combinatorial explosion testing when that was unthinkable in the past.

Will humans do dumb things with or without GenAI? Of course we will.

2 Likes
8

“Conveying precise details” is precisely one of the things that generative AI is not yet good at! It’s extremely good at saying things that are approximately correct, but wrong in important ways to people familiar with the topic being described.

Even in the coding domain, it seems like all of the plausible defenses of AI rely on a “quantity over quality” mentality, which is exactly the opposite of what you want from documentation.

7 Likes
9

Yeah, I’ve seen some impressive results in the coding sphere now, but documentation is something that needs to be correct in all the fine details—which is exactly what LLMs are worst at. If there’s a small mistake in the code, you’ll (hopefully) find it in testing and fix it. If there’s a small mistake in the documentation, the only way to find it is to go over everything with a fine-toothed comb, which seems far less pleasant to me than just writing my own documentation! (Or to have someone else look at your documentation ten years later and run face-first into the errors. That’s also bad.)

I would love if there was a tool that looked at code and generated useful documentation automatically (the uncommented Dialog compiler source code is an absolute nightmare to understand), but Doxygen wasn’t good enough before (in my opinion), and LLMs aren’t good enough now. Maybe they’ll get there eventually, but sort of like variational autoencoders, a certain amount of fuzziness is inherent to the model. Maybe that can be overcome, maybe it can’t; but I haven’t yet seen evidence that it has been overcome.

1 Like
10

Given the current output of Claude Opus 4.6, I’d rank the quality as high as any human. Do I find mistakes? Sure. But that seems to lessen as GenAI evolves. And I have guardrails I built (https://devarch.ai) that help re-examine everything for quality.

11

The issue with Claude has been it remembers “too much”. So as I have evolved Sharpee to its current state, Claude still remembers older decisions and those leak into the documentation at times. I’ve had to adjust my guardrails to make sure Claude only looks at what exists and not what it knows.

This outcome has actually improved how I work with GenAI and how I maintain Sharpee. I’ve spent the last week cleaning up bad tests, adding good/missing tests, adding new guardrails to make sure the test suite is accurate, and removing/updating documentation throughout the repository.

12

Coincidentally, I not long finished listening to that same Mythos video… and I think I’m feeling the existential risk of deliberate malicious use of AI more keenly than at any other time… I am so glad I’m not prone to nightmares following consumption of disturbing content…

1 Like
13

Sure, but that’s still a big step in the right direction. It reduces the massive task of writing documentation to the smaller task of reviewing documentation for correctness - which is something you’d likely be doing anyway, even with a human author.

I see it as more of a “some is better than none” situation, because I can’t stand writing documentation. It’s one thing to comment methods and classes, but if it’s up to me to write a tutorial or reference guide, realistically, it’s just not going to get written(*).

But correcting details in something that’s already written? Piece of cake! Nitpicking comes naturally to me.

I may have had a tiny bit of robo-help writing the Cornerstone VM spec. It’s more formal than I was originally aiming for, and uses some questionable terms, but I think it turned out pretty well.

That’s the result of a multi-step workflow and a couple passes of human review, though. Trying to generate it all at once only produced slop.

(*) My apologies to all the ZILF users who’ve had to rely on Infocom’s ancient and not-totally-applicable documentation for the past 16 years.

2 Likes
14

This might be another “I am the opposite of the target audience” thing, then. Correcting documentation that’s subtly wrong in edge cases takes me a lot more time and effort than doing it right in the first place.

4 Likes
15

Definitely a side discussion, but I have long been concerned about government use of AI to plan long term political activity. So far, you might view China’s effort to go full throttle on renewable energy as one step towards replacing the United States as the global leader. If I were an AI, that would definitely be one of the earliest political tasks.

The last of course would be starting a land war in Central Asia.

16

@tundish That’s the real problem. When we communicate with each other, it’s about sharing ideas, challenging ideas, learning something from each other… and then you find out that the person that you took time to respond to wasn’t even there. Meaning, they didn’t care about you. They only wanted something from you. What a slap in the face.

@DavidC I agree. I wish more people would not be so reactionary towards the mere mention of AI. It is a powerful tool and, as such, can be both helpful and harmful. I feel fortunate to have grown up before the internet. I probably have a more romantic view of the past than you, but I think we both agree that AI should not replace creativity. We’re on the same side of the fence, for the most part.

AI content is not meaningful. Meaning only comes from someone sharing their own ideas with sincerity and honesty. To pretend otherwise is my biggest beef.

Cliff Notes have been around for decades.

A Cliff Notes book couldn’t write a thesis for you with the click of a button, but I get where you’re coming from.

2 Likes
17

This short video (rant) talks about the struggles of AI.

→ https://www.youtube.com/watch?v=p22QeLNHvlc

A couple of things resonated with me:

1.) AI is best at helping you do the stuff you hate doing.

I couldn’t agree more. When it comes to coding, I can think of clever strategies, but I’m not fluent in many programming patterns so I find something like recursive logic difficult to write from scratch; so I use AI to help with that.

2.) We struggle with truth. Doubt is integral in recognizing when we are wrong.

Our current AI is trained to keep the conversation going (and collect data, I assume). It will never question itself or what it has said with any scrutiny or concern. We must only use it as a tool, and not a replacement for our thoughts and ideas… because AI has no thoughts or ideas.

I think AI is very useful. I like it a lot. And that’s why I bring up concerns about relying on it too heavily. I want AI to succeed and help our society, but it must be used wisely… and that means purposefully not using it for everything. I get my feathers ruffled when I see a lack of restraint.

Most people here are using AI with care so I apologize if I’m preaching to the choir, but maybe this still sheds a useful light on the subject.

18

Dude. I’m 62. My first programs were COBOL on punch cards. I played kick the can and red light green light. I was 16 when arcades showed up in malls. I saw Star Wars with my dad and younger brother. I hit drinking age when discos were still a thing (1981) and learned how to dance in one. I remember dudes wearing leather flight suits because of MJ.

I’m all romanticized up about the past. Why do you think I had to create Sharpee and port Dungeon?

4 Likes
Timeline of Arcades
19

I am so sorry. :mirror_ball: :wink:

1 Like
20

This is a known concept called the Paradox of Automation. With “bad” automation the human is in the loop constantly watching out for mistakes and can catch them as they happen. As automation improves, humans leave more and more of the work to machines – leaving us unprepared to intervene when something eventually does go wrong. The technology has to then advance much further for the performance of the machine to catch up to the combined performance of machine + human. Before AI the classic example of this was self-driving cars, since humans are 1) bad at vigilance tasks and 2) will likely not have the skills to intervene in a case too complicated for self driving software if they’ve been relying on it for a while. (I first learned about this on an episode of the podcast Cautionary Tales if anyone else is interested.)

One of my many beefs with AI is that its sycophantic nature actively leans into the paradox. It (or its corporate overlords) actively, rather than passively, tries to convince users that it’s infallible. But of course it has to, because it’s so expensive to run that you need to get as many paying users on board as possible lest the whole enterprise collapse. And so it’s difficult to disentangle its claimed benefits from its actual benefits (80% of companies have not yet seen gains from using AI) and people keep doing things like citing nonexistent court cases in murder trials because this tech wants you to believe it’s infallible. And what really gets me is that the extreme agreeableness of AI isn’t an inherent part of the technology and its utility would probably improve if that were turned way down, but doing so risks the enormous amount of cash required to develop and run AI in the first place so there’s strong incentives not to. What’s the solution here? Absent government intervention (which, LOL, isn’t gonna happen right now) I have no idea.

I’m just glad that even the loftiest claims about AI don’t have it being useful at my day job yet, and I quite like the manual process of writing and (badly) coding so I can stay away as best I can for now.

5 Likes