I think “basically impossible” overstates the case. I say that because many of the posts in my AI and Testing series are designed to show testers (acting as AI evaluators) how to approach AI systems experimentally, rather than treating them as inscrutable black boxes. Depending on the system, that can involve different levels of observability and controllability, including access to internal model parameters. That’s not me saying you’re wrong; that’s just me saying there’s nuance.
At the lowest level, a model’s weights are observable. A trained model is ultimately a very large collection of numerical parameters. Those parameters can be inspected, modified, saved, compared, and restored. There is nothing hidden in the sense of being fundamentally inaccessible. Obviously, this isn’t available to everyone unless you’re working with an open-weight model, but closed models are still observable by the organizations that build and operate them.
The next question is whether changing a weight has observable consequences. Again, yes. You can change one weight (or a million weights) and run the model before and after on the same prompts. The outputs may change dramatically, subtly, or not at all. This is exactly the sort of perturbation experiment scientists perform in many fields: change one thing, observe the effects. This is actually how counterfactual reasoning is being tested in AI models. For example, a model can produce a counterfactual statement without necessarily having a mechanism that supports counterfactual reasoning. Testing that distinction requires intervention. You don’t just ask the model, “What would happen if things were different?” You very specifically and deliberately change the conditions and observe whether its behavior tracks the change.
The hard part is the inverse problem: given a behavior you care about, which weights should you change to reliably affect that behavior without breaking others? That’s much more difficult because representations are distributed. It’s like asking which molecules in an engine are responsible for fuel efficiency. Every molecule participates, but the meaningful explanation lives at a higher level of organization.
So, if I had a point here, I would say it’s that we have to be careful and avoid conflating “hard to interpret” with “impossible to investigate.” In fact, this connects nicely to that emphasis on causality I brought up upstream. One of the strongest ways to establish a causal relationship is intervention. If changing X consistently changes Y, you’ve learned something causal about the system. That’s very different from merely observing that X and Y tend to occur together.
If the information on the Ehang and the Evtol Travel sites can be believed, then that company has passed the requirements to get the relevant certifications & licenses to run pilot-less air-taxis in the Guangzhou and Hefei economic/cultural hubs. And if the EHang’s Pilotless Air Taxi Takes Flight In The Swiss Alps article can be believed, Ehung may soon be doing something similar in the Swiss Alps soon.
And as shown in this eVTOL Air Taxis: What You Need to Know Before 2026 Ends, Western countries like the US aren’t that far behind in getting the relevant certifications, as the US FAA granted Joby Aviation a Stage 4 clearance back in March 2026.
However, if the humans decide to connect that same “AI” to a CCTV system (which they have), and some of those CCTV cameras are in environments that have a large body of water in them (which they are), and if ducks visit and/or gather in at those bodies of water (which they do), then the “AI” can get it’s own images of ducks.
Thus in that specific use-case, is a human really needed to help the “AI” to improve its understanding of what a duck looks like?
I think you answered the question here with your own example. The human installed the camera edit - and connected it - in your example so they were the agent that created new information for the model, training, and all that stuff.
Right up until AI can control versatile robots to act as their eyes and ears. Given the advances happening in robotics, that doesn’t seem far off. Still, I think we’re a long way off from AI being self-sufficient to the point of being able to coordinate a team of robots to mine materials and use them to construct a factory to make more robots, etc. But complete self-sufficiency isn’t a requirement to be able to self-improve.
I think for me the disconnect is that technical agency is being conflated with philosophical agency. AI never wakes and decides it wants to take new portraits of ducks. AI has the ability to say that a duck is not a swan. One is philosophical agency and one is technical agency. There is no reward system in technical agency, it is a closed system.
Debatable but defensible. AI is already advancing with AI-generated synthetic training data. This is here today and shall continue (one citation here). But your statement is “AI to improve.” This is where the debate shall live, and probably for some time.
You bring up a really good point. Eventually, we’ll build tools that the AI will use to build tools of it’s own design. 3D printers would allow that to happen with ease, with pre-assembled moving parts, working circuitry and all. One day, the AI will explore the analog world and beyond (if we let it).
I hope after it catalogues what the inside of a duck looks like that it doesn’t get too fascinated with us.
Shit. Has it already happened?! Are we living in the Matrix?! There is a spoon… there is a spoon… there is a spoon… *assumes fetal position in the corner of the room*
I think this is a key point. The AI will only do what it thinks the user wants. If the second user is also an AI, undesirable behavior results.
Honestly, I think this technology should have been named something else: certainly it is artificial, but intelligence is something else. A book can contain information, but it is not intelligent. A machine can manipulate data, but it cannot think.
For a long time I never believed these LLMs had any actual intelligence. The phrase “stochastic parrot” was said a lot, and I generally agreed that that is all they were – mimic machines.
I’ve changed my mind recently, the latest models do seem to have some kind of intelligence. Plus I don’t see the term “stochastic parrot” said much anymore, that feels like a concept which is out-of-date now.
Please be aware we can discuss development and usage of AI on this forum in categories specifically devoted to this discussion with the ai tag.
We do ask that you please refrain from just posting examples like “Here’s what [AI] thinks of this…” with just a a link to the output or a quote.
If you’re quoting AI/LLM output, it should be constructive to the discussion and not the sole content of the message.
COC:
AI or LLM generated content is not considered “Your Own Stuff” and we recommend you cite the source and clearly demarcate generated prose when discussing it. It’s okay to use AI to assist your writing, but please don’t use an AI to generate your entire message.
I have to be honest, I use GrammarlyPro when I write to assist me. The software AI will analyze where my writing gets weak based on the tone and genre of what I’m writing. It will change and suggest new wording, phrases, and descriptions. I also pass my output to an LLM as a second pass to check if the AI thinks there is anything lacking. Of course, ultimately the final prose choice is all mine, but honestly I have no idea at what point that might cross the no-AI content rules. I’m sure I can’t be the only one who feels that way.
Using AI to generate your message content or feeding a message to AI to create a reply to someone that you paste verbatim Pretending you wrote AI content Posting AI content as if it’s a Forum Member deceptively in an attempt to “pass the Turing Test” Posting illustrative examples of AI/LLM content as relevant to the discussion on working with AI, citing the source and demarcating the AI content with quotes or a “hide details” fold. Using a grammar checker with AI components to proofread your message before posting. Using AI to review/spot-check your post before posting in your own words
[!tip] Keep in mind
AI/LLM prose has a very identifiable tone that most people can pick up on. If all your posts start sounding like they are AI generated, you run the risk people zoning out and ignoring your content or having it flagged and moderated as “not your own stuff”.
If you are creating a post that is one line of your comment and then pasting pages of AI content after it, you may want to consider narrowing your quote to relevant material, or folding it behind a “details” tab if you need to display an extensive output that is opt-in for readers.
But is it a marketing stunt? Because AI has been a catch all for just about anything that gives a machine any semblance of thinking/making decisions going back at least as far as the Ghosts in the original Arcade version of Pac-Man or the first time someone coded a Tic-Tac-Toe game that could always force a tie in a player-v-computer game, if not to the very dawn of electronic computing or even computer science itself. The people pumping up the LLM bubble didn’t invent the term, they inherited it from generations of the term being in common usage, and I doubt they could stop people using it for LLMs if they tried.
Now, you’ll get no argument from me that it’s not mostly effective marketing that AI companies are presenting Large Language Models as if they are on par with AI from utopian science fiction and marketing largely to the unskilled who can’t tell the difference between AI generated art/music/writing/code and that produced by competent human creators and banking on those people having not read any of the dystopian or [insert term here for tone that is neither utopian nor dystopian]* that warn about the potential misuse and abuse of AI and being able to convince them their paid models are a better deal than hiring human talent for those who can afford human talent and a good option for those who can’t afford to hire human talent.
*I thought of calling such stories realistic or balanced, but realism in Science Fiction is arguably a different dimension from the utopian/dystopian spectrum and balance in fiction usually refers to things like light/dark, good/evil, order/chaos, etc. I know Utopia literally means no place and was given as the name for a hypothetical perfect society because there is no such thing as a perfect society in reality and if one existed, it would likely be rather fragile, and I’m not entirely sure on the etymology of dystopia(it could just be someone mistaking the u in utopia for the eu prefix and flipping it to dys to turn a true paradise into an anti-paradise, but it could also be a case of someone whent back to the no place origin of utopia and did a swap on the root meaning no), and in either case, I have no clue what a good term for a hypothetical society that is neither shaped by the idealism of the Utopian nor the cynicism of the dystopian… Mesotopia, maybe if I thought I was in a position to coin a word with any chance to be more than personal shorthand and I’d rather use a term where the established meaning is close enough to make sense in the new context.
Been interested in writing novels and was watching some YouTube videos and was inundated by videos of AI infiltrating the literary world. Apparently, an award was given out to an AI-written story recently (by accident, of course). The LLMs are getting scary good.
I’m curious. For those that champion AI, does this bother you on some level?
For me, I like to get behind an author who speaks to me… and a part of that appeal is they are the creative architect behind their work. I’m attracted to their unique voice and feel a connection through their writing. Now, I’m supposed to applaud someone who can edit AI content to make it not sound like AI? I’m supposed to enjoy the product and ignore the lack of true craftsmanship? Sounds like a hollow proposition to me.
Using AI to generate your message content or feeding a message to AI to create a reply to someone that you paste verbatim
Posting illustrative examples of AI/LLM content as relevant to the discussion on working with AI, citing the source and demarcating the AI content with quotes or a “hide details” fold.