AI Realism
– 14 minutes read – AII’m writing this because I am worried. Worried about the future. The future of software, the role AI will play in the creation and maintenance of it, and how I will fit into this. What will software engineering look like in a few years? Is there a place for me and my colleagues in this world? What will our jobs look like?
And then, I am worried about the divide I’m witnessing. Online, but also among colleagues.
Those who love and embrace the seemingly endless possibilities of modern AI.
Those who hate what our “vocation” is being reduced to. The art of software creation that’s losing its soul.
On the one hand, it’s hard to ignore the possibilities that AI and LLMs are bringing to the world of software. I’ve never generated a landing page for an idea I had so quickly (and it looked good)! We can argue about the code behind it and how I might’ve done it differently. But I surely could not have created anything similar at the speed at which my LLM of choice took my written ideas and desires and transformed it to something workable.
On the other hand, I see communities being flooded with new projects almost hourly. Normally, this would be a great thing! But it’s not. Most of it is “abandonware”, an idea someone cooked up last Thursday. Spend a weekend talking to an LLM and generating code, and shotgunned it over the internet just to see if it would stick. To then get distracted by yet another cool idea of something you could build. This has been dubbed “AI slop” or “slopware”, and when you see the sheer volume of posts that look identical, with codebases not even a day old: you understand why.
We also see similar effects in the Open Source community. Projects large and small closing doors (to PR’s or issues, or worse), going closed source or changing their licensing. The influx of AI assisted and LLM powered issues, nonsense pull requests and others has proven to be just too much for humans to handle. To me this seems like we’re gutting what has been the core of the software development world for decades, by destroying it from within.
And there is the professional angle. We’re all convinced our future is changing and the way we do our work will probably never look like it did before. And then there’s that moment where reality strikes again. A lot of our enterprise clients work in heavily secured environments. Most (sane) companies would not just let any AI get full access to all of their data… Right?
And we do see some hesitations there. Not every company I’ve talked or worked with has just blindly accepted an AI to consume all of their data in the hopes making their employees faster and more efficient. But the hype is powerful and the risk of falling behind to the unknowns of what AI can do is even greater. So a lot of companies are exploring. And they should! The benefits to have your developers go through the software development cycle faster and more efficiently is incredible. And the sometimes mediocre results seem to be cheap enough for trial, that it easily pays for the times where a debugging session is sped up and improvement significantly when an LLM does point you in the right direction for example. Or the suggested solution to a problem or implementation does actually somewhat make sense within the project.
Then there are the companies that have a completely different problem. Their software has been in existence for years. To them, factors like continuation, maintenance and reliability is much more important than speeding up future development or adding new features.
And this is where the current generation of LLMs seems lacking to me.
New project? Fastest start you’ve seen. Set up some boilerplate? I can’t type that fast and frankly it’s boring to do. Add a feature? I never thought of doing it that way. Refactor something bloated to a clean and modular architecture? It sure looks nice, but how do I know it has covered everything that went into the previous solution? Let’s not even get into about legacy software and the problems that carries, like outdated versions and practices. (Right now)
So why is it that AI shines in the creation of new things, but it is seemingly much less reliable at maintenance work?
Context
My opinion on this can be summarised into a single word: context.
Imagine the following: you’re working on a new project, creating a plan together with your AI of choice. Writing down some choices, making decisions and setting up the guidelines on which you want to develop a feature.
This “AI first” approach is fantastic. You slowly build up the necessary context the LLM requires to generate things according to the plan as you envision them. If you have some experience, you might even set up some context that will make the LLM write code in a specific way that you prefer, using a certain framework you know well or using a particular set of tools you’ve enjoyed working with in the past.
When you set up a repository in this way, spending time planning and describing the way you want things done, the LLM is a fantastic productivity booster and increases the speed at which you can start producing software.
But let’s now flip the situation around. We’ve inherited some legacy codebase. A project predating the AI assistance and generation as we know it today. This context we’ve deemed so valuable for AI to make the “correct choices” has not been gathered in the same way we were describing just now. And more importantly, the relation and reasoning between choices as they were made in the past might not be as obvious to understand anymore.
And surely, we could go over the existing documentation, through SharePoint’s and Confluence pages. Retrieve data from old Jira tickets, Git history and slowly build up some of this context back (using AI of course!). It does seem like a completely different approach to how the greenfield project came to be and gathered its context.
From practice, we saw that this lack of context made AI much less effective in actually coming up with useful solutions. The idea behind the solution wasn’t necessarily wrong, but within the context of an existing project they often just didn’t make sense. For example, bringing a modern Java API that would theoretically make a whole class lot cleaner, into a project that’s been locked to Java 8 for years (and simply didn’t support modern Java 21 APIs). Or creating an elegant solution in an Angular application with Signals, theoretically bringing a much more reactive approach to a problem we were tackling, into a project running on a pre-signal version of Angular (and locked there cause of external library dependencies).
Or the other way around, bringing an older way of working to a question and solving things the way they’ve always done. For example using old RxJS practices and Angular dependency injection, when we had just upgraded a project to the latest and greatest Angular versions. The reason it did these things, was because of the context. Or lack thereof. And that is not even touching upon much deeper and complicated hidden logic that an LLM might miss because of its lack of context.
When it has no guardrails to go off, no specific instructions, the AI uses what is in its context. A certain set of files automatically referenced because they seemed relevant. Creating a context of old patterns and ways of working, where you may want to move into more modern practices and do a little bit of refactoring to help bring your application into more modern standards.
This in itself is nothing new of course, and it surely isn’t a problem with just AI usage. The same problem might have happened when the job was handed to less experienced junior developers for example, who might also take a copy/paste approach to a problem, when there’s similar solutions at hand. It always takes experience to determine and guide what context is relevant to the situation at hand. (And unfortunately just saying “You are a senior developer” doesn’t cut it.)
But it brings me back to one of these fears I have for the future. How will we train our junior developers? Where will they pick up the experience to know why a certain decision will end in tears? The way we learned and got experience was by doing, by doing it ourselves. And then dealing with the problems for months to come as they may have arose. With AI as a tool on your toolchain, that pain might become less obvious.
Or will it become more obvious? With more painful lessons to come? Security breaches? Stolen data? Unfixable problems that can only be resolved by redoing the whole thing? AI tools that keep coming with more broken solutions to an already broken codebase? Who knows?
Ownership & responsibility
In the end, someone has to answer and bear the responsibility of the system as a whole. The argument of “AI wrote it” simply doesn’t cut it when an important system goes down or makes a complicated mistake that’s hard to recover from. Someone will have to go in, understand the problem and take the right mitigations to get the systems back to healthy.
It’s not unthinkable that this will be some AI-driven agent in the future, that goes in and takes a first stab. Especially with the enormous flight and hype that AI-agents have received recently. But even when that is the case, and the LLMs might catch and fix some situations. A person somewhere will have to be its backup. That someone will have to understand how the system works, how it ties together. How it all talks to each other and what an obscure error might point to. Because in the business world, blaming an AI for any mistakes, or a database that was dropped, doesn’t solve your problems. Certain guardrails are required, both on the AI and on the system to make sure that you can only do so much damage when things go wrong.
This in itself is not new to the world of AI. We’ve had juniors make mistakes like this, and they were plenty useful as learning mechanisms. Both for the junior in question, but also for the rest of the development team that was responsible. They created better guardrails, re-checked their backup strategy and disaster recovery plans. Automated the recreation of their infrastructure perhaps. All things we can also do with AI of course, but understanding why (and how) it all ties together and why we need it is something we’re going to have to tell AI. But we’ll have to provide the instruction about why it is important and what we have to think about.
With the enormous rise of AI, and all the possibilities it unlocks, we have to think about responsibility: Who owns the system? Who controls it? And if we only rely on AI to keep our business running, are we really in control?
Cognitive debt
A term coined by blogs (when was the last time you read a good blog?), relating to technical debt as we know it in software engineering in very much the same way. But instead of needing to repay technical choices (or shortcuts) from the past to your codebase, we’re lending out understanding. Not just as a developer within the codebase, but as teams. Our shared understanding of how our software works, what we document and the confidence we have in changes we make diminishes by the (over) use and reliance on AI for doing (part of) our job.
And it’s not just a “vibe” some people are having. People like Martin Fowler, Dax (creator of OpenCode) have made mentions of seeing the same. And even Anthropic themselves have published results of a research they conducted concluding that:
One reason that the atrophy of coding skills is concerning is the “paradox of supervision”—as mentioned above, effectively using Claude requires supervision, and supervising Claude requires the very coding skills that may atrophy from AI overuse.
In my own experiences I’ve seen something similar. When experimenting with Agentic development to determine what was possible, and what I would think of the resulting quality of the code, I noticed that the speed at which it created made it difficult for me effectively review and understand what is was doing. Finding a bad decision or misunderstood (or incorrect) part of the spec felt sometimes more like a stroke of luck, then having the mind to stay thorough through thousands of lines of code being generated at a blazing pace.
And although in most cases I was blown away by some of the results, most things also didn’t feel like what I was looking for exactly. Something I found hard to put into words what was off about, so that made it difficult to spec for or describe as my AI agent and I iterated on the problems and features I was experimenting with.
When it comes to the future of software development, I feel this is where the biggest trap and dangers lie for teams. If teams get gutted and the team can do the same work with less people, that’s when this cognitive debt will show itself most. The knowledge of how the system works will not walk away, but the understanding of why and the ownership of how it works a certain way will have left. The “cognitive ownership” if you will. We as software developers should be afraid of this and think about how we can protect ourselves, our team and the software we work on from this happening. By that I don’t mean how can we stop the rise of AI and decrease our possible productivity; no. I mean how can we create the correct guardrails and documentation so that other developers, but perhaps also our future Agentic and LLM based colleagues can easily retrieve the context required to understand the why and how behind certain decisions.
Summary & Conclusion
So what does this all mean for us? Are we doomed? Are we effectively working towards the resolution of our own jobs? Perhaps. There was an “old” saying that every good software engineer and developer should work towards making themselves obsolete. The purest form of automation. Although I couldn’t find this exact quote, I found one that seems to imply the same meaning:
Good engineers make solutions obsolete. Great engineers make themselves obsolete. –Jordan W
And perhaps the introduction and use of AI is just another (big) step into that direction. But even Antrophic in their previously mentioned research admitted to seeing the diminishing of coding skills (“atrophy”).
atrophy
verb | uk/ˈæt.rə.fi/ us/ˈæt.rə.fi/
(of a part of the body to be reduced in size and therefore strength, or, more generally, to become weaker:
Where in this case that would mean the brains, and more specifically the skillsets developed over years of solving logic puzzles in your mind. This is where I think the biggest personal risk lays for any software developer. The slow decay of your own (hard earned) critical thinking and puzzle solving skills that resulted in the writing of code. It was never about the writing of code itself, it was about being able to reason through a problem and understand it (or most of it) from start to finish. Then being able to express these solutions in a syntax that a computer could understand.
Then again, we should also be wary of how much power we’re giving to our LLM providers. How much of our workflows (and critical “thinking”?) we’re directly attaching to external providers and the power that gives these companies. What happens when we’ve fully adopted AI based workflows and engineering, but the price per token goes up so much that it’s no longer profitable? We’ve already seen price surges. Not to mention other more ethical parts of the discussion like models being made unavailable by external (political) forces, the amount of water and energy this technology and their datacenters uses and how unsustainable it all seems to be right now. I’d argue that’s food for thought in being an AI realist and staying in control over our own ways of working.
Programming is not about typing, it’s about thinking. –Rich Hickey
And there in, to me, lies the biggest factor of all. And the thing I hope you’ll take away from my talk this post today. When we’re using AI to speed up, to be more productive. We should sometimes slow down, stop and think. Am I using AI to think better? Or am I using AI to avoid having to think?
Sources
Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task — MIT Media Lab
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt
What I’m Hearing About Cognitive Debt (So Far)
Agentic Coding is a Trap | Lars Faye
https://blog.devgenius.io/software-engineering-great-quotes-3af63cea6782