Developer journal · Building With LLMs

AI Can Write Code. Can It Actually Engineer Software?

Why generating thousands of lines of code is easy, but building software that actually works is still hard.

A three-part series exploring the technology, the engineering, and the experience of building Far Reach Industries as an independent developer.

In my previous article, It’s Not Really Artificial Intelligence. At Least, Not the Kind You’re Thinking Of, I explored what large language models actually do, why the term artificial intelligence has become so heavily marketed, and why sounding intelligent doesn’t necessarily mean understanding a problem.

But there’s an interesting consequence of that discussion.

Even if we recognize that LLMs aren’t artificial minds in the science-fiction sense, they can still do some genuinely impressive work.

Particularly when it comes to programming.

I’ve been using LLMs extensively while developing Far Reach Industries, an ambitious colony and industrial simulation game.

And I’ve discovered something that I think gets lost in a lot of the excitement about AI coding tools.

Writing code and engineering software are not the same thing.

Generating a thousand lines of code is easy when a model can produce them in seconds.

Determining whether those thousand lines belong in your application, whether they’ll scale, whether they introduce architectural problems, and whether they’ll remain maintainable six months later is another matter entirely.

That’s where the real challenge begins.

Writing Code Was Never the Hardest Part

There’s a common misconception about software development that seems to have become even more widespread with the arrival of AI coding tools.

People assume that programming is primarily the act of writing code.

If a developer spends several hours implementing a feature, the assumption is that most of that time was spent typing instructions into an editor.

But experienced developers know that’s rarely where the real effort goes.

The difficult work is figuring out what the software should do, how it should behave under unexpected conditions, how it interacts with everything else, and how to keep it maintainable as the project evolves.

Writing a function that calculates electrical power consumption isn’t particularly difficult.

Designing an electrical distribution system for an entire simulated colony is another matter.

Where is power generated? How does it reach individual buildings? What happens when demand exceeds generation? Can one section of the colony lose power without affecting another? What happens when infrastructure is damaged? How do you make all of that understandable to the player?

And perhaps most importantly, how does the power system interact with the rest of the simulation?

Those are engineering questions.

The code is simply how the answers are expressed.

Making code cheaper to produce doesn’t automatically make the decisions behind it easier.

The Problem With Code That Looks Right

One of the most dangerous characteristics of AI-generated code is how convincing it can be.

An LLM can produce cleanly formatted classes, sensible method names, explanatory comments, and what appears to be a thoughtfully organized architecture.

Everything looks professional.

That doesn’t mean it’s correct.

A model can misunderstand a requirement, invent a library function, overlook an edge case, or introduce a subtle bug that doesn’t become apparent until much later.

Worse, it can do all of this while providing a perfectly confident explanation of why its implementation is correct.

This is particularly troublesome when the generated code is doing something the developer doesn’t fully understand.

If I ask an LLM to implement an unfamiliar algorithm, and it returns 400 lines of sophisticated-looking code, how do I know whether the implementation is correct?

I can’t judge it by appearance.

I can’t judge it by how confidently the model explains itself.

And I certainly can’t assume it’s correct just because the application compiles.

I need tests, measurable behavior, and some way to independently verify the result.

That means understanding enough about the problem to know what correctness looks like.

In other words, I still need to do engineering.

A Working Feature Isn’t Necessarily a Good Feature

Let’s imagine I’m adding a transportation network to Far Reach Industries.

I give an LLM a prompt explaining that colonists need to move between habitats, industrial facilities, storage depots, and transportation hubs.

The model produces a working pathfinding system.

Buildings connect to a network. Colonists can move between destinations. Routes are calculated. Everything appears functional.

Success, right?

Not necessarily.

What happens when there are 10,000 colonists?

What happens when a transportation connection is removed?

Does every colonist independently calculate a path every simulation tick?

Is the network represented in a way that can be efficiently updated?

Can the simulation run without the graphical interface?

Can the player save the game and restore the network without introducing inconsistencies?

Will the system continue working when I add underground facilities, elevators, cargo transport, and emergency evacuation logic?

A feature can satisfy its immediate requirements while creating serious architectural problems for everything that follows.

This is one of the central challenges of developing a large game.

Local correctness doesn’t guarantee system-wide correctness.

An LLM can solve the problem you described without adequately accounting for the problems you haven’t described yet.

To be fair, human developers do this too.

The difference is that AI can generate a tremendous amount of locally plausible code before anyone notices the larger architectural problem.

Architecture Matters More When Code Is Cheap

There’s an interesting contradiction in AI-assisted programming.

Because code generation is becoming easier, software architecture is becoming more important, not less.

When writing code takes substantial effort, that effort naturally limits how quickly a project can grow.

When a model can implement multiple subsystems in an afternoon, the project can accumulate complexity much faster than the developer can fully understand it.

You can suddenly have a simulation with power networks, resource extraction, manufacturing, construction, environmental systems, and population growth.

Each subsystem might work reasonably well in isolation.

But now you have to make them work together.

What owns the authoritative state of the world?

How do systems communicate?

Which calculations happen every tick?

How do you prevent one subsystem from directly modifying data owned by another?

What guarantees that loading a saved game produces the same simulation state?

How do you avoid creating circular dependencies between systems?

These decisions are much harder to change once thousands of lines of code depend on them.

That’s why I’ve become increasingly interested in establishing clear boundaries in Far Reach Industries.

I want the core simulation to remain logically separate from how the game is rendered.

The game engine should be responsible for presenting the world, handling input, and displaying the simulation’s results.

The simulation itself should own its rules and state.

That separation should make it easier to test systems independently, evaluate different rendering technologies, and reason about failures without involving the entire game.

It’s also a useful boundary for AI-generated code.

Instead of asking an LLM to modify the game indiscriminately, I can constrain work to a subsystem with a defined interface and explicit expectations.

The model has less room to make assumptions that affect unrelated parts of the project.

And I have a clearer way to evaluate the changes.

The Most Valuable Thing an LLM Can Generate Might Be a Test

If you’ve used AI coding tools for any length of time, you’ve probably experienced the satisfaction of watching one implement a feature in minutes.

But I’ve come to think that one of the most valuable uses of these tools is helping create tests.

Not because AI-generated tests are automatically trustworthy. They aren’t.

A model can write a test that merely confirms its own incorrect assumptions.

But when tests are based on independently defined requirements, they provide a way to evaluate generated implementations.

Consider a resource-transfer system.

A cargo vehicle moves 100 units of material from one storage facility to another.

The obvious test is whether the destination receives 100 units.

But what happens if the destination can only hold 60?

What happens if the vehicle is destroyed midway through the journey?

Can the same cargo be delivered twice?

What happens if the game is saved while the cargo is in transit?

What if a transfer is canceled?

Where does the material exist at every point in the process?

Those questions suggest properties that should always hold true.

For example, unless the simulation explicitly models production, consumption, or loss, transferring resources should not change their total quantity.

That gives us a useful invariant:

Resources cannot appear or disappear merely because they’re being moved.

Now I have something concrete to test.

The same approach applies to power networks, manufacturing, population, construction, and practically every other simulation system.

An LLM can help enumerate edge cases and generate automated tests.

But the underlying expectations need to be deliberate.

And ideally, the tests should be validated separately from the code they’re supposed to verify.

Otherwise, we’re simply asking the same system to grade its own homework.

The AI Doesn’t Have to Be the Architect

There’s a growing temptation to treat AI coding tools as autonomous software engineers.

Give the model a goal, let it explore the repository, and allow it to implement whatever changes it decides are necessary.

That approach can be genuinely useful, especially for well-defined tasks.

But I think the amount of autonomy a model should have depends on how easily its work can be verified.

Fixing a narrowly scoped bug with a reproducible test is very different from redesigning the simulation’s data model.

Changing a UI component is different from replacing the system responsible for maintaining consistent world state.

For Far Reach Industries, I find it useful to think about three levels of responsibility.

1. Decisions I want to own

These are the decisions that define the game and determine the long-term structure of the project.

What kind of simulation am I building? What should be realistic, and what should be abstracted? How should subsystems communicate? What are the performance requirements? What architectural trade-offs am I willing to accept?

An LLM can help me explore these questions.

It can challenge assumptions, suggest alternatives, and explain consequences I might have overlooked.

But I want to understand and own the final decisions.

2. Work I can delegate with constraints

This is where LLMs are particularly valuable.

Implementing a defined interface, adding tests, building a parser, refactoring a self-contained module, or implementing a known algorithm can all be good candidates.

The important part is establishing boundaries.

What files can change? What behavior must be preserved? What tests must pass? What constitutes completion?

The clearer those constraints are, the easier it becomes to evaluate the result.

3. Work that must be independently verified

Some problems simply can’t be resolved by asking the model whether its own solution is correct.

Performance is one example.

If I need to know whether an engine can handle tens of thousands of simulated objects, I need a benchmark.

If I need to compare rendering approaches, I need representative scenes and measurements.

If I need to know whether a simulation produces stable results, I need repeatable tests.

An LLM can help design the experiment.

It cannot replace the evidence produced by running it.

The model can propose a solution. Reality gets the final vote.

When AI Accelerates the Wrong Thing

There’s another problem that doesn’t get discussed enough.

An LLM can make it extremely easy to keep adding features.

Have an idea? Ask the model to implement it.

Need another menu? Generate one.

Want to expand the simulation? Add another subsystem.

Suddenly the game has an enormous collection of features, but no coherent playable experience.

This is especially dangerous for solo developers.

Traditionally, limited time and resources create a kind of natural discipline. You have to choose what is worth implementing.

AI reduces some of those constraints.

That’s wonderful for productivity, but it can also encourage unchecked scope expansion.

And for a simulation game, almost every new feature creates dependencies.

Adding atmospheric pressure isn’t simply adding another number to the interface.

It might require changes to buildings, environmental systems, population health, emergency events, power consumption, construction, and the underlying world representation.

The implementation of the individual feature might be quick.

The cost of integrating it into the entire game may not be.

This has forced me to think more carefully about what Far Reach Industries actually needs.

Not what would be interesting to implement.

Not what an LLM could generate in an afternoon.

What contributes to the experience I’m trying to build?

That’s a product-design question as much as an engineering question.

And no amount of automatically generated code can answer it for me.

The Problem of Understanding Your Own Project

One of my biggest concerns with AI-assisted development is the possibility of becoming disconnected from the software I’m building.

It’s easy to imagine a developer repeatedly asking a model to implement changes, accepting the results, and moving on.

Over time, the codebase grows beyond what the developer can comfortably understand.

Eventually, something breaks.

The developer asks the AI to fix it.

The AI modifies several other systems.

Something else breaks.

Another prompt. Another fix.

The project slowly becomes a collection of generated changes layered on top of previous generated changes.

At that point, the developer is no longer really directing the software.

They’re negotiating with it.

I don’t want that to happen with Far Reach Industries.

I don’t necessarily need to personally write every line of code.

But I do need to understand the important boundaries, responsibilities, assumptions, and failure modes of the systems I’m building.

That doesn’t mean memorizing an entire codebase.

No developer does that, especially on a large project.

It means maintaining enough architectural understanding to make informed decisions and recognize when an implementation violates those decisions.

Documentation helps.

Tests help.

Clear module boundaries help.

Frequent reviews of what has actually changed help.

And sometimes the right response to an AI-generated implementation is simply to throw it away and start again with a better design.

The ability to generate code cheaply also means we should be more willing to discard bad code.

Can an LLM Reason About Software?

This is where the question gets more complicated.

Because modern LLMs can do more than mechanically produce syntax.

They can compare architectural alternatives, identify potential race conditions, trace execution paths, analyze error messages, and propose changes across a repository.

Some of that is reasonably described as reasoning.

And with access to compilers, test runners, debuggers, and other tools, coding agents can iteratively investigate and solve problems rather than simply generating a single response.

So it would be unfair to claim these systems are incapable of software engineering activities.

They clearly can perform meaningful portions of the work.

The distinction is between performing engineering tasks and reliably carrying responsibility for the engineering outcome.

Engineering isn’t simply producing a plausible solution.

It’s establishing that the solution satisfies the requirements, understanding the consequences of trade-offs, and maintaining confidence in the system as it changes.

Today, I don’t think an LLM should be assumed capable of taking that responsibility independently across a large, evolving software project.

That doesn’t mean future systems won’t become substantially more capable.

It means that right now, I need to treat their outputs as work to evaluate, not conclusions to accept.

Building Far Reach Industries With AI

All of this might sound like an argument against using LLMs for serious development.

It isn’t.

I’m building Far Reach Industries with substantial help from these tools precisely because I believe they’re valuable.

They’ve made it possible to explore ideas, implement systems, and investigate technologies at a pace that would have been difficult for me to sustain as a hobby developer.

I can spend more time thinking about how the game should work and less time on repetitive implementation.

I can examine architectural alternatives without committing weeks to each one.

I can build testing infrastructure that might otherwise get postponed because there’s always another gameplay feature to implement.

And I can pursue a project with a level of ambition that might have been unreasonable for me to attempt alone.

But there’s a catch.

The faster the code arrives, the more disciplined I have to be about deciding what belongs in the project.

I still have to make the systems coherent.

I still have to test them.

I still have to understand why they work, why they fail, and whether they’ll continue working as the game grows.

And I still have to make a game that’s actually enjoyable.

The LLM can help me build the machinery.

It can’t decide whether the machinery makes for a good game.

Engineering Still Matters

So, can AI actually engineer software?

I think the answer is more interesting than a simple yes or no.

Modern LLMs can perform meaningful engineering tasks. They can evaluate designs, identify potential problems, generate implementations, develop tests, and help investigate failures.

But performing engineering tasks isn’t the same as reliably assuming responsibility for an entire software system.

For now, I still need to provide the architectural direction, understand the trade-offs, independently validate the results, and decide what belongs in the project.

And as code becomes easier to generate, those responsibilities become even more important.

The code is getting cheaper. The engineering still matters.

But that leaves another question.

If these tools still require so much oversight, are they actually worth using?

For me, the answer is an emphatic yes.

Not because they eliminate the difficult work, but because they fundamentally change how much work one developer can realistically attempt.

That’s exactly what I’m exploring with Far Reach Industries.

In the next article, I’ll move beyond the theory and explain how I’m actually using LLMs to build the game: from architecture and implementation to engine benchmarks, testing, and the engineering decisions I still have to make myself.

Up next: AI Isn’t Building My Game. It’s Changing How I Build It.

How LLMs are changing my development workflow—and making an ambitious solo project more realistic.

Far Reach Industries is a hard-science colony simulation I’m developing under Crown & Crabapple, my independent game studio. I’m documenting both the development of the game and my experience using large language models as programming tools—the successes, mistakes, benchmarks, and architectural decisions along the way.

Follow the development of Far Reach Industries, a science-fiction colony and industrial simulation game from Crown & Crabapple.