Why LLMs Can't Really Build Software - Zed Blog

They cannot build software because they cannot maintain two similar "mental models", identify the differences, and figure out whether or not to update the code or the requirements.

I asked three agents to fix the same issue yesterday and they all came up with different solutions.

Clearly LLMs are useful to software engineers. They can quickly generate code, and they are excellent at synthesizing requirements and documentation. For some tasks this is enough: the requirements are clear enough, and the problems are simple enough, that they can one-shot the whole thing.