The post is about using LOC as a metric when making any sort of point about AI. Nowhere do I suggest someone shouldn't use it, nor that they should expect negative results if they opt to.
No one I’ve ever worked with in 40 years has ever seriously used loc as a measurement of progress or success. I honestly don’t know where this comes from.
That’s the odd psychosis here. Everyone knew loc was a terrible measure. But perhaps the instinctual pull was always there, and now that you can generate halfway coherent tens of thousands of loc in hours, our sensibilities are overwhelmed.
Yes, but it comes up in conversations of LLMs a lot. Thus, the rant in question. I think we are in agreement, or at least we lack disagreement, because that is the only stance I endeavored to take in the post.
The full PDF is available for download. It's mostly a series of essays, so you can pick and choose and read nonlinearly. It's worth thinking about beyond nihilistic takes.
> ALL OF IT is meaningless. It's a pointless discussion.
That is a nihilistic take. That it's pointless and everything about the domain (measurement of productivity) is meaningless.
"Productivity" is no more made up of a concept than "good work". The fact that many attempts to measure it are quite dumb (e.g., the idea of using LOC as a metric) does not mean the whole idea is not worth considering or discussing.
> In practical terms, "productivity" is any metric that people with power can manipulate (cheating numbers, changing narratives, etc) to affect behavior of others to their interests.
Metrics can be gamed, and to me this is the deepest flaw in accepting that metrics can even tell something meaningful.
It is a nuanced discussion, and it's not entirely dismissals. It's just that to enter it, one must accept the reason why productivity metrics exist in the first place (even if provisionally, just for the sake of the argument).
As stated in the article, I have unlimited access to multiple frontier models and I use Claude Code, among other harnesses. The rest of your list is not directly addressed in the post, because it is irrelevant to the point being made, but I do all of those things and more. You will note that in the appendix on LLM usage, some of the things I constantly have to correct in LLM-generated code are testing mistakes. And if you care to ask, yes I have context files to address these mistakes, and I iterate them to try to improve the experience.
I would honestly appreciate constructive feedback on LLM usage, because, as I stated, I am constantly having to rework code that LLMs generate for me. The value I get from LLMs is not in code generation.
> The complaints about LLM's that lack any information about the domains being worked in, the means of integration (deep in your IDE vs cut and paste into vim) and what your asking it to do (in a very literal sense) are the critical factors that remain "un aired" in these sorts of laments.
I'm not sure if this is a direct response to the article or a general point. The article includes an appendix about my use of LLMs and the domains I have used them in.
Not GP, but your appendix about LLM usage matches exactly how I use it too: mainly for rubber ducking and research. The codegen it's useful for (that I've found) is generating unit tests. Code coverage tools and a quick skim are more than sufficient for quality checks since unit tests are mostly boilerplate and you want to make sure that different branches are being covered.
I've had a large project recently which has biased my view on unit testing from LLMs. It includes a lot of parsing and other workflows that require character-specific placement in strings for a lot of tests. Due to how tokenization works, that is a gnarly use case for LLMs. I am trying not to form too many strong opinions about LLM-driven-TDD based on it. My forays into other domains show better results for unit tests, but the weight of my experience is in this particular low point lately.
I didn't set out to teach you anything, change your behavior, or give you practical takeaways, so it's a rant (: Emotions can be expressed with citations.
I am fully on board with gen AI representing a paradigm shift in software development. I tried to be careful not to take a stance on other debates in the larger conversation. I just saw too many people talking about how much code they're generating as proof statements when discussing LLMs. I think that, specifically---i.e., using LOC generated as the basis of any meaningful argument about effectiveness or productivity---is a silly thing to do. There are plenty of other things we should discuss besides LOC.
I wonder if you have a take on measuring productivity in light of the potential difficulty of achieving good outcomes across the general population?
You mention in the second appendix (which I skipped on my first read), that you are a rather experienced LLM user, with experiences in all the harnesses and context management which are touted as "best practice" nowadays. Given the effort this seems to take, do you think we're vulnerable to mis-measuring.
My mind is always thrown to arguments about Agile, or even Communism. "True Communism has never been tried" or "Agile works great when you do it right", which are still thrown about in the face of evidence that these things seem impossible, or at least very difficult, to actually implement successfully across the general population. How would we know if AI-driven-development had a theoretical higher maximum "productivity" (substitute with "value", "virtue", "the general good", whatever you want here) than non AI-driven-development, but still a lower actual productivity due to problems in adoption of the overall paradigm?
Measuring productivity in software development is a hard problem, beyond the typical categorizations used in computer science. Unfortunately, I think my best answer is to go read the book I linked in the conclusion: https://link.springer.com/chapter/10.1007/978-1-4842-4221-6_...
That is an unsatisfying answer. I can point to anecdotes that suggest AI is hurting productivity or improving it, but those don't make an argument. And the extremes on either side make it very difficult to consider. How do you weigh "An LLM deleted my production database" against "I built a business on the back of AI-assisted software"?
I think we have to wait and see. And we should revisit questions of cost and value continuously, not just about LLMs, but generally in life. Most of my motivation (though not an overwhelming majority) around using LLMs right now is a mix of curiosity and wanting to avoid the fate of the steam shovel.
That’s my entire issue with AI. How quickly people are pushing adoption without the evidence to back that up. My buddy works for block and he said they fired 70% of their engineers in a bid to force the remaining 30% to use AI in order to keep up.
My very large tech company has made it a goal for each engineer to spend their salary in tokens.
You can make a big bet on AI without risking the entire company. How about we wait for some evidence that shows measurable productivity increases before betting the farm.
I actually consider that the claim is not that bold, and in fact has been common in our industry for most of the short time it has been around. I included a few articles and studies with time breakdowns of developer activity that I think help to illustrate this.
If an activity (getting code into source files) used to take up <50% of the time of programmers, then removing that bottleneck cannot even double the throughput of the process. This is not taking into account non-programmer roles involved in software development. This is akin to Amdahl's law when we talk about the benefits of parallelism.
I made no argument with regard to threat to the profession, and I make none here.
