> It almost sounds like you’re saying there’s essentially an LLM inside everyone’s brain. Is that what you’re saying?
>Pretty much. I think the language network is very similar in many ways to early LLMs, which learn the regularities of language and how words relate to each other. It’s not so hard to imagine, right?
Yet, completely glosses over the role of rhythm in parsing language. LLMs aren’t rhythmic at all, are they? Maybe each token production is a cycle, though… hmm…
I think it's obvious that she means that it's something _like_ LLMs in some aspects. You are correct in that rhythm and intonation are very important in parsing language. (And also an important cue when learning how to parse language!) It's clear that the human language network is not like LLM in that sense. However, it _is_ a bit like an _early_ LLM (remember GPT2?) in the sense that it can produce and parse language, not that it makes much deeper sense in it.
However ... language production and perception are quite separated in our heads. There's basically no parallel to LLMs. Note that the article doesn't give any, and is extremely vague about the biological underpinnings of language.
> language production and perception are quite separated in our heads
Do you have any evidence for this?
I am a former linguistics student (got my masters), and, after years of absenteeism in academia, interested in the current state of the affairs. So: "quite separated in our heads" Evidence for? against?
Afasia, and general measures of "normal" performance.
There are various kinds of afasia, often linked to specific brain areas (Wernicke's and Broca's are well-known). And M/EEG and fMRI research suggests similar distinctions. It is difficult to reconcile with the idea that there is only one language system.
And you will also have noticed that your skills in perception and production differ. You can read/listen better than write/speak. Timing, ambiguity and errors in perception and production differ.
And more logically: the tasks are very different. In perception, you have to perceive the structure and meaning from a highly ambiguous, but ordered input of sound triggering auditory nerves, while during production, meaning is given (in non-linear order), and you have to find a way to fit it in a linear, grammatical order with matching words, which then have to be translated to muscle movements.
Ah, totally agreed. At least there is a clear auditory / motor part in the tasks that seems quite separate.
However, I find it also unlikely that the networks are totally separate, and I wonder if there are any evidence of areas that encode the "core/abstract" linguistic de/serialization (multidimensional and messy internal semantic information ←→ linear morphophonological information) both ways, or at least mechanism that manages to use gained input network competence to "train" or "manage" output network competence.
Why? Because even though, as you say, there is a differing performance in perception and production, there is also plenty of evidence of gaining linguistic competence from input, and then managing to convert that to performance in output.
> It's clear that the human language network is not like LLM in that sense.
Is it though? If rhythm or tone changes meaning, then just add symbols for rhythm and tone to LLM input and train it. You'll get not just words out that differ based on those additional symbols wrapping words, but you'll also get the rhythm and tone symbols in the output.
>Yet, completely glosses over the role of rhythm in parsing language.
If you're talking about speech cadence/rhythm, then we also parse written language which doesn't have that. And we're quite capable of parsing a monotone robotic voice speaking with a monotonous mechanical rhythm too.
I used this to integrate image generation and gif generation in my Claude coding — very helpful for making beautiful websites. They have Midjourney API
I looked that up. It sounds like you have to start an entrepreneurial business there (in which you invest at least 5k). I have no interest in starting my own business, so that is probably not an option.
Is housing really unaffordable in US? How many years average Joe needs to work to buy a 1bd apartment in the big city, let's say 500sq ft? Price in annual net salary is much lower in USA than in big EU cities.
> I find it very appealing to consider the idea that the world is not somehow running “hidden mathematics”, somewhere and somehow, to solve some complicated equations in a seemingly magical way, but rather, that things are radically simpler, in that the world is simply implementing a set of trivially simple rules. The world is not concerned with, or made with mathematics, mathematics just emerges, with inherent and irreducible complexity, from extreme simplicity.
Wouldn’t those simple rules be mathematics? It’s very hard for me to see how the world isn’t made of math. Then again, I am a Pythagorean.
Mathematics is just a tool, like language, for describing reality, not reality itself.
A cake is not made of numbers like 5 cups flour + 3 eggs, but we can model it as such. In principle we could invent any such system of symbols to describe the physical world but those symbols don’t define it. The physical world only nudges us toward what symbols work and which don’t.
Not a strawman. I’m not stating that math isn’t real. It’s real as an abstract framework that humans create and refine. It’s not, however, foundational to the physical world in the way fundamental particles or gravity is. Numbers and equations don’t push particles around, they simply help us represent that kind of phenomena that we observe.
There is a distinction between "What the universe is made of or how it works just happens to be really compatible with how math describes things" and "The universe is just "running" math and we discovered that math and use it for other things"
But like, words stop working at these levels of rigor.
What the hell does "The universe is made of math" mean? How can something be made of a field of study? Where is the "Addition" particle? How does 1+1=2 give rise to what we see as an electron?
Like it's bad enough dealing with "quantum fields" that might be "real" or maybe are just really nice mathematical objects that happen to be useful for calculating the future.
Does math take up space? Does space take up math? Does blue afraid of seven? Can I eat integrals or will they go straight to my thighs?
If the universe is "made of" math, what is the consequence? For example, the consequence of being made of "quantum fields" in my lay mind is that we get observations like entanglement and the hilarity of whatever is going on in the higgs field.
>Then again, I am a Pythagorean.
Ah, let me just move this sqrt(2) out of the way real quick :P
I want simple rules because I am a simple man, and if those simple rules happen to actually be math, that sucks for me because the "simple rules" are really hard math.
Unsayable numbers (the way the Greeks said irrational numbers) can take the wrong meaning. Like, why are they unsayable? Because you’d die before you could say them. Well, it’s not a threat!
Then it turns into this whole ahistorical fabrication impugning Pythagoras who was, otherwise, pretty much the most incredible guy ever.
Now, the “addition particle” is a strawman, but harder to deal with is just numbers. Are numbers real? Are there discrete “things” in the universe? Well, yes there are. Frequencies or quanta do just fine. Now, when there are numbers, they can be added, whether we want to or not.
Another example would be geometries. Are spheres real? Surely! Do they exist on any planet in the universe? It would seem. Are there any perfect spheres? Nope. Do they precede matter and energy? It would seem.
I think we are saying the same thing. Unfortunately, these beliefs are slippery and metaphysical. I take pride, though, in the pythagoreanness of so many of the scientific greats, from Newton to Penrose.
> Just as the physician must often make painful incisions during the treatment of individuals, we must also make incisions in the national body, out of a sense of responsibility: we must make sure that those patients who would pass on their diseases to distant generations, to the detriment of the individual and of the Volk, are prevented from passing on their diseased hereditary material
90% of human devs can fit more than 3-5 files into their short-term memory.
They also know not to, say, temporarily disable auth to be able to look at the changes they've made on a page hidden behind auth, which is what I observed Gemini 3 Pro doing just yesterday.
Ok, and that’s your prediction for 2 years from now? It’d be quite remarkable if humans had a bigger short term memory than LLMs in 2 years. Or that the kind of dumb security mistakes LLMs make today don’t trigger major, rapid improvements.
Do you understand what the term "context window" means? Have you ever tried using an LLM to program anything even remotely complex? Have you observed how the quality of the output drastically reduces the longer the coversation gets?
That's what makes it bad at security. It cannot comprehend more than a floppy drive worth of data before it reverts to absolute gibberish.
There's about a dozen workarounds around context limits, agents being one of them, MCP servers being another one, AGENTS.md being the third one, but none of them actually solve the issue of a context window being so small that it's useless for anything even remotely complex.
Let's imagine a codebase that can fit onto a revolutionary piece of technology known as a floppy drive. As we all know, a floppy drive can store <2 megabytes of storage. But a 100k tokens is only about 400 kilobytes. So, to process the whole codebase that can fit onto a floppy drive, you need 5 agents plus the sixth "parent process" that those 5 agents will report to.
Those five agents can report "no security issues found" in their own little chunk of the codebase to the parent process, and that parent process will still be none the wiser about how those different chunks interact with each other.
You can have an agent that focuses on studying the interactions. What you're saying is that an AI cannot find every security issue but neither do humans otherwise we wouldn't have security breaches in the first place. You are describing a relatively basic agentic setup mostly using your AI-assisted text editor but a commercial security bot is a much more complex beast hopefully. You replace context by memory and synthesis for instance, the same way our brain works.
In one instance it could not even describe why a test is bad unit test (asserting true is equal to true), which doesn’t even require context or multi file reasoning.
Its almost as if it has additional problems beyond the context limits :)
You may want to try using it, anecdotes often differ from theories, especially when they are being sold to you for profit. It takes maybe a few days to see a pattern of ignoring simple instructions even when context is clean. Or one prompt fixes one issue and causes new issues, rinse and repeat. It requires human guidance in practice.
Strongman: LLMs aren't a tool, they're fuzzy automation.
And what keeps security problems from making it into prod in the real world?
Code review, testing, static and dynamic code scanning, and fuzzing.
Why aren't these things done?
Because there isn't enough people-time and expertise.
So in order for LLMs to improve security, they need to be able to improve our ability to do one of: code review, testing, static and dynamic code scanning, and fuzzing.
It seems very unlikely those forms of automation won't be improved in the near future by even the dumbest form of LLMs.
And if you offered CISOs a "pay to scan" service that actually worked cross-language and -platform (in contrast to most "only supported languages" scanners), they'd jump at it.
And that buys you what, exactly? Your point is 100% correct and why LLMs are no where near able to manage / build complete simple systems and surely not complex ones.
Why? Context. LLMs, today, go off the rails fairly easily. As I've mentioned in prior comments I've been working a lot with different models and agentic coding systems. When a code base starts to approach 5k lines (building the entire codebase with an agent) things start to get very rough. First of all, the agent cannot wrap it's context (it has no brain) around the code in a complete way. Even when everything is very well documented as part of the build and outlined so the LLM has indicators of where to pull in code - it almost always cannot keep schemas, requirements, or patterns in line. I've had instances where APIs that were being developed were to follow a specific schema, should require specific tests and should abide by specific constraints for integration. Almost always, in that relatively small codebase, the agentic system gets something wrong - but because of sycophancy - it gleefully informs me all the work is done and everything is A-OK! The kicker here is that when you show it why / where it's wrong you're continuously in a loop of burning tokens trying to put that train back on the track. LLMs can't be efficient with new(ish) code bases because they're always having to go lookup new documentation and burning through more context beyond what it's targeting to build / update / refactor / etc.
So, sure. You can "call an LLM multiple times". But this is hugely missing the point with how these systems work. Because when you actually start to use them you'll find these issues almost immediately.
To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too. It’s not more reliable than just taking a random person off the street of SF and giving them instructions on what to copy paste without any context. They might also change unrelated things or get sidetracked when they encounter friction. My point is that when you try to compensate by prompting repeatedly, you are just adding more chances for entropy to leak in — so I am agreeing with you.
> To add onto this, it is a characteristic of their design to statistically pick things that would be bad choices, because humans do too.
Spot on. If we look at, historically, "AI" (pre-LLM) the data sets were much more curated, cleaned and labeled. Look at CV, for example. Computer Vision is a prime example of how AI can easily go off the rails with respect to 1) garbage input data 2) biased input data. LLMs have these two as inputs in spades and in vast quantities. Has everyone forgotten about Google's classification of African American people in images [0]? Or, more hilariously - the fix [1]? Most people I talk to who are using LLMs think that the data being strung into these models has been fine tuned, hand picked, etc. In some cases for small models that were explicitly curated, sure. But in the context (no pun) of all the popular frontier models: no way in hell.
The one thing I'm really surprised nobody is talking about is the system prompt. Not in the manner of jailbreaking it or even extracting it. But I can't imagine that these system prompts aren't collecting mass tech debt at this point. I'm sure there's band aid after band aid of simple fixes to nudge the model in ever so different directions based on things that are, ultimately, out of the control of such a large culmination of random data. I can't wait to see how these long term issues crop and and duct taped for the quick fixes these tech behemoths are becoming known for.
Talking about the debt of a system prompt feels really weird. A system prompt tied to an LLM is the equivalent of crafting a new model in the pre-LLM era. You measure their success using various quality metrics. And you improve the system prompt progressively to raise these metrics. So it feels like bandaid but that's actually how it's supposed to work and totally equivalent to "fixing" a machine learning model by improving the dataset.
Agreed, but "vibe coding will be better at security" is not one of them. Better by which metric, against which threat model, with which stakes? What security even means for greenfield projects is inherently different than for hardened systems. Vibe coding is sufficient for security today because it's not used for anything that matters.
It'll play a role in both securing and security research I'm sure, but I'm not confident it'll be better.
But also, you'd need to have some metrics - how good are developers at security already? What if the bar is on the floor and LLM code generators are already better?
reply