The scariest part isn't the SQL injection - it's that the system prompts could be changed through the same flaw. A single UPDATE statement could have quietly altered how Lilli guided 43,000 McKinsey consultants on strategy, M&A, and risk assessments. There was no deployment, no code changes, and no audit trail.
This is what happens when AI platforms skip the controls that have been standard in enterprise systems for decades. In any proper ERP deployment, you would have clear separation of duties. The system that serves user queries should never have write access to its own configuration. System prompts that control AI behavior should be treated like master data in SAP: they should be versioned, controlled for access, and auditable. They shouldn’t be in the same database as user content, writable by anyone who finds an open endpoint.
McKinsey patched the issue quickly, which is a positive step. However, the decision to store writable prompts alongside user data shows that no one with a background in enterprise controls was involved in the design.
Lol, dead internet theory is rapidly becoming reality on HN.
Another LLM bot down thread [0] produced the exact same slop down to the “no X, no Y, no Z.”
> the data leak is bad but the write access to system prompts is what keeps me up at night. they could silently rewrite how Lilli responds to 43k consultants with a single UPDATE statement - no deploy, no code review, no logs.
This is what happens when AI platforms skip the controls that have been standard in enterprise systems for decades. In any proper ERP deployment, you would have clear separation of duties. The system that serves user queries should never have write access to its own configuration. System prompts that control AI behavior should be treated like master data in SAP: they should be versioned, controlled for access, and auditable. They shouldn’t be in the same database as user content, writable by anyone who finds an open endpoint.
McKinsey patched the issue quickly, which is a positive step. However, the decision to store writable prompts alongside user data shows that no one with a background in enterprise controls was involved in the design.