I'm currently working on a project that is self-improving most of the time. Most of the plans for next steps are written by the agent itself, and executed by the agent itself, and the result feeds into choosing which plans to pursue next. It's not 100% autonomous yet, but self-improvement loops are real, and essential to getting the most out of AI.

AI currently lacks agency but if it can achieve greater goal setting and agency I can't see why self-improvement could not be achieved.

I think the most disappointing thing will be that even we do achieve ASI, everything will carry on as business as usual for a while before it starts making an economic impact because of how resistant to change we have made society.

This is something that I have been wondering about. SuperIntelligence or not, it's clear that significant change is going to happen.

There are a lot of people working on the cause of the change. There are a lot of people criticising the nature of the change. There are a lot of people rejecting the change.

How many are there preparing the world for the change?

Some form of change is coming, how are we preparing society to deal with what is happening?

Job losses due to technology have happened over and over again. Rendering particular forms of employment redundant (typing pools, clearing horse manure, Video rental store workers, and of course, the loom). Most agree that the world is better when those are jobs that need to be done. It's the livelihood of the workers that is the concern.

Instead of fighting the change we need to address the inevitability of change the responsibility to those who it will affect.