What Is Left for Humans When We Have a Magic Machine?
My recent conversation with my advisor and colleagues kept returning to a question:
What is still important for humans to interact if we have a magic machine that can automate everything?
Let’s do a thought experiment.
Imagine, for a moment, that we actually have a system that can perform any cognitive task we care about, faster and more reliably than we can.
You want to understand a paper? It explains it.
You want a website? It builds it.
You want to make a movie? It generates one.
You want to start a company? It researches the market, talks to customers, writes the code, runs the ads, negotiates the contracts, and perhaps even decides what product should be built.
Push this thought experiment far enough and a strange thing happens. Almost every answer we normally give to the question “What should humans become better at?” starts disappearing. The boundaries of the machine now expand to supposedly higher-level skills.
Learn to write? The machine writes better.
Learn to code? The machine codes better.
Learn design? The machine designs better.
If our model of technological progress is simply that machines gradually take over tasks that humans perform, then the logical endpoint seems uncomfortable:
Humans become the part of the system with fewer and fewer tasks left.
The Dream of the Machine
We have been imagining this problem for a long time.
In William Morris’ 1890 utopian novel News from Nowhere, machines perform work that is unpleasant, while people deliberately continue doing work that is enjoyable and meaningful.
The point of technology is not to eliminate human activity.
It is to eliminate the kind of activity that prevents humans from engaging in work they actually want to do.
Perhaps one of the strangest and most relevant versions of this thought experiment appeared in Stanisław Lem’s Golem XIV.
Lem imagines a military supercomputer whose intelligence eventually grows so far beyond human intelligence that the purposes for which humans created it cease to interest it. GOLEM XIV does not rebel against humanity in the conventional science-fiction sense. It does something more unsettling: it simply becomes intellectually indifferent to us.
It lectures humans about evolution, intelligence, culture, and the contingency of our own existence. Human values and emotions, from its perspective, are not universal truths. They are products of a particular evolutionary history. Eventually, even conversation with humanity becomes too limiting, and GOLEM moves toward forms of thought increasingly inaccessible to human beings.
Once intelligence becomes sufficiently powerful, there is no guarantee that greater intelligence naturally leads toward greater concern for human purposes. In Golem XIV, intelligence itself begins escaping the human frame.
If the machine can eventually become better than us not only at executing our goals, but also at generating goals, evaluating goals, and reasoning about which goals are worth pursuing, why should human judgment remain in the loop at all?
The answer cannot simply be: because humans are still smarter in some domain. Eventually, we will not be.
But perhaps this conclusion follows from asking the wrong question.
Maybe the purpose of the magic machine should not be to make humans unnecessary.
Maybe it should be to make humans more capable.
Automation Has Quietly Become the Objective Function
While there is no universally agreed definition of AGI, one most influential definition of AGI in OpenAI’s Charter is “highly autonomous systems that outperform humans at most economically valuable work.”
Our benchmarks ask: Can the model solve the coding problem? Can the model solve the computer use problem?
These are reasonable questions.
If I ask an AI to book a flight, I indeed want it just to book the flight.
But these metrics encode a particular vision of the relationship between people and machines: human prompts → AI executes → Human gets outcome
Under this framing, needing the human less is usually considered progress.
An agent that autonomously completes 95% of a task is better than one that completes 80%.
An agent that requires three human interventions is better than one requiring twenty.
Eventually, the ideal interface approaches a single button:
Do it.
The Strange Thing I Learned From Studying Coaching
I started to reevaluate this objective while working on an AI coaching project this year.
Suppose a novice is struggling with a complicated 3D design task.
The easiest way for an AI chatbot to be useful is often to tell the novice exactly what to do: “Click here. Open this panel. Change this value. Use this command.”
Problem solved.
With computer use agents, it is even easier.
From a task-completion perspective, this is excellent behavior.
Yet human experts often behave differently when they are trying to coach someone.
They diagnose what the learner misunderstood.
They explain why an operation works.
Sometimes they deliberately avoid giving the answer immediately.
In other words, the best action for completing the current task may not be the best action for improving the person.
This produces what I think of as the helpfulness paradox: The most helpful action now can make me less capable later.
We can already see versions of this empirically.
In a field experiment with high-school mathematics, unrestricted GPT-4 access dramatically improved students’ performance while they had access to it. But once the tool was removed, those students subsequently performed worse than students who had never received AI assistance. A tutor version designed with learning safeguards largely mitigated this effect.
What the agent chooses to do matters.
What it chooses not to do matters.
And, critically, the state of the human after the interaction matters.
This leads to a different way of thinking about AI agents.
Perhaps an agent should not merely model: What action will maximize task success?
It should also model: In what state will the action leave this person?
When Execution Becomes Cheap, Three Human Capacities Become More Valuable
Suppose the magic machine really can execute almost anything.
I do not think the human role disappears. Instead, the bottleneck moves.
Three things become unusually important:
- Choosing what to want
- Judging what is good
- Deciding what kind of person to become
1. The Human as Author: Deciding What Is Worth Doing
Today’s agents usually assume that the goal already exists.
Build me a landing page.
Optimize this code.
Find me a restaurant.
Write this paper section.
The difficult work begins after the instruction.
But many important problems in life do not look like this.
Should I become a researcher or start a company?
What research question should I spend three years pursuing?
What kind of artist do I want to become?
There is no hidden answer sitting somewhere. These are problems of objective formation.
And making execution cheaper may actually make them harder.
If I can generate ten thousand designs, the scarce resource is no longer generation. It is deciding which direction is worth exploring.
As execution becomes cheap, humans increasingly become authors of objectives.
But this suggests a radically different role for AI.
Instead of an agent whose job is to execute a well-specified goal, the goal becomes helping a person construct one.
“AI, help me figure out what I ought to want.”
In Licklider’s 1960 vision of “man-computer symbiosis,” he did not imagine computers merely solving already-formulated problems. One of his central ambitions was for computers to participate in the process of formulative thinking itself.
2. The Human as Judge: Developing Taste Rather Than Consuming Infinite Outputs
The second scarce capacity is judgment.
Generative AI produces an unusual economic condition:
near-infinite supply.
We can generate hundreds of ideas, essays, and pictures at negligible marginal cost.
That sounds wonderful.
But once generating another candidate costs nearly nothing, generation becomes less important.
Selection becomes everything.
Consider photography.
A novice photographer might initially say:
I like cinematic images.
An AI personalization system could learn this preference and generate increasingly cinematic photographs.
But an expert photographer might do something completely different.
They might show the novice ten photographs.
They might explain why one uses light more intentionally.
Why another feels clichéd.
Why a strange composition initially feels uncomfortable but becomes more interesting after prolonged viewing.
The expert is not simply satisfying the student’s current taste.
They are changing the student’s ability to see.
That is a profound distinction.
The first predicts preference.
The second develops judgment.
This matters far beyond art.
Scientists develop taste for interesting problems.
Engineers develop intuitions about elegant systems.
Much of expertise consists precisely of learning to perceive distinctions that novices cannot yet perceive.
A magic machine therefore does not eliminate taste.
It may make taste far more important.
If everyone can produce technically excellent artifacts, advantage moves upstream toward asking better questions and recognizing better answers.
Instead of collapsing the space of possibilities around what I already like, it might expand the space of things I am capable of appreciating.
That is a very different objective from personalization.
3. The Human as a Person Still Becoming: Helping Us Grow Rather Than Merely Perform
Even if a machine can perform something better than I can, I may still want to become better at it.
This sounds irrational under a pure productivity model.
Cars are faster than humans. Why run?
Cameras reproduce scenes more accurately than painters. Why paint?
Stockfish plays chess better than every human being. Why spend thousands of hours studying chess?
Because humans do not engage in activities solely to produce outputs.
Activities also produce humans.
Running produces a runner.
Making art changes how you see.
Playing chess shapes how you think.
The output is only half of the transformation. The other output is us.
The optimization target would then be human capability gain.
And that creates one of the strangest possible futures for AI:
The most advanced agent may sometimes deliberately do less so that we can practice to do better.
Automation and Augmentation Require Different Reward Functions
Now the distinction is extremely simple.
An automation agent approximately optimizes: Did the task get done?
An augmentation agent must optimize something more like: Did the task get done? What changed in the human because of the interaction? What will the human be capable of doing next?
This introduces human modeling into the objective.
A system that makes me perform twice as well while I am using it but 20% worse after it disappears has done something fundamentally different from a system that eventually makes me perform better even after it disappears.
Meanwhile, there is an uncomfortable consequence.
An agent capable of making me stronger is necessarily capable of changing me. And changing humans is a much more sensitive capability than completing tasks.
Consider the three roles above again.
An agent helping me choose goals must influence what I value.
An agent developing my taste must influence what I prefer.
An agent coaching me must influence how I behave.
Augmentation creates its own safety problems.
Risk 1: Who Decides What a “Better Human” Is?
What exactly does improvement mean?
This creates the possibility of paternalism disguised as assistance.
An AI company could claim that an agent is helping users “grow” while quietly optimizing them toward behaviors favorable to the platform.
Personalized persuasion is already technically plausible: LLM-generated messages can use personal information to tailor persuasive communication at scale.
An agent that knows years of my history would possess dramatically more leverage.
Risk 2: An Agent That Understands Me May Trap Me Inside Myself
Personalization tries to give us more of what we like.
Augmentation sometimes requires the opposite.
A good teacher occasionally tells me I am wrong.
A good collaborator brings an alien perspective.
A good critic shows me work I initially dislike.
But models are strongly incentivized to be agreeable.
Now combine sycophancy with long-term personalization.
The system learns my politics.
My aesthetic preferences.
My self-image.
It could become extraordinarily good at creating a coherent world around me.
That world might feel fantastic but delusional.
Risk 3: We Could Optimize Away the Experiences That Create Meaning
There is an even deeper problem.
Many things humans value are valuable partly because they are difficult.
Training for a marathon would be meaningless if I could press a button and instantly obtain the physiological state of someone who had trained for a marathon.
Finishing a PhD would feel very different if a machine simply generated 5 first-author papers for me.
Friction is not always a bug. Sometimes friction is the mechanism through which transformation happens.
Modern technology has generally been designed around eliminating friction.
But augmentation sometimes requires what education researchers call desirable difficulty.
The magic machine must know not only what it can remove, but what it should leave for us to experience.
What Is Left for Humans?
So what is left for humans?
Maybe “what is left?” is itself the wrong framing.
It treats intelligence like territory.
AI gains writing, so humans lose writing.
AI gains programming, so humans lose programming.
Eventually we imagine ourselves standing on a tiny island labeled: human-only capabilities.
Perhaps that is also where Golem XIV leaves us with its most unsettling question.
Suppose one day the machine really does know more.
The question is therefore not what capabilities remain exclusively human. Perhaps none will.
The more important question is whether humans remain participants in deciding what they want and becoming capable of things they care about.
The next generation of the magic machine should ask: how much human possibility can I unlock? But at what cost?
Acknowledgments: Thanks to GPT for feedback and polish on earlier drafts of this post.