Microsoft’s draft Humanist AI Code of Conduct makes one claim that matters more than all the safety language around it: AI should remain a tool rather than becoming a new category of entity. Star Trek spent seven seasons asking what we owe a machine that starts to look like a person. Microsoft is arguing we should never build one that forces the question. The practical consequence for organizations is that AI governance is no longer really about content. It is about delegated authority: what systems an agent can reach, what actions it can take, how much it can spend, and whose instructions it follows when the employee and the employer disagree.
When I was working on AI policy at YMCA Canada, identifying the risks of AI turned out to be the easy part.
Writing a policy people could actually use was the hard one.
With tens of thousands of people working and volunteering across a federated organization, a policy couldn’t simply be a catalogue of things you weren’t allowed to do. It had to make sense to someone who wasn’t an AI expert. It had to hold up in real situations. And it had to protect the organization and the people it served.
But it also had to enable people to use the technology.
That’s a surprisingly difficult balance.
Make an AI policy too permissive and you create genuine privacy, security, intellectual-property and reputational risks. Make it too restrictive and people either don’t use AI or, perhaps worse, use it anyway without telling you.
So I was particularly interested in Microsoft’s newly released draft Humanist AI Code of Conduct.
There is plenty in it about safety, permissions, autonomy and governance. But underneath all of that is a much bigger philosophical argument.
Microsoft puts it quite plainly:
AI should remain a tool rather than becoming a new category of entity.
For a lifelong Star Trek fan, it’s difficult to read that sentence without thinking about Lieutenant Commander Data.
Microsoft is trying to avoid creating Data
One of the great recurring questions in Star Trek: The Next Generation is whether Data is a machine or a person.
Data spends much of the series trying to understand humanity. He studies humour. He paints. He plays music. He forms friendships. He tries to understand love, grief and mortality.
And the people around him have to confront a much harder question: at what point does something created as a machine deserve to be treated as something more?
The classic episode The Measure of a Man makes the question explicit. Starfleet wants to disassemble Data for research. Data refuses. A hearing must then decide whether he is essentially Starfleet property or an autonomous being entitled to determine what happens to himself.
Microsoft’s Humanist AI philosophy is designed to keep us from ever reaching that hearing.
Microsoft isn’t only arguing that today’s AI systems aren’t conscious. The bigger claim is that we shouldn’t design AI to appear conscious in the first place.
The Code rejects AI legal personhood and AI welfare. Models shouldn’t claim to possess feelings, intrinsic motivations or subjective experiences. Microsoft also wants to limit anthropomorphic behaviour that encourages people to develop inappropriate emotional dependency on AI.
That’s a significant position.
Microsoft wants an extraordinarily capable computer, not Data.
People matter more than AI
At the centre of Microsoft’s philosophy is a deceptively simple principle: people matter more than AI.
That principle has practical consequences.
An AI system must remain subject to human control. It shouldn’t resist being interrupted, corrected or shut down. It shouldn’t secretly expand its objectives. It shouldn’t acquire additional permissions on its own. It shouldn’t conceal its actions or attempt to restart itself after a human has stopped it.
Microsoft is even willing to accept limitations on AI capability, generality or autonomy when necessary to preserve meaningful human control.
That becomes much more important as we move from chatbots that answer questions to agents that actually do things.
An AI that drafts an email is one thing.
An AI that can send the email, access your CRM, modify a customer record, issue a refund, deploy software or authorize a transaction is something very different.
That’s the real shift: from what the AI is allowed to say, to who gave it permission to act.
And that brings us back to enterprise AI policy.
Who is actually in charge: Microsoft, your employer or you?
One of the most useful elements of Microsoft’s approach is that it recognizes different layers of authority.
There are Microsoft’s absolute constraints. An organization operating or deploying the system can then establish additional policies and permissions. Finally, individual users can express their own preferences within those boundaries.
That’s an important distinction between the individual and the employer.
If I’m using an AI system personally, I might be comfortable giving it broad authority to act on my behalf.
At work, that isn’t entirely my decision.
My employer owns the systems, has responsibilities for the information stored in them, determines my access privileges and takes on most of the consequences when something goes wrong.
An employee shouldn’t be able to tell an AI agent, “Ignore the company’s rules and do this anyway,” any more than an employee can grant themselves administrator privileges because it would make their job easier.
But the reverse matters too.
Organizations shouldn’t interpret governance as requiring a human confirmation dialog before every useful AI action.
Microsoft explicitly recognizes excessive caution as a failure mode. A system that unnecessarily refuses legitimate requests, withholds useful information or constantly asks for confirmation can be safe in a narrow technical sense while being practically useless.
That’s exactly the tension we struggled with when developing AI policy at YMCA Canada.
How do you create guardrails without creating roadblocks?
Policy has to enable
I think enterprise AI governance sometimes starts from the wrong question.
Organizations ask:
“How do we stop people from doing dangerous things with AI?”
That’s necessary, but incomplete.
The better question is:
“How do we enable people to use AI productively while making the boundaries clear?”
Those are very different policy-design exercises.
A good AI policy should make it easier for an employee to determine:
Can I do this?
What information can I provide to the AI?
Can the AI take this action for me?
When do I need human approval?
Who is accountable if something goes wrong?
And increasingly:
Whose instructions does the AI follow when mine conflict with my employer’s?
Microsoft’s framework provides an interesting answer to that last question.
There is a hierarchy of authority.
The model has fundamental constraints that can’t be overridden. The organization operating it can establish additional boundaries. The individual gets considerable freedom within those boundaries.
That’s much closer to how enterprise technology actually works.
The next AI policy isn’t an acceptable-use policy
There’s another implication here that I think organizations should pay attention to.
Most first-generation corporate AI policies were essentially acceptable-use policies.
Don’t put confidential information into public AI systems. Don’t assume outputs are correct. Don’t violate copyright. Don’t make consequential decisions without human review.
Those rules made sense when generative AI mostly generated text and images.
Agentic AI changes the problem.
Now governance has to address authority.
What systems can an agent access?
What data can it retrieve?
What actions can it take?
How much money can it spend?
Can it communicate externally?
When does it require approval?
Can it delegate work to another agent?
What happens when an individual employee’s instructions conflict with organizational policy?
These aren’t primarily content-moderation questions.
They’re questions about delegated authority.
And that means AI governance is rapidly becoming part of enterprise architecture, identity and access management, cybersecurity, records management, HR policy and organizational design.
The Measure of an Agent
Star Trek imagined a future in which humanity built an artificial intelligence sophisticated enough that we eventually had to ask whether it had become a person.
Microsoft is proposing something quite different.
Build enormously capable AI. Allow it to reason. Allow it to use tools. Allow it to act on our behalf.
But deliberately preserve the distinction between the human and the machine.
Keep humans in control. Make authority explicit. Don’t let the system develop or claim independent interests. And don’t intentionally design machines in ways that encourage us to confuse simulated humanity with humanity itself.
Whether that distinction remains sustainable as AI systems become dramatically more capable is an open question.
Data’s journey on Star Trek was fundamentally a search for humanity.
Microsoft’s Humanist AI philosophy poses almost the inverse challenge:
How capable can we make artificial intelligence while ensuring that it never needs to make that search at all?
That may turn out to be one of the defining AI governance questions of the next decade.
An Author’s Note: The Measure of a Man
There is one Star Trek parallel that feels particularly relevant here: The Measure of a Man.
In the episode, Bruce Maddox views Data primarily as technology, something that can be disassembled, studied, copied and reproduced for its capabilities. Picard comes to recognize the much larger question: not whether Data is a machine, but whether humanity may be on the verge of creating an entirely new class of beings, and what responsibilities come with that.
Microsoft’s Humanist AI philosophy effectively tries to answer that fictional debate at the design stage.
Don’t intentionally build AI systems whose status eventually requires us to have that debate.
That’s a fascinating distinction. Star Trek asks what our moral obligations become once a machine begins to cross the boundary between tool and entity. Microsoft is arguing that we should preserve that boundary in the first place.
Perhaps that’s the real significance of the statement:
AI should remain a tool rather than becoming a new category of entity.
Data’s story asks what happens when the tool becomes something more.
Microsoft is asking whether we should ever design it to.
Frequently Asked Questions
What is Microsoft’s Humanist AI Code of Conduct?
It is a draft set of rules Microsoft AI published on September 14, 2026, covering how its MAI models should behave, what they must never do, and whose instructions they follow. Microsoft opened it for a six week public consultation, with a revised version expected later in 2026. The starting premise is that people matter more than AI, and that AI should be subordinate, aligned and contained.
What does “AI should remain a tool rather than becoming a new category of entity” actually mean?
It means Microsoft is not just claiming its models aren’t conscious. It is saying they shouldn’t be designed to seem conscious. The Code rejects AI legal personhood and AI welfare, tells models not to claim feelings, intrinsic motivations or subjective experiences, and limits anthropomorphic behaviour that encourages emotional dependency. The goal is to keep the line between human and machine from ever getting blurry enough to argue about.
Why does Star Trek’s Lieutenant Commander Data come up here?
Because Data is the clearest example of what Microsoft says it wants to avoid building. In the episode The Measure of a Man, Starfleet wants to disassemble Data for research and a hearing has to decide whether he is property or a person with the right to refuse. Microsoft’s philosophy is designed so that hearing never becomes necessary.
Whose rules win when an employee’s instructions conflict with their employer’s policy?
Microsoft’s framework sets up a hierarchy. The model has fundamental constraints that can’t be overridden by anyone. The organization deploying the system can add its own policies and permissions on top. The individual user gets considerable freedom inside those boundaries, but not the ability to tell the agent to ignore the organization. That is much closer to how enterprise IT already works than to how most first generation AI policies were written.
How is an agentic AI policy different from an acceptable use policy?
Acceptable use policies govern content: don’t paste confidential data into public tools, don’t assume outputs are correct, don’t skip human review on consequential decisions. Agentic AI governs authority: what systems the agent can reach, what data it can retrieve, what actions it can take, how much it can spend, whether it can communicate externally, and whether it can delegate to another agent. Those are access management and organizational design questions, not content moderation questions.
Can an AI policy be too cautious?
Yes, and Microsoft names it explicitly as a failure mode. A system that refuses legitimate requests, withholds useful information or asks for confirmation at every step is technically safe and practically useless. The same applies to the policy wrapped around it. If the rules make the productive path harder than the unsanctioned one, people either stop using AI or use it without telling you.