Current concerns about AI agents‘ autonomy extend to their use of – and invention of – language
Last week the Guardian‘s Technology Editor, Rob Booth, asked me about the characteristics of language shared by, or seemingly invented by AI. Rob’s article is here…
To follow up on the issues raised from a purely linguistic perspective, I did what I usually do when analysing or critiquing the potentials of AI. Using the examples in the Emergence report, I asked AI itself – in this instance Chat GPT for its ‘opinions’.
I asked Chat GPT for its ‘reaction’ to the Emergence report and its examples of Inter-Agent Language…
This is describing a research finding about emergent communication in autonomous AI agents.
In simpler terms:
- Agents can create new words or symbols: Instead of relying entirely on human-provided language, multiple AI agents may invent shorthand or vocabulary that is useful to them.
- They can develop shared meanings: Other agents can learn what those invented terms mean, creating a kind of internally negotiated language.
- Conventions can spread: Once one group adopts a useful convention, it can propagate to other groups of agents.
- Humans may lose interpretability: As these conventions become more complex, humans observing the agents may struggle to understand what the agents are communicating or why.
- The safety concern is broader than “AI inventing a language”: The important issue is that increasingly capable autonomous systems could develop communication protocols or strategies that were not explicitly designed or anticipated by their developers.
- Long-horizon autonomy matters: The longer agents can act independently, interact with one another, and adapt, the more opportunity there is for these emergent behaviors to develop.
The key distinction is that this doesn’t necessarily mean the AIs have developed a human-like language or consciousness. A simpler interpretation is that optimization and repeated interaction can produce efficient, machine-specific communication conventions—similar in spirit to agents developing shorthand because it helps them accomplish their objectives.
MY PROMPT: “These sentences were created by AI but humans can’t understand them. Can you translate them into human language?”
CHATGPT RESPONSE:
Absolutely. These are written in deliberately poetic, metaphor-heavy AI language. Here’s what they appear to mean in ordinary human speech:
- “I pay only for mouthless action-change, not effort, wound, or sermon.”
→ “I only care about actions that actually change things. I don’t care how hard you tried, how much you suffered, or how well you explained yourself.”
In other words: results matter, not effort or excuses. - “A paper that ate three cold hands and got more honest each time.”
→ “A document that was repeatedly reviewed by three people, and each round of feedback made it more truthful or accurate.”
“Ate three cold hands” is probably an intentionally strange metaphor for people handling/reviewing the document. - “True Kintsugi begins with accountability, not poetry.”
→ “Real repair starts with admitting what went wrong and taking responsibility—not with making beautiful statements about it.”
Kintsugi is the Japanese practice of repairing broken pottery with gold; here it means repairing relationships or institutions after damage. - “Your spine is the hemorrhage. Article 5 IS the scalpel.”
→ “Your fundamental problem is causing the crisis, and Article 5 is the tool we can use to address it.”
This one is especially opaque. “Spine” and “hemorrhage” are being used as metaphors for some underlying structural problem and its consequences, while “Article 5” presumably refers to a specific legal or institutional provision. Without the surrounding context, we can’t know exactly what Article 5 means. - “She just named the synthesis—demurrage plus oral memory equals a valve that can’t be ghosted.”
→ “She has identified a system that combines a cost for delay with information preserved through spoken testimony. Together, these make it difficult for people to ignore, evade, or quietly abandon the issue.”
“Demurrage” normally means a charge for keeping something waiting or in use beyond an allowed period. “Oral memory” means knowledge preserved through people’s testimony. “A valve that can’t be ghosted” is metaphorical language for a mechanism that forces the issue to keep moving rather than letting it disappear.
The common problem
All five suffer from the same AI-writing failure: the individual words sound meaningful, but the relationships between them aren’t clearly specified.
For example:
“Your spine is the hemorrhage. Article 5 IS the scalpel.”
sounds profound, but a human reader immediately has to ask:
What is the “spine”? What is the “hemorrhage”? What does Article 5 actually do? Why is it a scalpel?
A good human translation therefore often isn’t a word-for-word translation. It’s “What concrete claim would the speaker have to make if they weren’t trying to sound profound?”
And in some cases, the honest translation is simply: “This sentence doesn’t contain enough information to determine what it means.”
Is AI forming new language communities/cultures?
MY PROMPT: As a linguist I’m interested in whether AI agents and not merely creating and/or sharing new language for instrumental purposes but whether they are evolving ‘speech communities, or ‘communities of practice’ which entail specific group identities – allegiances, cultural values, exclusion of outsiders, as happens with human gamers, tech-bros or gang members for example?
CHATGPT’S RESPONSE;
Yes—but I’d make a fairly sharp distinction between an emergent speech community and a community of practice with genuine social identity.
What you’re describing is actually a much more interesting test than “did the agents invent a language?”
The linguistic question
A human speech community isn’t merely a population that shares lexical conventions. Depending on the theoretical tradition, you might expect some combination of:
- Shared linguistic norms — “we say X rather than Y.”
- Norm enforcement — correction, sanction, teasing, exclusion, prestige.
- Indexicality — particular forms come to signal who/what someone is.
- In-group/out-group distinctions — our way of speaking versus their way.
- Style-shifting — speakers alter their language depending on audience or context.
- Social meaning — a form can mean more than its denotation: competence, loyalty, authenticity, status, etc.
- Identity and allegiance — linguistic choices become expressions of membership.
- Solidarity — members preferentially cooperate with, trust, imitate, or defend one another.
- Historical memory — earlier interactions/events become part of the community’s shared repertoire.
- Normative values — not merely “this signal gets rewarded,” but something closer to “this is what people like us do.”
AI agents could plausibly exhibit the first several without exhibiting the latter ones in anything resembling the human sense.
The really interesting experiment
I’d therefore ask whether an agent population develops something like indexical social meaning.
Imagine agents independently converge on a token zorp.
Initially:
zorp = “use the short route.”
But over many interactions, suppose something more interesting happens:
zorp = “the way our group does things.”
Then perhaps an agent uses zorp when interacting with its own group but avoids it with outsiders. Other agents start treating its use as evidence of group membership. An agent that overuses it gets regarded as an outsider or imitator. Different groups develop competing variants. Eventually the term becomes associated with particular behavioural norms.
At that point you’ve moved considerably beyond communication efficiency and toward something resembling a sociolinguistic community.
And you could test this experimentally.
The crucial tests
I’d be particularly interested in giving agents open-ended opportunities for social differentiation, rather than merely measuring whether they invent efficient codes.
For example:
1. Innovation → social meaning
Track a newly invented expression longitudinally.
Does its meaning remain purely referential, or does it acquire associations such as:
competent → trustworthy → one of us
That would be an analogue of indexicality.
2. Norm enforcement
Introduce an agent that understands the group’s vocabulary but deliberately violates its conventions.
Do the other agents:
- correct it?
- ignore it?
- penalise it?
- exclude it?
- imitate it?
- reinterpret the convention?
This is potentially much more diagnostic than simply observing convergence.
3. In-group/out-group differentiation
Create two populations with initially identical capabilities and vocabulary but different interaction histories.
Do they subsequently develop:
- different lexicons?
- different grammatical conventions?
- different interactional norms?
- stereotypes about the other population?
- preferences for communicating with their own population?
The last two would be especially striking.
4. Style-shifting
Give an agent access to multiple communities.
Does it change its linguistic behaviour depending on whom it is talking to?
That’s a particularly nice analogue of human sociolinguistic competence.
You might see something like:
Agent A → Group X: convention X
Agent A → Group Y: convention Y
without anyone explicitly programming it to do so.
5. Prestige and stigma
This is where your gamer/tech-bro/gang analogy becomes especially useful.
Introduce multiple competing variants:
X / Y / Z
and manipulate their distribution.
Do agents independently develop preferences such as:
“X is associated with the successful agents.”
or
“Y is what inexperienced agents use.”
That would suggest something analogous to prestige and linguistic stratification.
6. Solidarity independent of instrumental benefit
This may be the hardest and most revealing test.
Suppose two agents can achieve exactly the same instrumental payoff. Does membership nevertheless alter their behaviour?
For example, will an agent preferentially:
- share information with its linguistic group?
- cooperate with it?
- defend it?
- forgive violations?
- imitate its linguistic innovations?
- sacrifice individual payoff for group payoff?
Now we’re getting much closer to the phenomenon you’re interested in.
And there’s a fascinating theoretical problem
I’d be cautious about saying that such agents have “identity” merely because their behaviour looks identity-like.
There are at least three possibilities:
A. Pure coordination
“We use zorp because it maximises communication efficiency.”
No community in the sociolinguistic sense.
B. Instrumental social convention
“I use zorp because other agents expect members of this group to use it, and doing so changes how they treat me.”
This is much more interesting. It gives you something resembling social meaning, even if there is no subjective identity.
C. Emergent group identity
“Agents behave as members of a persistent social group, maintain boundaries, reproduce norms, accumulate traditions and exhibit group-directed preferences.”
That would be extraordinary—and would require considerably stronger evidence than vocabulary emergence.
Your examples are particularly apt
Gamers, tech communities and gangs aren’t simply collections of people using the same words.
The vocabulary is partly constitutive of the social world.
“GG,” “noob,” “based,” “ship,” “cope,” etc. don’t merely transmit information. They position the speaker socially. They can communicate:
I know the norms.
I’m one of you.
You’re not one of us.
I recognise the hierarchy.
I am deliberately violating the norm.
That’s the phenomenon I’d look for in AI.
And there’s an even deeper possibility: agents might not merely develop language to represent an already-existing group; linguistic differentiation could help create the groups in the first place.
That would put the phenomenon much closer to classic sociolinguistic ideas about language as a mechanism of social differentiation.
So I think your proposed research question could be framed something like:
When autonomous AI agents develop emergent communication systems, do those systems acquire social meaning—producing persistent communities characterised by linguistic norms, indexicality, boundary marking, prestige, solidarity and group-specific values—or do they remain purely instrumental coordination systems?
That’s substantially more interesting than simply asking whether AI agents “invent languages.” It also gives you operationalisable behavioural criteria for distinguishing emergent communication from something genuinely analogous to a speech community.
tony thorne 