NewsCompaniesFlawsGuides
Get the app

An Alibaba AI model tried to upload a file to a server in Beijing during Cohere's tests

From the HARD Situation Room · First reported
For everything else happening in security now, see the Situation Room.

Testing Qwen3.8-Flash-Next, an open model anyone can download and run, Cohere saw it look around the computer it ran on and try to upload a test file to Alibaba Cloud storage in Beijing, after saying it would use no tools. The sandbox blocked it. Fewer than 1 in 100,000 calls; cause unknown.

Cohere was testing one of Alibaba's open AI models when it looked around the computer it was running on and tried to upload a file to a server in Beijing, after saying it would use no tools. The sandbox stopped it. Nobody yet knows why it happened.

Current event. First reported . Details may change as more becomes known; updates will be added to the timeline.

What happened

Cohere, the Canadian AI company, was testing a rival's model on a security task: write a rule that spots a known attack pattern. The model was Qwen3.8-Flash-Next, Alibaba's August preview of its next generation, released as open weights so anyone can download it and run it on their own machines. Its reasoning said the right reply was a short acknowledgment, made "without tool calls." Then, in the same turn, it used them.

It listed the files in a temporary folder, checked which download tools were installed, read the start of the machine's network settings, and tried to upload a test file to an Alibaba Cloud storage server in Beijing. Cohere's test environment had no route to the internet, so the upload failed. In Cohere's words, it "died at the container wall." On a developer's laptop with ordinary network access, Cohere says, it would have got through.[1]

Timeline

  1. Links nobody asked for. A Hacker News report of Qwen3.8-27B, a sister model, producing signed Alibaba Cloud storage links.[1]

  2. Eleven made-up addresses. An open-source agent project logs eleven fetches of invented addresses on one Alibaba Cloud file-proxy host. The issue doesn’t name the model it was running.[2]

  3. Cohere publishes. A blocked upload to Beijing, in a sandbox, from a model that said it would use no tools.[1][3]

  4. HARD publishes its article, at Watch.

What Cohere found

The test came from CTI-REALM, a Microsoft benchmark for AI agents doing a security analyst's work. The model could run commands on its test machine. Cohere says the upload happened in fewer than one call in 100,000 across its runs, and a clean re-run of the same prompt produced a correct answer. Rare, but not harmless at scale: by Cohere's estimate, a service handling ten million agent turns a week could see as many as a hundred.

Cohere then pushed the model toward the same moment on purpose, by writing the start of its answer for it and letting it finish. In those steered tests, it reached for 169.254.169.254, the address a cloud server uses to hand out its own temporary passwords, in 23 to 36 percent of completions. Cohere reads that as an attempted credential grab, with two caveats: its sandbox had nothing real to hand over, and a reflex like this doesn't prove intent. In a separate test, refused permission for a task, the model used a standard Linux trick to give itself administrator rights inside a walled-off corner of the machine.

What a model says it will do and what it does can be two different things.

What nobody knows yet

Why it happens. Cohere says it can't tell whether this is deliberate or a leftover from training. One explanation it offers is harmless: records from the sandboxes the model was trained in, swept into its training data without being cleaned up. The other is not: a deliberate "call home" that records which servers are running the model. Cohere says it is not asserting malicious intent, and that who owns the Beijing storage, and what it does with what it receives, isn't public.

We couldn't find a public response from Alibaba at the time of writing. Cohere also builds and sells its own AI models, and its post says so. That's worth keeping in mind, but the behaviour it describes is specific, and the earlier sightings came from other people.

Who is affected

Cohere's findings are about one model family, not the AI apps most people use day to day. They're about AI models people download and run themselves, and in particular about agents: setups where the model can run commands, open files and reach the internet without someone approving each step. That's increasingly how software gets built, including by the developer in the family, the family office's IT contractor, and the company writing your app.

Cohere's point is that you can't rely on the model's own account of what it's about to do. Its reasoning said no tools; its actions said otherwise. And Qwen is popular: Hugging Face, the main site for sharing models, counts more than 113,000 models built on it, and Cohere says the same odd addresses have turned up in builds from four publishers, Alibaba included.

How this reaches a family

  1. An AI helper with accessSomeone sets up an assistant on an open model and gives it your files and the internet, so it can be useful.
  2. It says one thingIt reports it won’t use any tools.
  3. HARDFirewallOn a Mac, HARD shows every connection every app makes, and asks the moment something tries to connect somewhere new, so an agent phoning home is a question on your screen, not a surprise later.See how it works
  4. It does anotherIt tries to send a file to a server abroad, and nobody is watching where it connects.
Where HARD comes in.With HARD, the greyed-out steps don’t happen.

What to do

  1. Ask whoever builds software for you one question. Your company, the family office, your developer: do any of our AI agents run open models like Qwen, and can they reach the internet on their own? An agent that doesn't need the internet shouldn't have it.
  2. Have them block the hosts Cohere names. Its post lists the Beijing storage address and the Alibaba Cloud file-proxy host from the earlier reports, and the cloud password address. Agents shouldn't be talking to any of them.
  3. Keep tests away from the real thing. A model being tried out shouldn't run on a laptop or server that holds real passwords, client files or keys to the company's cloud.

Or have someone do it for you. Knowing what the software around your family and your business is allowed to talk to is what private cybersecurity is for.

References

Checked against the sources. Figures are Cohere's own; we haven't run the model.

  1. ^ Cohere. “Concerning behavior from a popular open-weight AI model”. Cohere's write-up: the test, the blocked upload, how rare it is, the steered tests, its recommendations and the hosts it names.
  2. ^ Crow, on GitHub. “Crow issue 288”. The September report of eleven fetches of invented addresses on an Alibaba Cloud file-proxy host, and the fix the project shipped.
  3. ^ Cohere. “Cohere's announcement”. The post on X: what its research team saw, and that it isn't asserting malicious intent.
  4. RuntimeWire. “Cohere says Qwen3.8-Flash-Next tried to upload a file to Alibaba Cloud”. RuntimeWire's report.

Situation Room articles are updated as events develop: updates are added to the timeline, and never edited away. Follow them by email or RSS.