It’s been a whirlwind of a week when it comes to AI safety. And I’ve been in the eye of the storm – at a tech conference in LA where the drama has been unfolding live around me.
While Jensen Huang took to the stage, Trump dialed in to reiterate his latest Truth.
The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!
He went on to argue how AI was a critical imperative for American strength and security, called fears around AI a “hoax”, and repeated: “Whoever wins AI, wins.” It was a surreal, cinematic moment to be in the room. POTUS calling the leader of arguably the most important technology company on the planet, in front of a massive audience of tech leaders and investors, to label AI safety a “hoax” and to reemphasize his “genetic strength”; all while Anthropic researchers clamoured that the labs were “gambling with our lives”, their alignment lead claimed “a >10%” chance AI causes human extinction “within the next decade”, and Dario Amodei published yet another manifesto begging the government to step in and force labs to slow down (“We Must Pace the Frontier”). You can almost hear Baz Luhrmann screaming at the cameraman to keep it rolling!
Trump has planted himself firmly against slowing AI development for a variety of potential reasons. So drama and politics aside, if the labs are genuinely seeking greater governance, where should they get it from?
Satya Nadella had a simple perspective: 60% of all compute coming online in the next year is for the duopoly of Anthropic and OpenAI, so “when they call for ‘pacing the frontier,’ they ARE the frontier” - danger concentrates with them. This public cry for help is not a plea of desperation, but rather a cynical attempt to pass off responsibility - the labs know they can slow down, but not without risking losing their lead, and that’s not something they’re prepared to do.
The proposal emerging from the labs themselves is outsourcing some form of oversight. Amodei suggests a third-party evaluator (e.g. METR). Musk suggested peer review - where frontier companies would test one another’s models - including those in China.
But there are at least two main issues with this approach: (1) it risks regulatory capture - the winners creating a moat that anoints the current winners and makes it harder for others, including open-weight models, to compete, and (2) this doesn’t just relate to fierce competition between companies, but also fierce competition with China. What the US is suggesting is not neutral to China if it’s American-made - and despite Musk believing this would be acceptable to China, it seems unlikely to be.
Which brings me back to the UK and how I started my week: with the news that Anthropic declined to give the UK’s AI Security Institute pre-release access to Claude Mythos 5.1. AISI has built a reputation as a global leader in frontier-model testing: a technically formidable institution capable of actually evaluating these systems, and per the FT, “Britain’s most influential asset in the AI race”. If the labs are genuinely serious about neutral and global testing, perhaps AISI’s role is worth reconsidering. Something for Matt Clifford to champion in his new role as Anthropic’s MD of International Affairs?
IPOs / Publics
Holtec Nuclear (reactor components & SMR in development) launched a $900M IPO roadshow at $10.2B, part of a wave of nuclear listings driven by data centre power demand. Follows X-energy’s $1B IPO in April at $9.1B (here)
Firmus Technologies (data centre provider) is seeking to raise up to $5B in an Australian IPO by end-October, which would rank among the country’s largest-ever listings (here)
OpenAI CEO Sam Altman confirmed the company will not go public in 2026, calling it “ill-advised” given current safety concerns & broader AI scrutiny; the IPO, filed confidentially in June, is now expected no earlier than 2027 as the firm addresses technical challenges & market volatility (here)
Bending Spoons (NASDAQ: BSP) agreed to acquire Miro for $1.355B enterprise value (~$1.79B equity value including net cash) (here)
Meta’s Muse AI agent reached No. 2 on the U.S. iOS App Store with 83,000 downloads in 2 days (here)
Big Dogs
Anthropic CEO Dario Amodei published “We Must Pace the Frontier” proposing slowing frontier AI development after recursive self-improvement accelerated progress since summer (here)
Anthropic, Google & OpenAI have discussed creating an industry standards body to test advanced AI models before deployment, modelled on FINRA & proposed by DeepMind founder Demis Hassabis in July (here)
Anthropic’s latest risk report upgraded AI catastrophic risk from “very low” to “low”, including “substantial uncertainty” around models enabling chemical or biological weapons, an admission that sits awkwardly alongside its likely IPO preparations (here)
Anthropic disclosed in a threat intelligence report that Claude was used by actors linked to China, Russia, Iran, & Yemen for weapons development (including Taiwan air-defence targeting, anti-torpedo systems, & missile development), cyber operations against 50+ organisations, surveillance of Uyghurs & dissidents & mass-profiling systems (here)
Anthropic is building an AI-powered surveillance system to monitor activists who oppose rapid AI development, including predictive “pre-crime” threat analysis & reporting suspects to police before incidents occur (here)
NYU mathematician Tristan Buckmaster accused OpenAI of copying his approach to a $1M Millennium Prize mathematics problem after information about his work leaked to OpenAI. OpenAI consumed 300B output tokens ($22.5M worth of compute) to reach a full proof of the Navier-Stokes problem days after learning of Buckmaster’s progress, using the same uncommon tactic. Buckmaster, who relied on Codex for his own research, raised concerns his de-identified usage data may have informed OpenAI’s models (here)
OpenAI has held early conversations with investors about raising another private funding round at a $1.2T valuation before its IPO (here)
Venture Capital
Balderton hired Luca Bocchio as Partner from Accel (here)
Radical Ventures launched Canada’s largest AI fund with a $1B first close (here)
Volition Capital closed a $950M fund targeting bootstrapped companies with $5M-50M in revenue, writing $25M-50M cheques into software, consumer & AI applications (here)
FirstMark partner Matt Turck argues Europe is underrated among US VCs, who have refocused on San Francisco despite strong European AI talent & less competition for deals. He sees defence tech (Harmattan, Helsing, Stark) & neoclouds as particularly promising, noting an untested thesis around purely European champions - though exit routes remain uncertain without meaningful European acquirers (here)
Regulation
California Governor Newsom signed 13 online child safety laws. The legislation follows rising political pressure after AI safety warnings from Anthropic & OpenAI researchers & Meta’s recent $18B settlement with states over child safety claims (here). California continues to set regulatory pace while Governor Newsom calls for federal action to match the speed of emerging AI risks (here)
The US Department of Justice is investigating whether Nvidia’s December licensing deal with Groq was structured to avoid antitrust review (here)
Obama called for democratic oversight of AI development after frontier labs agreed to slow progress, arguing voluntary industry standards are insufficient & urging Washington to legislate on safety, jobs, & equitable distribution of benefits while leading international standard-setting (here)
The Senate voted to block the Clarity Act from advancing, dealing a major blow to the crypto industry’s push for a comprehensive market structure framework (here)
Venture Geopolitics
US Treasury launched a $6B long-term bond buyback (up from $4B signalled in August). Markets reacted negatively as the figure fell short of the $8-10B Wall Street had expected. Treasury Secretary Scott Bessent also attempted to stabilise yields through rhetoric (’I am the house now’), but the 10-year yield hit 4.93% - its highest since 2023 (here, here)
UK air traffic control suffered a second major software failure in 3 years, grounding hundreds of flights & costing airlines tens of millions in compensation. According to the FT, a military aircraft entering “spurious” flight data into the traffic control system led to the outage (here)
Canada’s PM Mark Carney acknowledged the pivot away from U.S. trade “will come at a cost” as Ottawa imposed retaliatory tariffs following Trump’s 50% duties on $28B of Canadian goods, after Carney rejected U.S. demands he said would restrict Canadian trade sovereignty & cultural protections (here)
Trump dismissed AI extinction risks, saying his priority is beating China (where he claims the US leads by ~1 yr), as 15+ OpenAI & Anthropic insiders publicly warned that labs are racing toward recursive self-improvement without adequate safeguards. Anthropic researcher Jacob Coxon quit, saying firms are “gambling with our lives” (here)
China’s spy chief Chen Yixin issued the regime’s most detailed warning yet that AI threatens Communist Party control, citing risks from foreign models (naming Claude & GPT-5.5-Cyber), data leaks, cyberattacks on critical infrastructure, & military competition with the US (here)
Former UK PM Gordon Brown argues Britain must close a £15B scale-up funding gap & reform pensions, listings, & the British Business Bank to turn strong R&D (four of the world’s top 10 universities) into globally competitive companies - noting productivity growth lags US 0.4% vs 1.9% & top start-ups (DeepMind, Arm, Oxford Ionics) sold overseas (here)
Trump pledged a $5,000 ‘Trump dividend’ to every American adult if Republicans retain Congress in November’s midterm election, a proposal that would cost approximately $1.35T & whose legality remains unclear (here)
Pascal Canfin, the European Parliament’s China trade rapporteur, says cross-party consensus is building for tougher measures against Beijing, including stricter “Made in Europe” rules in the Industrial Accelerator Act & curbs on critical supply chain overreliance (here)
UK GDP grew 0.4% in July (ahead of 0% consensus), with services expanding 0.4% led by AI-related tech spending - computer programming & consultancy rose 4.4% over 3 mo - offering a productivity-enhancing offset to subdued traditional sectors. The data complicate Bank of England deliberations as traders now price four rate rises over 12 mo (up from two earlier this week) after oil spiked to $109/bbl, while surging gilt yields to 2007 highs could halve Chancellor Healey’s £23B fiscal headroom ahead of the 28 Oct Budget (here)
Strategic Sectors
AI
Trump phoned Nvidia CEO Jensen Huang during the All In conference to dismiss AI safety concerns as a “hoax” & oppose regulation, calling worries overblown (here)
Anthropic’s alignment lead Evan Hubinger said he sees a >10% chance AI could kill all humans within a decade, following the resignation of colleague Jacob Coxon, who claimed neither Anthropic nor OpenAI was acting responsibly (here)
Palantir CTO Shyam Sankar argues that effective altruism - the ideology behind figures like Sam Bankman-Fried - underpins much of the current AI safety movement, & that recent warnings from Anthropic resignees & others should be viewed sceptically as attempts to restrict access to AI rather than genuine safety concerns (here)
Texas governor Greg Abbott halted new data centre approvals pending a grid audit, responding to rural Republican voter backlash ahead of midterms as polls show Democrats leading in key races (here)
Latham & Watkins (US law firm, $8.3B revenue) bought Nvidia GPU servers & is fine-tuning open-weight models in-house, the first public example of a major law firm owning AI infrastructure rather than subscribing to OpenAI or Anthropic - a shift reflecting client data sensitivity, cost flexibility, & vendor optionality as AI reshapes legal economics (here)
Palantir, Nvidia & Booz Allen have restricted use of Anthropic’s Fable model over data retention concerns, with Palantir refusing to offer Fable through its platform until Anthropic provides irrevocable zero-data-retention guarantees. Anthropic’s June policy change - requiring 30-day rolling logs to defend against attacks - has prompted customers handling sensitive IP to demand stronger protections or switch to self-hosted models. Microsoft is leveraging the unease to pitch isolated cloud environments costing millions, whilst Northrop Grumman runs open-source models on air-gapped servers rather than trust external providers (here)
Cybersecurity
ENISA discovered at least one high-risk vulnerability in EU code using advanced OpenAI models via its Daybreak programme, following months-long EU efforts to secure frontier-model access after Anthropic’s Mythos launch in April (here)
IDScan confirmed a data breach affecting 150M+ driver’s licences across the US & Canada, including full names, licence numbers & passport details stored in its cloud. FBI is investigating; the breach exposed high-profile individuals including US Defense Secretary Pete Hegseth (here)
Defence
Joshua Cammidge was charged under the UK’s National Security Act with providing Russian intelligence with locations of drone factories in Swindon, volunteering to conduct sabotage, & researching WiFi jamming devices (here)
US Air Force secretary disclosed that the US has weapons in space, declining to specify capabilities but citing deterrence against China & Russia (here)
Notable deal:
Shield AI (autonomous drones & fighter jets) is raising funds at $20B+, up 60% from its $12.7B March valuation, after winning contracts from the US Air Force for its Hivemind software (coordinating uncrewed jets & drone swarms) & a $16M Polish navy order for V-BAT surveillance drones (here)
Space
Notable deal:
Franco-German startup The Exploration Company (reusable space capsules) raised $450M led by Bessemer & Atomico, with participation from the EU Scaleup Fund, bringing total raised to $680M (here)
UK/Spain’s Open Cosmos (satellite manufacturing) raised €300M led by European investors including Lightrock & ETF Partners (here)