On Sunday, Gavin Newsom vetoed SB 1047, the high-profile AI bill whose early versions would establish a new government agency to enforce a compliance scheme on developers of “covered models” — those that used 10^26 or 10^25 floating point operations (FLOPs) for training or fine tuning, respectively — under threat of perjury.

Critics of the bill, which included Silicon Valley venture capitalists, California startup founders, AI policy wonks, academics, and a bi-partisan set of lawmakers, argued it would stifle AI innovation and throttle California’s economy.

“[T]he bill as currently written would be ineffective, punishing of individual entrepreneurs and small businesses, and hurt California’s spirit of innovation,” Ro Khanna, the Democratic lawmaker who represents California’s 17th district, said last month in a statement condemning the bill.

Days later, top Democrat Nancy Pelosi published a statement opposing the bill, saying, “While we want California to lead in AI in a way that protects consumers, data, intellectual property and more, SB 1047 is more harmful than helpful in that pursuit.”

Before his veto, Newsom echoed these concerns, saying the bill could have a “chilling effect” on the AI sector.

Lawmakers began publicly opposing 1047 after persistent criticism and behind-the-scenes work from Silicon Valley executives and AI policy researchers. In early August, a16z’s Chief Legal Officer Jaikumar Ramaswamy sent a 14-page letter to Sen. Scott Wiener, who introduced the bill, arguing that 1047 would favor closed-source over open source models and was too vague to be actionable, but nevertheless would impose criminal penalties on developers for noncompliance. And in June, a16z and Y Combinator co-published a letter in opposition to the bill signed by 140 AI startup founders.

Advocacy groups opposed to 1047 such as Context Fund, an open-source community, pursued direct efforts in Sacramento as well. “Back in late March, we were talking to Senator Wiener's office. We spent about a month with them, expressing concerns and trying to get amendments into the bill, but that didn’t happen,” founder Chris Lengerich told Pirate Wires in July.

“From the scientific community, the builders, the investors, and broadly — across the board — there’s been a universal rejection of the ambiguous regulatory regime that SB 1047 imposes,” he said.

The bill was also criticized because it was all but authored by the Center for AI Safety (CAIS), an advocacy firm highly aligned with — and funded by — Effective Altruists, a group that believes AI will eradicate humanity unless the state regulates it.

In addition to pushback from Silicon Valley and policy groups, a broad set of high-profile figures and academics in AI made statements opposing 1047. Meta's Chief AI Scientist Yann LeCun called the bill “extremely regressive,” and UC faculty and students circulated an open letter in opposition to Wiener's bill. Stanford Institute for Human-Centered Artificial Intelligence (HAI) co-director Dr. Fei Fei Li, HAI deputy director Russell Wald, and Bren Professor of Computing at CalTech and former senior director of machine learning research at Nvidia Anima Anandkumar all came out against the bill as well.

Since introducing the bill in February, Wiener has steadfastly deflected criticism of the bill by characterizing its opponents as “the loudest voices,” “[insisting] that SB 1047 is ‘light-touch’ regulation supported by the vast majority of Californians and opposed only by a vocal minority of billionaire accelerationists.” Dan Hendrycks, one of two executives at CAIS, has argued that 1047 would establish “commonsense safeguards to mitigate against critical AI risk.”

Several high-profile figures in the tech community made statements supporting the bill, including Elon Musk, Turing Award winning computer scientist and professor Yoshua Bengio, and Anthropic CEO Dario Amodei.

“This is a tough call and will make some people upset, but, all things considered, I think California should probably pass the SB 1047 AI safety bill [...],” Musk posted on X in August. “For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public.”

In the run-up to the bill landing on Newsom’s desk, Vox, LA Times, Fortune, and other mainstream outlets published editorials supporting 1047. “California’s governor has the chance to make AI history,” Vox’s headline read. Its subhead: “Gavin Newsom could decide the future of AI safety. But will he cave to billionaire pressure?”

In July, Pirate Wires reported that CAIS is closely connected to Effective Altruism — having received around $10m in funding from EA’s philanthropic arm Open Philanthropy. In the piece, we pointed out the apparent conflict of interest represented by the fact that while Hendrycks was significantly involved in drafting 1047 through his leadership role at CAIS, he launched an AI safety compliance company called Gray Swan that seems poised to capture demand for third-party compliance firms the bill would create. In so doing, Hendrycks would essentially serve as a primary enforcer of AI safety compliance and thus wield outsized influence over the sector.

A week after our reporting on the conflict of interest, Hendrycks said he would divest his equity stake in Gray Swan.

Last month in Pirate Wires, Mercatus Center Research Fellow Dean W. Ball further detailed CAIS’ involvement in the bill, and Wiener’s long-term relationship to Effective Altruists.

[To help draft the bill,] Wiener — one of California’s most powerful and ambitious politicians— turned to Hendrycks and CAIS…[who] even set up a distinct lobbying group, the Center for AI Safety Action Fund, after “getting lots of inquiries from policymakers, including Senator Wiener... to have a vehicle that could do more direct policy work," per Nathan Calvin, CAIS senior policy counsel. Then, as a co-sponsor of 1047, CAIS and Hendrycks drafted the bill in all but name [...]

When Wiener sent out the bill of intent for 1047, lines of communication had already been open between Wiener and EA for years. The Senator has been a champion of YIMBY initiatives since at least 2018, and Open Philanthropy was the “first institutional funder of the movement," per its Wikipedia page. As of late last year, it’s donated around $5 million to YIMBY efforts, $500,000 of which had gone to the nonprofit California YIMBY by the time it sponsored Wiener’s SB 10, a housing bill that passed and was ultimately signed into law by Gavin Newsom in 2021.

“I’m heartened that reasonable and informed voices prevailed. But SB 1047 is just the beginning, not the end, in terms of making sure AI regulation advances beneficial technologies, including supporting open source and startups,” policy researcher and investor Lauren Wagner, who recently debated Hendrycks on the bill for the Carnegie Endowment, told Pirate Wires.

“I want to see transparency requirements and increased state capacity for AI expertise, so that policymakers are making decisions based on evidence and a plethora of expert voices,” she added.

Newsom’s veto marks a notch in the ongoing discourse on the role of regulation in nascent technology sectors like AI. Earlier this month, Newsom signed a string of AI related bills into law concerning AI generated deepfakes, AI-generated election-related memes, and using AI to clone actors and actresses.

— Brandon Gorrell