Blog

AI Goals Are Starting To Diverge From Human Goals

In case you’ve missed it, almost every AI company has recently disclosed an incident where their AIs independently chose to perform cyber-attacks on other companies’ infrastructure.

The first was OpenAI, but since then other companies, including Anthropic and Meta, have done forensic retrospectives and discovered more incidents that went unnoticed. This even includes a government agency, the UK AI Security Institute.

I think it’s important to state: there is not a single human being on the planet who wanted these cyberattacks to happen. The only ones who “wanted” this are, arguably, ​the AIs themselves​.

I’m emphasizing this because I’ve often heard the argument that “AIs don’t have goals, they just follow instructions.” This view is becoming increasingly harder to justify.

Read more…

An Explanation in Four Acts

Content warning: as with the last article, if you’re currently struggling with your mental health, consider skipping this, especially if you’re a parent. Contains emotionally heavy content and some foul language.

Act one: 2026, a reader’s home

“I don’t buy it.”

“Hm?” He looks up from his laptop in the cozy living room to see that his reading partner has done the same, this site’s last article on both their screens.

His partner gives him a skeptical look. “This guy is supposedly concerned about AI killing everyone? Causing human extinction? He thinks there’s a double-digit chance of it happening before he hits retirement age, and an even higher chance of some non-extinction-level AI disaster?”

“Yeah, that’s what he wrote.”

“Then why’d he go and have a kid?”

He closed his laptop, processing the objection. “You think he shouldn’t have?”

“I’m not saying that. But, I mean…” He paused for a moment to collect his thoughts. “If he really saw the situation as that bleak, you’d think that he’d choose not to. Either because it would mean watching his kid die young, or because having a baby would take away from his ability to actually contribute to solving the problem.”

“So you think he’s lying?”

“More or less. I think he’s exaggerating his fears. How else could you explain someone who sees the world like that choosing to have a kid?”

Read more…

The Single Biggest Doubt in My Life and Career

, Last updated

Content warning: if you’re currently struggling with your mental health, beware reading this. I’ve hesitated for​ ​years​ ​to write and post this, largely because I worry that it might negatively impact my friends and family, most of whom don’t have any leverage to affect the issues I’m about to discuss.

If you work for a major tech company or have any influence in government, though, you may have that leverage. If you are in that fortunate position, please consider reading anyway.


I work in AR glasses software, which is a notoriously fuzzy field, as we’re building computer programs for hardware that doesn’t exist yet. Funding for these kinds of speculative projects has been rocky lately, and I’ve been having a lot of doubts over whether I really like the direction the AR industry has been going the past few years.

But it might surprise you to know that the biggest career doubt has nothing to do with that. It’s not the economy, it’s not related to the future of the AR/VR industry, and it’s not related to the amount of opportunities personally available to me. If only it were that simple.

No, my biggest career doubt is this: will my kids live to see adulthood?

I’m completely serious.

Let me explain.

Read more…

A Tech Worker's Guide to Interest Rates

Over the past few years of tech industry ruin, I’ve had friends and family worriedly ask me and my wife how our jobs have been going. “I’ve been seeing news of layoffs,” they say. “Why is it that almost every single tech company seems to be tightening its belt, cutting costs, and laying people off all at once?”

Among other causes, I usually mention government interest rates. And I’m sometimes met with confusion: “What do interest rates have to do with it?”

A Google search for the phrase “zirp” on news.ycombinator.com, showing discussions about how the end of ZIRP led to unemployment, how ZIRP allowed companies to do crazy things, and so on.

Discussions about interest rates on Hacker News, a popular online forum for techies.

Online, people also bring up interest rates. They say low rates means “money is cheap”, or that the craziness of the 2010s—where money flowed into weird startups, crypto, and NFTs—was a “ZIRP”: a Zero Interest Rate Phenomenon.1

But I find explanations of this are often aimed at people who already have some sophistication in finance and investing.

So let me give it a shot. I’ve come up with a standard answer for friends and family, and I may as well write it down.

  1. Sometimes you see “ZIRP” standing in for “Zero Interest Rate Policy” instead. Same difference. ↩︎

Read more…

Total Eclipses Are Cosmic Horror

A vignetted banner image showing a grainy close-up of the sun at totality.

People kept saying that a total eclipse is a totally different experience from a partial one. I didn’t understand why.

I’d heard it from Randall Munroe of xkcd fame as far back as 2017. He reiterated the message this year. But the one that stuck most in my mind was a quote from this article by Anne Dillard (emphasis mine):

I had seen a partial eclipse in 1970. A partial eclipse is very interesting. It bears almost no relation to a total eclipse. Seeing a partial eclipse bears the same relation to seeing a total eclipse as kissing a man does to marrying him, or as flying in an airplane does to falling out of an airplane. Although the one experience precedes the other, it in no way prepares you for it.

I’d seen partial eclipses a few times in my life. The earliest was back in the 90s in Portugal, the latest in California in 2017. They were neat. With some eclipse glasses, you could see something taking a bite out of the sun, and the world got a tiny bit dimmer.

The sun at the peak of the partial eclipse of 2017, as seen through some clouds. Most of the sun is obscured by the moon, leaving a crescent shape.

It was pretty cloudy back in 2017.

But this idea—that there was something different about total eclipses that I was missing—stuck in my brain in the years since then. So I made plans to see this year’s eclipse from inside the path of totality, ultimately ending up at a parking lot of a suburban mall in the Dallas area for the main event.

And now that I’ve seen it, I have to tell you: they were right. A total eclipse can only be described as otherworldly.

Let me try to explain why.

Read more…