Some people think of it purely in terms of intelligence. Today, the models are smashing academic olympiads left and right, as well as completing longer tasks up to 10 hours at a time. To them, it might feel like we’re pretty close to AGI.

I went into university expecting to do an economics PhD, and switched to working on ML research due to some optimistic extrapolations of the GPT-3 paper, so my view of AGI is driven by its potential to be economically transformative.

From this perspective, the diffusion and impact of AI across the broad economy remains limited: you can see the AI age everywhere except in the productivity statistics.

Is there a capability overhang?

One explanation for this divergence is that products haven’t caught up to model intelligence. While there are valuable products which are yet to be created by scaffolding current models, this need for scaffolding reflects a deeper issue: people buy products for usefulness, not intelligence.

Usefulness requires agency, reliability, low cost and latency, alignment with user intent, integration with business processes, and much more. Along all of these dimensions, the models are simply not capable enough right now.

One direction which people have been betting on is using RL to bake these traits into the model, making it less reliant on scaffolding. RL seems to be very sample-efficient at getting domain-specific improvements, and certainly more cost-effective than pre-training.

Yet the majority of RL progress has been limited to a few domains like software engineering and competitive mathematics. Can we actually use it to accomplish economically valuable labour?

How do you apply RL to a real task?

The best way to understand this is to actually go after a real in-the-wild task, so I picked the problem of giving inline completions for financial data as you are typing.

This maps onto a decent chunk of work in finance, which is retrieving information when writing a memo or creating a deck. It is also an unsolved problem: there is no good inline autocomplete for writing, and certainly not one which retrieves financial data accurately. Large models are far too slow and expensive, while on-edge models struggle to chain tool calls, figure out when data is needed, and avoid making up numbers altogether.

To simulate this task, I populated a database full of 10-K and 10-Q data, synthetically generating many prompt-completion pairs using this data, as well as some prompts which did not require any completion.

Then I built an environment where a model would be given a prompt, as well as the ability to use a “search” tool to find the right information from the database and use a “return_answer” tool to either return the datapoint or an empty string for the no-completion cases. The answer would be judged on whether it was accurate compared to the ground truth pairs I had synthetically generated.

For evaluation, I held out some companies, time periods, metrics, and prompts, and as a baseline, I evaluated GPT-4.1, which achieved 80% accuracy with a p95 latency of 3.3 seconds. By contrast, Qwen 2.5 3B Instruct, which can be run locally on a laptop, achieved only 43% accuracy with a p95 latency of 2.6 seconds.

I then RL fine-tuned the Qwen model for a few hours on a single A100, and it went up to an accuracy rate of 93% while dropping its p95 latency to 1.1 seconds. In other words, it transformed the model into a genuinely useful product with frontier level performance at 3x the speed.

It seems like RL gives us a clear line-of-sight towards model improvements. If it’s such a free lunch, why isn’t everyone doing RL?

What are the footguns in applied RL?

Let me describe 4 lessons from doing this experiment, which make this less straightforward.

The first is the importance of obsessing over data quality: the “it” in the AI models is the dataset. For example, some of the financial metrics were getting saved with the wrong units, such that $9 billion was getting shown to the model as $9 billion billion. This is the kind of thing that can confuse a model, but is hard to notice without manually looking through lots and lots of rollout trajectories.

The second is that “model empathy” matters. Try and simulate what you would do if you had only the information it was given, and no additional context. This requires talking with the model a bunch, to calibrate on what the model knows.

For example, looking at the logs made me realise that the model was getting told that its tool calls were invalid, because it had used “selling, general & administrative” and “sg&a”, instead of “selling, general and administrative”. It’s just not very reasonable to expect a model to iterate through all the different ways of typing about a particular metric in order to figure out which is acceptable, so I made the environment more tolerant of different names for the same thing.

The third is that reward shaping is hard, reward hacking is always looming, and every step forward can feel like a step back.

Here’s an illustration:

• You start by giving the model only a reward signal depending on correctness.

• This is very sparse, so the model struggles to learn to reason about tool use.

• You add a reward signal on the intermediate steps e.g. rewards for using the “search” tool correctly, penalties for using “search” in a no-completion case or too many times.

• The model only returns “no completion needed”.

• You do some back-of-the-envelope-calculations and realise that with the relative weights of the rewards and penalties, the likelihood of it getting at least one wrong tool call, and the fraction of the dataset which is no-completion, the expected value of trying to answer is less than simply giving “no completion needed” every time.

• You re-adjust the weights, but notice that the model now brute forces by using many turns to try many different tool calls, so you impose a maximum number of turns.

• The model suddenly seems to be performing well, but the number of turns has gone up.

• You take a look and realise that the model is spamming non-ASCII characters in each turn and hitting the maximum turn limit. This leads the environment to return “No response” to the judge, which takes that to be correct in the no-completion cases.

In other words, you need to be paranoid every step of the way.

The fourth is a corollary of the three above, which is that tooling matters. One part of this is observability e.g. inspecting the data, seeing what the model is seeing, and tracking what is happening in the rollouts. A lot of the time spent babysitting a training run is spent on looking through metrics on wandb, and using them to diagnose if issues are arising e.g. right as the model went into the weird bit of the policy which involved spamming non-ASCII characters, both the KL divergence and the gradient norm spiked.

The other part of tooling is the infrastructure to make iterations faster. This can help with the more usual ML science e.g. sweeping hyperparameters, but it is especially useful for RL, since the speed of your rollouts can make a big difference to wall clock time due to a larger fraction of training being spent on inference.

What determines the right to win?

None of these pitfalls are individually intractable, but the reason they nonetheless deter many folks from applying RL is because they require the co-design of product and research.

Unlike pre-training, you aren’t trying to just improve capabilities broadly, but instead you are after a specific goal in RL. This means you need to know how the model fits into the end product, so that you can decide on what the right data and environment is, as well as what to optimise for and how to craft the right rubric.

Figuring this out looks a lot like what I spent time doing as a founder i.e. talking with customers, dealing with the enterprise sales cycle, and finding an internal champion at the company. In many cases, what is most helpful to getting a customer to conviction is being able to give them a working demo very quickly, which they can play with and use to advocate internally.

At the same time, your ability to ship quickly is so dependent on the research tooling. This project took me a bit over a week, but much of that was spent debugging infrastructure issues. Now that I have done it once, I can recycle it for the next project, and the one after that,

Very few teams are set up to ship at startup speed while also building a reusable research platform. One view from the full-stack AI startups and the private equity rollups is to run a “services-to-product” playbook that starts with owning a vertical and gradually building the tooling around it.

However, I’m actually more excited for a different approach, one where the frontier AI labs and RL-as-a-service companies leverage the infrastructure they have to rapidly deploy across domains. Unlike incumbents at many other points in history, these companies are still incredibly young.

They also have an advantage in cost structure, since they can amortise fixed research costs and bring down the average cost of each deployment. Their ownership of the research means they can integrate traits they care about into the training pipeline e.g. baking in certain general capabilities into pre-training to make RL easier or distilling smaller models to make inference cheaper.

If they can scale this co-design of product deployment with model research, we have a real shot at AGI, not just as a singular academic milestone, but as a diffuse economic achievement.

The launch video didn’t do the model any favours. Rather than focusing on GPT-4.5’s strengths, it highlighted how small the improvements were relative to GPT-4o and that it underperformed o3-mini, a 50x cheaper model.

Only a month and a half after the model’s release, they announced it was going to be deprecated.

Why did GPT‑4.5 disappoint?

When GPT-4 was released in 2023, it was seen as something fundamentally unprecedented. It blew past GPT-3.5 on the majority of benchmarks, and in some cases, saturated them entirely by reaching human-level performance.

GPT-4’s launch also came with a developer livestream full of feats that seemed unimaginable for GPT-3.5, such as converting a hand-drawn wireframe into a fully functioning website in a single request.

Where GPT-4 felt like a step change, GPT-4.5 felt like a letdown.

The biggest contributor to that was the many models released in between the two. Just from OpenAI, we saw updates to GPT-4, GPT-4 Turbo, the many iterations of GPT-4o, and the o-series of reasoning models. This made it hard to see the gains from purely increasing pre-training compute from GPT-4 to GPT-4.5 and begged the question: is pre-training a dead end?

Is Pre-Training A Dead End?

To answer this question, we need to know what happens when we only vary pre-training compute.

A natural experiment in scaling pre-training comes from Meta, who released 8 different text-only Llama 3 language models in 2024. The models range from 1B to 405B parameters, and since they came out over the course of only 8 months, we assume that there was relatively little compute efficiency improvement.

Based on these models, we can approximate a scaling relationship between the amount of pre-training compute and benchmark performance. This lets us extrapolate how GPT-4.5 should perform relative to GPT-4, since we know that it used 10x more pre-training compute than GPT-41.

What we find is that GPT-4.5 is on track with, if not beating, the pre-training trends2:

In other words, scaling still works. However, there’s a very good reason most frontier models have been smaller than GPT-43. It’s because researchers found ways to get better performance for far less money in other parts of the training stack.

Consider the models from the mid-2024 vintage, like GPT-4o and Claude Sonnet 3.6. They were 4 to 8 times smaller while still having comparable or better performance. This was driven by a range of techniques in mid-training and post-training:

• Annealing on higher quality, distributionally different or longer context data

• Instruction-tuning on synthetic data

• Preference-tuning on human and AI feedback e.g. reinforcement learning from human feedback (RLHF)

Meanwhile, the late 2024 / early 2025 vintage, like o1 and o3, showed staggering results by doing reinforcement learning (RL) on chains of thought (CoT).

The Future of Model Training

Scaling RL and CoT is a very effective strategy for getting improvements at specific domains. That’s why the trend in the immediate future will be continuing in that direction. However, there will come a point when the industry’s focus will return to pre-training.

This is because of the economic realities of serving models.

Firstly, if customers have a range of different use cases, it will require doing RL training on a very large set of domains. This is especially because RL in one area can degrade unrelated capabilities. Models that have been RLHF’ed for chat are worse at playing chess than peers which are closer to just pre-trained models, and anecdotally, Claude 3.7 Sonnet feels more reward-hacky than its less RL’ed predecessor. Doing lots of RL is expensive4.

Secondly, if customers are using many CoT tokens at inference, models will become very expensive to serve. It will eventually become cheaper to have a bigger and smarter model which requires fewer CoT tokens to get the answer5.

It would be far cheaper if there were a way to make the model both more intelligent across domains and also more intelligent in absolute terms, such that it requires fewer tokens to get to the same quality output at inference time. This is exactly what pre-training gets you6.

Thus, pre-training will eventually come back and the OOMs of compute will continue to grow. Pre-training isn’t dead, it’s just resting.

Thanks to Aidan, Sam and Sheryl for reading drafts of this.

In many cases, these labs have had shared investors: Sequoia Capital has invested in OpenAI, xAI, SSI, Reflection AI, Magic, Keen Technologies and Harmonic, while Andreessen Horowitz has invested in OpenAI, xAI, SSI, Character AI, Mistral AI, and Anysphere.

If you are investing in five or more AI labs, is that because you expect there to be an oligopoly where each oligopolist is a good investment, or do you simply lack conviction on who will win?

The former is a popular story, with a leaked 2023 Mistral memo claiming that “an oligopoly is shaping up”. At first glance, this tracks with API usage on OpenRouter:

One oligopoly story could be that different frontier AI labs are making different technical trade-offs, resulting in their own niche and specialties, and many of their fundraising announcements echo this sentiment:

• Magic’s 2024 fundraise focused on their 100m token context window

• Cohere’s Series C focused on enterprise use cases, like RAG and embeddings

• Mistral’s initial memo focused on privacy and domain-specific finetuning

• Adept’s launch focused on tool- and computer-use models

The proprietary trading industry

An industry that parallels this “many roads to Rome” quality is proprietary trading. Not only does it draw from a very similar talent pool as frontier AI labs, but it also sees strong differentiation between the few oligopolist players, due to the variety of asset classes and time horizons.

Secrets are all you need

Underpinning this differentiation is the incredible emphasis on secrecy. Beyond the “loose lips sink ships” culture that pervades the industry, all of these firms use non-competes that can be easily 3 years long. Even after you sit out this garden leave, you’ll still be bound by stringent confidentiality agreements.

These agreements are enforced litigiously. Perhaps the most famous and recent example is Jane Street suing Millennium, but this is not an isolated incident. Back in 2003, Millennium faced a similar lawsuit from Renaissance Technologies, and in 2014, G-Research used private prosecution to put one of their former researchers in prison for stealing their trading secrets.

This means these differentiated edges persist, helping firms preserve their market share in that asset class, even if they might otherwise struggle.

Flash boys: not so fast

One example of edge with this “secrets”-esque property of being difficult to replicate is the use of microwave tower networks to minimize trading latency.

Although its usage is well-known, optimal tower locations are severely limited, and there is only a limited amount of money to be made from being the fastest. Due to this steeply diminishing return, if you are a firm that has already invested heavily in this infrastructure, it might not be worth it for everyone else to engage in a red queen race to be faster than you.

Unsurprisingly, the industry has consolidated its investments. The microwave network in the Chicago-New Jersey corridor that connects America’s biggest exchanges boils down to three main groups:

• McKay Brothers, which is used by IMC, Tower Research, Citadel Securities, SIG, Jane Street etc.

• New Line Networks, which is run by Jump Trading and Virtu Financial

• Vigilant Global, which is run by DRW

Thus, firms with access to this infrastructure can keep the market share available just by being fastest, since it is not worthwhile for new entrants to go after that edge.

Meanwhile, new players can go after new edges. For example, XTX Research, which famously is opposed to the low latency game, managed to break out and become the largest marketmaker in European cash equities in the span of a few years, thanks to their superior pricing and investment into AI data centers.

The combination of these two effects – the defensibility of secrets and diminishing returns to exploiting secrets – means proprietary trading has tended to be oligopolistic, with a few firms dominating each asset class where they have some persisting comparative advantage.

The bitter lesson strikes again

On the surface, it may seem like the different API use cases in AI map on to the different asset classes in proprietary trading, and each AI lab could focus on getting really good at one of them. For example, the top models on OpenRouter vary a lot by use case. In the recent weeks, these have been:

• Gemini Flash 2.0: Roleplay, marketing, translation, legal, trivia, academia

• Claude 3.7 Sonnet: Programming, SEO

• GPT-4o-mini: Technology, science

• DeepSeek V3: Finance

• Gemma 3 4B: Health

However, the broader trend of overall usage makes this far less compelling, and suggests models have very limited sticking power. On average, a model only stays in the top 5 most used models for just over 6 weeks. This is drawn up by a few really long-lasting models, with the median landing at a mere 3 weeks. This is because API users tend to be very sensitive to performance.

So far, performance in AI has been defined by the Bitter Lesson: that “leveraging of computation” is more effective than leveraging “human knowledge of the domain”. Thus, performance improvements have not been isolated to specific niches, and instead depend on how much money is deployed on compute.

This means that secrets are much less important, and even if they were, the regulatory prohibition on non-competes in California, coupled with its open tech culture, means secrets do not stay secret for very long.

Scaling up to 2028

To put a number on compute, the jump between GPT-4 and GPT-4.5 over the past 2 years has involved increasing effective compute by two orders of magnitude, of which half came from algorithmic progress and half came from more physical compute. If we assume that algorithmic progress continues at its current rate and want to maintain the same overall rate of scaling, we’d need to increase physical compute by another 1.5 orders of magnitude from today to 20281.

Since GPT-4 took around $63M to train, the model in 2028 should take 300 times that, which is around $19B. A naive estimate for converting the training run costs into the capex spending required for the datacenter is to assume that the cost of the datacenter gets paid back by the total value of training runs you can do across its operational lifetime. Assuming a 4 year depreciation schedule, the GPT-4 training run which took 3 months (and 25000 A100s) implies a 16x multiple, giving a total cost of $1B.

With A100s going for $10k a pop, this gives a 4x multiple between the cost of the raw GPUs and the cost of the datacenter. As a sense-check, xAI's $700M Atlanta datacenter had around 12000 H100s and 370 A100s. H100s are around triple the price of A100s, leaving a GPU cost of $364M and a 2x multiple.

These bounds imply that a 2028 training run would require an upfront datacenter capex of anywhere between $152B and $304B. What are the market dynamics which this implies?

Incumbency in a capital intensive market

One analogy which has this similarly ramping capital intensity, and happens to be AI’s most critical upstream dependency, is chipmaking.

If we take a look at the CPU market, Intel became the dominant market player after IBM selected its x86 CPUs for the original IBM PC. Within 10 years, it had reached 80-90% market share. AMD would claw its way to 25% market share in the mid 2000s, drop back into irrelevance by 2015 and start gaining market share again in the late 2010s.

What this shows is that while a single generation of technical edge can help you grab market share, the economies of scale needed to amortize good research still gives the incumbent a lot of momentum. Thus the brief period of weakness in the mid 2000s was not enough, and it took a decade of Intel stagnation to truly break its monopoly.

High capex drives consolidation

By default, this means that you should expect consolidation in these high capital intensity industries. This is exactly what you see in the foundry market too.

Moore’s second law states: the cost of a semiconductor chip fabrication plant doubles every four years. Although this law started to weaken in the 1990s, the cost of building a single advanced fab today is still 1,000x higher than in the early 1970s2.

As the cost of staying at the frontier rises, fewer firms can afford to compete. In the early 2000s, 26 foundries produced 130 nm nodes, but in 2020, only 3 foundries produced 7 nm nodes. Now only TSMC is at the frontier and accounts for around 67% of global market share.

Consolidation if scaling laws hold

If we look at the main funders of frontier AI labs, they're overwhelmingly big tech companies. Their capacity to finance these efforts can be roughly estimated from free cash flow and debt capacity (assuming a debt-to-EBITDA of 3):

The projected 2028 capex estimate of $152B to $304B presents significant financial challenges, even for big tech companies.

Consider Microsoft: the lower estimate of $152B would consume their entire cash reserves plus a full year's free cash flow (FCF). The upper estimate of $304B would require either all cash on hand plus four years of accumulated FCF, or alternatively, using up all their debt capacity.

Continuing the patronage for AI labs is becoming increasingly costly. If you believe that a handful of frontier labs will make it and don’t believe the oligopoly story, you are implying that all big tech companies will leverage their entire business just to sustain a handful of frontier AI labs. That’s not a straightforward bet to make.

Consumer

So far we’ve focused on the API market. The other half of frontier AI usage, and indeed the breakout product of ChatGPT, has been in consumer chatbots. This sub-industry looks even worse for oligopolies, since there is even less differentiation between the chatbots of labs.

Instead, consumers tend to stay with the company they know best. Right now, that’s OpenAI, and that’s what we see when we take the number of reviews in the app stores as a proxy for market share.

Search is a consumer monopoly

This is not surprising. ChatGPT shares many characteristics with one of the clearest consumer monopolies today: web search.

User behaviour is sticky and habitual. Importantly, consumers don’t query the same thing across five search engines, but pick a search engine and stay for the long run. They do not have multi-vendor preferences.

It is then not surprising that there have only been two search monopolies thus far: Yahoo and Google. The Google monopoly formed because it was obvious to any consumer that Google was superior and they were already used to seeing Google results from the time where Yahoo used Google for search3.

For AI, that means that if a model company manages to build up a persistent lead that is so obvious to consumers on first try that you don’t need to rely on benchmarks, there is a chance of it taking the monopolist’s position.

The model is the market structure

We can see that there are many different narratives for what happens to the market for frontier AI labs. These can be mapped onto different expectations about model capabilities.

1. If model capabilities stagnate at their current level, mostly products will be the ones which capture value, since open source models will commoditize the AI labs. This means that the value goes to the product and distribution, with the most value captured by a consumer monopoly.

2. If the scaling laws bend e.g. RL doesn’t generalize, model capabilities may continue to grow, but slower and only as basic research bets pay off. This means that each lab may end up finding their own niche for the type of research they are good at, and there will be an oligopoly of model providers who capture most of the value in each of their niches.

3. If the scaling laws hold until AGI, we can translate exponentially more compute into linear increases in intelligence and these linear increases into exponential or super-exponential economic returns. This creates strong incentives for more capex spend, and in turn, structural pressure to consolidate. Only a few of the labs will make it, with most bets on frontier AI labs being written to zero.

4. If we get beyond AGI to super-intelligence, then even research labs which are relatively commoditized will capture enormous value, owing to the size of the lightcone. However, it’s less clear if the current notions of property rights, value and returns are still as relevant.

At a portfolio level, the best case for the current spray-and-pray deployment of venture capital is case 2: indefinite optimism in some sort of AI progress, but without any clear conviction and with a long-term bet against scaling.

We would not bet against scaling! If you take scaling seriously, you end up in case 3: many of the enormous investments into frontier AI labs should be thought of as having the risk profile of early-stage investments but with the capital requirements of growth-stage investments. Diversifying on that basis isn’t prudent; it’s simply indecisive.

Thanks to Sam, Zeel, Atiyu, Arden and Mohit for reading drafts of this.

This publicly marks the start of the “search” paradigm in modern ML, just as ChatGPT’s launch in 2022 marked the arrival of the “learning” paradigm. 

With this new paradigm, you should expect progress in ML performance over the next two years to be at least as fast as it was in the last two years. In fact, there are reasons to expect it to be even faster.

Here’s why!

Bitter lesson 2: electric boogaloo

When I talk about “learning” and “search”, I mean the two “general methods that leverage computation” which Rich Sutton named in The Bitter Lesson. Loosely, “learning” is fitting to patterns in the world, while “search” is finding the best option in a space of possibilities.

The Bitter Lesson explains why a single model launch, like o1’s, can catalyse such rapid progress. Since “the most effective” methods in ML are these general methods, progress in ML doesn’t occur steadily, but instead comes in fits and spurts. This is because it relies on researchers finding a technique which gets predictably better with more computing power (“compute”). Once they have conviction in it, they can scale that technique with more compute. A model launch is simply a lagging indicator of this conviction.

With “learning”, some researchers had the realisation in 2020 that they could use more compute to train a model with more parameters on more data, and it would just improve. Thus they spent the next two years doing that, and in the process, learnt a lot about how to use that compute effectively. This unlocked ChatGPT in 2022, and has continued to power the last two years of improvements in AI.

(Check out yesterday’s post if you’re curious about how this came to pass!)

Moving from training to inference

However, most of this compute has been used during the training process of the models. This is a result of how “learning” works. If you want the model to learn more during training, you can scale that pretty arbitrarily, just by making it bigger or having it see more data. By contrast, the model can only learn during inference if you give it in-context examples, but this is constrained by the size of its context window, which isn’t infinitely scalable.

This means that spending more compute on inference was always the next frontier to be tackled. Many users of these models have tried to do clever prompting or build convoluted wrappers around these models to take advantage of that. You may have heard of techniques like “in-context learning”, “chain of thought” and so on.

This is a mistake.

By handcrafting these scaffolds, they are ignoring The Bitter Lesson at their own peril. What they needed to do instead was to find a method that could scalably absorb compute at inference time, without their intervention. Many researchers at top AI labs have long suspected that “search” fits this description:

• Noam Brown (July 2023): “I’ll now investigate how to make these [self-play and reasoning] methods truly general”

• The Information (November 2023): “among the techniques the [OpenAI] team experimented with was … test-time computation”

• Dwarkesh Patel (December 2023): “almost all the researchers I talk to in the big AI labs are quite confident they’ll get self-play to work”

The basic idea behind “search” is that you can solve hard problems by trying many different paths and seeing which gets you the best outcome. Since you can try arbitrarily many paths if you’re willing to spend the compute, this is something that can pretty naturally scale at inference time.

Verification is easier than generation

There’s a long history of “search” working in other domains. One notable example is AlphaGo Zero, which was DeepMind’s system for playing Go. This system had two parts: a neural network and a Monte Carlo Tree Search algorithm.

The neural net generates:

• An estimate of how good the current board state is

• The probability of different next moves given the current board state

The MCTS algorithm looks at all possible next moves and picks a move. Early in the game, it tries to pick moves it hasn’t encountered yet. As it becomes more certain about which move is good or bad, it focuses on picking the best possible move. Essentially, it’s trying to sensibly trade off “exploration” and “exploitation”. After the MCTS algorithm has picked a move, the neural net evaluates the board state conditional on that move being made and updates the algorithm’s knowledge of how good each move is.

To improve this system, they got it to play against itself. At the end of each game, the self-play produced a series of data about what the board state was after each move, what the MCTS algorithm thought it should do and who ended up winning the game. This data could then be used to train the neural network, making it better and better each time. 

At inference, the combination of the trained neural net and the MCTS algorithm produced the superhuman performance of AlphaGo Zero system. This was rated at over 1000 Elo points above the AlphaGo Lee system that had famously beat Lee Sedol.

Notice that search is key!

When doing inference without MCTS (“raw network”), it was over 2000 Elo points below the version with MCTS, and actually underperformed even AlphaGo Lee. In fact, if we wanted to bridge the gap between the raw network and the network with MCTS, but without doing any search, the neural network would need to be 100,000x bigger.

This training-inference trade-off is replicated across other games. For example, Jones found that you can attain the same Elo in Hex on a model with 10x less training compute by spending 15x on inference compute:

In general, the “search” process follows a certain loop:

1. Get the model to play against itself

2. Break down the “play” into steps

3. Have a search algorithm explore over the steps

4. Verify if the end result was good

5. Tell the search algorithm so it learns which steps are good

6. Train the model based on what the search algorithm has learnt

In the case of games like Go and Hex, the existence of pre-defined rules, moves and win conditions mean that this loop is pretty easy to do. However, what about domains that are less well specified?

Combining learning and search

To start with, researchers have focused on fields like mathematics or coding, where it is possible to do verification to some degree e.g. with theorem provers or unit tests. Much of the exciting research in the past two years has been about bringing “search” to these areas.

The simplest version of “search” is one step deep. That means doing many single step generations and then picking the best one. One way of picking is to take the most popular answer. It’s a bit like the “Ask the Audience” lifeline on “Who Wants To Be A Millionaire”!

This is the majority voting approach from Lewkowycz et al., and doing so (“maj1@k”) with the Minerva model shows dramatic improvements relative to just using model itself across STEM benchmarks:

A more sophisticated approach is to swap about majority voting for a neural network that tries to pick the best answer. This is known as an “outcome reward model”, and Lightman et al. find that doing this improves their results on the MATH benchmark from 69.6% accuracy with majority voting to 72.4%.

To make it even better, they break down the problem into multiple steps, and train a “process reward model”. This model looks at every step of the reasoning process, instead of just the output, and picking the answer using this model brings accuracy to 78.2%.

This is shaping up to a similar loop as before:

1. Get the language model to respond to a prompt

2. Each “move” is the model generating some reasoning steps

3. A search algorithm picks whether to keep taking more “moves”

4. Verify if the eventual answer is correct

5. Tell the search algorithm so it learns which “moves” were good

6. Train the model based on what the search algorithm has learnt

By repeating this many times per prompt for billions of prompts, you get a language model that is much better at generating reasoning steps, as well as a search algorithm that knows if a particular set of reasoning steps is good. At inference, you can use this by generating some reasoning steps, and then either re-generating it or taking the next set of reasoning steps, depending on whether the search algorithm thinks this was a good “move”.

Then you can scale this in two ways: by spending more time on training compute (e.g. more prompts, more attempts per prompt) or by spending more time on inference compute (e.g. setting a higher threshold for what a good “move” is, trying many “moves”).

Scaling laws, redux

This is an interesting hypothesis. Does it actually let us scale inference compute though?

Here are a few reasons it might not:

1. Maybe none of the “moves” are good enough i.e. if you asked a 3 year old an advanced college maths question, it doesn’t matter how many times you ask them and how well you can search through their answers: they simply aren’t smart enough

2. Maybe verification is hard enough that the search algorithm doesn’t learn much and can’t help us

3. Even if the first two aren’t completely true, they may be true enough that it is not economical to use inference compute, relative to training compute

Let’s tackle each in order.

It’s not super surprising that if you do more sampling, the probability that at least one of the samples will be the correct answer (“coverage”) increases, at least a little bit. What Brown et al. show us is that this scales to an unbelievable degree (i.e. to tens of thousands of samples) and scales following a predictable power law:

By simply sampling many times and picking the answer based on an automated verifier, this increase in “coverage” translates directly into better performance. In fact, they manage to beat the state-of-the-art SWE-bench Lite results by a staggering 13pp (and as of this post, would still top the leaderboard):

Thus, issue 1 isn’t a problem. However, they also show that for maths problems without theorem provers, verification isn’t that easy. Whether it is with majority voting or using a reward model to search over the samples, it seems that this plateaus quickly, causing a gap between “coverage” and the actual success rate:

This means that issue 2 is only half-solved, for certain domains with easy verifiers. What does this mean for issue 3 i.e. trading off training and inference compute?

This is where Snell et al. come in. They ask the question: “if you can 10x the combined amount of training and inference compute, how much should you spend on each?”. They find that the two types of compute are substitutable, but not perfectly, and inference compute is best spent on easier problems.

The unhobbling is coming  

Let’s take a step back.

Long before ChatGPT came out, and before even OpenAI existed, Ilya Sutskever had said that “the models, they just want to learn”. The belief that you could pour more training compute and scale up “learning” is not a new one. Yet it took ChatGPT to turn it into a public and undeniable fact.

For the past two years, we have been in the pre-ChatGPT era with “search”. While all of the research I’ve mentioned above kept pointing in the direction of scaling inference compute, no one had productionised and done it in a way that made it public and undeniable.

On September 12th, o1 changed that:

By themselves, these results on competition mathematics and coding problems are already impressive. These are hard problems where o1 blows GPT-4o out of the water, but in some ways, this can be expected, since maths and code can have very good verifiers that provide a clear reward signal to the search algorithm.

What is more staggering is the performance on other domains:

For example, GPQA is a “Google-proof” science benchmark that even domain PhDs struggle with, and yet o1 beats them and gpt-4o very handily. Nor is this limited to STEM: o1 also improves on gpt-4o’s performance on the LSATs, econometrics etc. 

In addition to validating how search can work on more general domains, o1 also shows that it can scale, across both training and inference compute:

Just as ChatGPT fired the starting gun for the race to scale “learning”, o1 has done the same for “search”!

Why does this matter?

Scaling “search” on its own will improve domain-specific capabilities. Instead of needing to re-train a model, you can simulate what a larger model’s performance would be by simply spending more inference compute to solve your problem. This pulls forward the ability to deploy AI systems into real world problems, and makes the price-performance trade-off less discontinuous and dependent on research labs releasing new models.

Just to give a sense of price, Zhang’s replication of the scaling plot lets us estimate that the maximum accuracy displayed can be achieved with $1.6 of inference compute per problem. That’s almost certainly cheaper than how much it would have cost to train a model that gets the same accuracy without “search”.

This alone would make scaling “search” as exciting as scaling “learning” has been. However, it turns out that "search" will also make "learning" better.

Already, lots of training data is synthetic i.e. generated by an AI. Now, every single set of reasoning steps by o1 can be used to train the next model. Since o1 is smarter, the quality of these reasoning traces is higher, and it can filter out the bad sets of reasoning steps better than before.

The reason we couldn’t make this data flywheel in the past is because the models weren’t smart enough to generate useful reasoning traces, or filter out the bad ones. Now that we seem to have reached that threshold, we can follow the same playbook which made AlphaGo superhuman: bootstrapping the next model from the outputs of the previous one.

That’s why Sam Altman is so confident that “deep learning worked”, and why you should expect even faster progress soon. Remember, the models are the worst they’ll ever be!

Thanks to Kevin Niechen, Zhengdong Wang, Jannik Schilling, Bradley Hsu, Devansh Pandey, Zach Mazlish and Basil Halperin for discussion and feedback.

These are the two ingredients of training ML models. The total amount of computation (“compute”) you need to do to train a model is proportional to the number of parameters multiplied by the amount of data (measured in “tokens”).

Four years ago, it was well-known that if you had more compute to train a model, you should spend most of it on parameters.

Two years ago, everyone changed their mind and believed you should spend it equally on parameters and data.

Just last year, it became widely accepted that you should spend orders-of-magnitude more on data than anyone had previously thought.

Why have the recipes to training these models changed so much, and so frequently? To understand this, we need to take a walk through the intellectual history of scaling modern ML models.

The Bitter Lesson

The science of training large models starts with Kaplan et al. in early 2020. In the paper, the OpenAI team tested different configurations of transformer language models. The two most important axes they varied were the number of model parameters (768 to 1.5B) and number of tokens in the dataset (22M to 23B). Implicitly, this also varied the total amount of compute used to train the model, since compute ~ parameters x tokens.

They found a stable and predictable power-law relationship between compute and the performance of the model. Here, performance is measured by the “loss”, which is simply the model’s error in predicting the next token across all the text it is trained on:

They also gave a formula for how to use your compute efficiently, since for any fixed amount of compute, there is a trade-off between making the model bigger and training it on more data. Kaplan showed that if you could 10x the amount of compute, you should 5x the number of parameters and 2x the number of tokens it is trained on.

The punchline was that 5 months later, the OpenAI team launched GPT-3. With a staggering 175B parameters trained on 300B tokens, this validated the Kaplan scaling laws with three orders of magnitude more compute than the biggest Kaplan model:

With GPT-3, OpenAI had demonstrated that the scaling law was really quite predictable. While “bigger is better” wasn’t the most surprising result, the idea that you could predict exactly how good it would be, and could get a recipe for the exact ratio of parameters-to-tokens was unprecedented.

They had also shown that size was qualitatively important: while the “loss” of the model changed steadily with size, the actual human-relevant capabilities were more discontinuous and emerged above certain thresholds. One “emergent capability” was the ability for models to reason in-context i.e. learn how to do a task accurately if you gave it one (“one-shot”) or multiple (“few-shot”) examples. By being given examples, large models gained a lot more accuracy than smaller models:

After an initial period of shock, everyone quickly followed suit: DeepMind released Gopher (280B parameters, 300B tokens) in late 2021 and NVIDIA released Megatron (530B parameters, 270B tokens) in early 2022.

Chinchilla outperforms Gopher

Yet just as the race to train big models was heating up, it would get interrupted in March 2022 by Hoffmann et al., better known as the Chinchilla scaling laws. In this paper, the DeepMind team revisited what the right ratio of parameters to tokens was for any fixed amount of compute. What they found was that data was far more important than people realised!

They took three approaches to this, but the second “isoFLOP” method is most intuitive:

• Take a fixed amount of compute

• Pick a range of model sizes where the larger ones will be trained on less data

• Plot each model’s performance against parameters and join the dots to get an isoFLOP curve, where each point has the same total compute

• The lowest point on the curve has the best performance

The Chinchilla team exhaustively trained over 400 models across 9 different isoFLOP curves:

By fitting a regression line to the minimum point of each isoFLOP curve, they found that a 10x increase in compute should be split equally between model size and dataset size i.e. 3.1x each of them, rather than Kaplan’s 5x and 2x split.

As with Kaplan, they validated this by training a model with even more compute than their experiments. In particular, they took Gopher’s FLOP count but trained it with the parameter-token split implied by their experiments. This was “Chinchilla”! With 70B parameters trained on 1.4T tokens, it blew past Gopher on every single benchmark e.g. the Pile, MMLU, BIG-Bench etc.

What did Kaplan get wrong?

The immediate difference between the two papers was the choice of the learning rate schedule. The learning rate determines how much the parameters of a model change when it sees each token of training data. A common LR schedule is to start training with a low LR and build it up linearly to let the model “warm up”, and then slowly decay from the maximum across the rest of training:

Kaplan had picked a LR schedule with 3000 steps of warmup followed by a fixed decay schedule. When Chinchilla first came out, they suggested that Kaplan’s problem was having the same LR schedule for every model, rather than matching the length of the decay to the amount of data the model was trained on.

In the past few months, however, new work from Porian et al. has suggested that there might be more at play. By recreating the Kaplan experiments, they found that Chinchilla’s hypothesis about LR decay wasn’t that important. What mattered more was:

• Kaplan ignoring the compute used by last layer of the model

• Kaplan’s fixed 3000 steps of warmup (which meant smaller models trained on less data spent relatively more time in warmup)

• Kaplan using the same hyperparameters for all model sizes, rather than tuning them for each one

All of this to say, the science of scaling is still pretty nascent! Regardless, one thing is clear: Chinchilla’s recipe set the bar for how to train large language models. Since then, every time someone says they are training a “compute optimal model”, they mean that they are following Chinchilla when deciding the parameter-token split.

Llama outperforms Chinchilla

Unfortunately, the description “compute optimal” ended up being rather misleading, even to researchers in the field. That’s because what Chinchilla means by “compute optimal” is “training compute optimal” i.e. the best parameter-token split if you only consider the compute you spend in training the model. However, you will also want to serve the model for inference, and larger models cost more to serve. Thus very rarely, if ever, do you actually want to train a Chinchilla optimal model.

Instead, you want to train on fewer parameters but more tokens i.e. moving left from the bottom of an isoFLOP curve (see the red arrow):

This gets you the same loss as before. You have to spend more compute during training, but in return, you get a smaller model that costs less compute at inference.

While this insight isn’t especially surprising, it took Meta releasing its LLaMA models in February 2023 to make it popular. By continuing to train models on even more data than Chinchilla implied, they were able to produce a 13B model that beat GPT-3 and a 65B model that beat Chinchilla.

Since then, subsequent LLaMA models like the LLaMA 3 series have gone even further. For a sense of scale, the 8B model was trained on 15T tokens. That is 75x the Chinchilla-optimal amount of 200B tokens for its size. This allowed it to match Chinchilla across a wide range of benchmarks, despite being an order of magnitude smaller.

What about inference?

It’s been four years since Kaplan first came out, and at this point, the core decisions in scaling up pretraining are pretty settled. While it’s hard to tell what the closed frontier labs are doing, the open-source researchers are broadly following the LLaMA recipe and producing models which are competitive with state-of-the-art closed-source models.

One key insight unlocked by the LLaMA models is that inference compute matters too, and you can trade off training compute and inference compute. What happens when we start to scale inference compute? Come back tomorrow to find out!

Thanks to Kevin Niechen, Zhengdong Wang, Jannik Schilling, Bradley Hsu and Devansh Pandey for discussion and feedback.

One obstacle to interpreting neural networks is polysemanticity. This where a single neuron represents multiple features.

If there are more features than neurons, it might be “necessary” for the model to be polysemantic in order to represent everything. This is the notion of “superposition” from Elhage et al. (2022).

Of course, a clear solution would be to train a model large enough to have at least one neuron per feature. However, what we find in “What Causes Polysemanticity?” is that polysemanticity can happen “incidentally” in the training process, even if we have a large enough model.

Hypothesis

When we initialise a neural network, the weights are random. Some neurons will be more correlated with some features than other neurons, just by chance.

As training happens, the optimiser pushes the weights of those correlated neurons in the direction of the features, so that they can represent the features well. If there is pressure for sparsity, only one neuron will represent each feature.

Most likely, this is the neuron which was most correlated to the feature at initialisation. If it happened to be the most correlated neuron for multiple features, then it would end up representing multiple features.

In that case, we get polysemanticity “incidentally”.

Experiments

To test this hypothesis, we consider the simplest possible setup: over-parameterised autoencoders similar to those in Elhage et al. (2022). That is:

where

These models were trained on the standard basis vectors.

To induce sparsity, we take two separate approaches: introducing ℓ1 regularisation for the model weights and adding noise after the hidden layer.

We find that:

1. ℓ1 regularisation induces sparsity

2. Some types of noise can induce sparsity

3. The amount of incidental polysemanticity can be predicted

4. It is due to the weight initialisations

Sparsity from Regularisation

Result 1: ℓ1 regularisation induces a winner-takes-all dynamic at a rate proportional to the regularisation parameter.

Loss with ℓ1

Taking the ith basis vector as the input, the output of the model is:

With ℓ1 regularisation, our loss function becomes:

where λ is the regularisation parameter.

Thus, gradient descent pushes us in the direction:

Forces for sparsity

We can split this into the three terms that push

1. Feature benefit: pushes it to be unit length

2. Interference: pushes it to be orthogonal to other ones

3. Regularisation: pushes it to be sparse, if non-zero

The feature benefit and regularisation forces are in competition. Since the regularisation force has a constant value while the feature benefit force is proportional to its length, the regularisation force will dominate for small values and the feature benefit force will dominate for large values.

Thus W_{i,k} will be pushed to 0 if it is below some threshold θ. Leaving the derivations for the paper, we find the net effect on sparsity is proportional to how far the weight is from this threshold:

Speed of sparsification

In fact, we can quantify the speed at which sparsity is induced. Again, leaving the maths for the paper, it follows from above that the ℓ1 norm at time t is inversely proportional to λt:

Since |W_{i,:}(t)|≈1 throughout training, the m′ non-zero values at any particular point should have a magnitude of around:

thus giving:

The weights should go from Θ(m) to Θ(1λt) as t≥1/(λ sqrt{m}) and Θ(1) as t≥1/λ. This is exactly what we see:

Sparsity from Noise

Result 2: noise drawn from a distribution with excess kurtosis induces sparsity.

Implicit regularisation

In practice, we don’t get sparsity in neural networks because of ℓ1 regularisation. A more realistic cause is via noise in the hidden layer, a la Bricken et al. (2023):

for

Having removed the regularisation term, the loss is rotationally symmetric with respect to the hidden layer (excluding the noise). That means there is no privileged basis, and no particular reason for features to be represented by a single neuron, as opposed to a linear combination of features.

However, if we take the noise into account, we find that one term in the loss is:

where μ_4 is the fourth moment of D, and μ_4/σ_4−3 is the excess kurtosis.

Thus, when D has negative excess kurtosis, this component of the loss will push to increase |W_{i,:}|_4.

We also have the constraint that |W_{i,:}|_2=1 from before.

This incentivises W_{i,j}=±1 for some j.

We also have W_{i,k}=0 for k≠j.

Bernoulli vs. Gaussian noise

Bernoulli noise of either ±σ has excess kurtosis of −2, while Gaussian noise has excess kurtosis of 0. Thus we would expect to see the former to induce sparsity (and a fourth norm of 1), while the latter would not. As expected:

Counting Hypothesis

Result 3: the number of polysemantic neurons can be predicted by a simple combinatorial model.

Possible model solutions

Recall that the output of the autoencoder is:

That is, we would like the dot product of W_i,: with itself to be 1, and with all other ones to be ≤0.

One way to satisfy this is if W_{i,:} equals the ith standard basis vector

This is because WW^T will just be the identity matrix, so

However, when m>n, we have another solution. Take m=4 and n=2, and consider the following weight matrix:

We see that ReLU(WW^Te_1)=(1,0) and ReLU(WW^Te_2)=(0,1), which still satisfies the constraints. This is a polysemantic solution!

Interference force

Knowing that it is possible, we can now ask why it occurs. One force we haven’t considered in detail is the interference force:

up to constants.

This is only non-zero if the angle between W_{i,:} and the jth is less than π/2.

Thus, we can simplify by only considering its effect in the direction of W_{i,:}. It has magnitude:

This means that the interference force should be weak at the start when the dot products (of different ones) are mean zero, and only kick in if they share some non-zero coordinate k. If the k coordinate for both have the same sign, the interference force will push at least one to zero. Thus, we would only expect that polysemanticity occurs when they have the opposite sign, since the ReLU will zero out the negative term and they will maintain their non-zero value.

Balls and bins

With nC2 pairs of features, 1/m probability of the most significant neuron being the same for both and 1/2 probability of them having the opposite sign, we would predict there to be

polysemantic neurons, which we find:

Initialisation Hypothesis

Result 4: polysemanticity occurs in m>n models due to weight initialisations.

If initialisations were the cause of polysemanticity, the weights at the start of training should be correlated with the weights at the end. That is, the diagonals of WW^T should be larger than the off-diagonals. As predicted:

Future Work

The incidental polysemanticity we have discussed in our work is qualitatively different from necessary polysemanticity, because it arises from the learning dynamics inducing a privileged basis. Furthermore, the fact that it occurs all the way up to m=n^2 suggests that making the model larger may not solve the problem.

We look forward to future work which investigates this phenomenon in more fleshed-out settings, and which attempts to nudge the learning dynamics to stop it from occurring.

Update: we have now turned this into an academic paper!

In this post, we point out that short AI timelines would cause real interest rates to be high, and would do so under expectations of either unaligned or aligned AI. However, 30- to 50-year real interest rates are low. We argue that this suggests one of two possibilities:

1. Long(er) timelines. Financial markets are often highly effective information aggregators (the “efficient market hypothesis”), and therefore real interest rates accurately reflect that transformative AI is unlikely to be developed in the next 30-50 years.

2. Market inefficiency. Markets are radically underestimating how soon advanced AI technology will be developed, and real interest rates are therefore too low. There is thus an opportunity for philanthropists to borrow while real rates are low to cheaply do good today; and/or an opportunity for anyone to earn excess returns by betting that real rates will rise.

In the rest of this post we flesh out this argument.

1. Both intuitively and under every mainstream economic model, the “explosive growth” caused by aligned AI would cause high real interest rates.

2. Both intuitively and under every mainstream economic model, the existential risk caused by unaligned AI would cause high real interest rates.

3. We show that in the historical data, indeed, real interest rates have been correlated with future growth.

4. Plugging the Cotra probabilities for AI timelines into the baseline workhorse model of economic growth implies substantially higher real interest rates today.

5. In particular, we argue that markets are decisively rejecting the shortest possible timelines of 0-10 years.

6. We argue that the efficient market hypothesis (EMH) is a reasonable prior, and therefore one reasonable interpretation of low real rates is that since markets are simply not forecasting short timelines, neither should we be forecasting short timelines.

7. Alternatively, if you believe that financial markets are wrong, then you have the opportunity to (1) borrow cheaply today and use that money to e.g. fund AI safety work; and/or (2) earn alpha by betting that real rates will rise.

An order-of-magnitude estimate is that, if markets are getting this wrong, then there is easily $1 trillion lying on the table in the US treasury bond market alone – setting aside the enormous implications for every other asset class.

Interpretation. We view our argument as the best existing outside view evidence on AI timelines – but also as only one model among a mixture of models that you should consider when thinking about AI timelines. The logic here is a simple implication of a few basic concepts in orthodox economic theory and some supporting empirical evidence, which is important because the unprecedented nature of transformative AI makes “reference class”-based outside views difficult to construct. This outside view approach contrasts with, and complements, an inside view approach, which attempts to build a detailed structural model of the world to forecast timelines (e.g. Cotra 2020; see also Nostalgebraist 2022).

Outline. If you want a short version of the argument, sections I and II (700 words) are the heart of the post. Additionally, the section titles are themselves summaries, and we use text formatting to highlight key ideas.

I. Long-term real rates would be high if the market was pricing advanced AI

Real interest rates reflect, among other things:

1. Time discounting, which includes the probability of death

2. Expectations of future economic growth

This claim is compactly summarized in the “Ramsey rule” (and the only math that we will introduce in this post), a version of the “Euler equation” that in one form or another lies at the heart of every theory and model of dynamic macroeconomics:

r=ρ+σg

where:

• r is the real interest rate over a given time horizon

• ρ is time discounting over that horizon

• σ is a (positive) preference parameter reflecting how much someone cares about smoothing consumption over time

• g is the growth rate

(Internalizing the meaning of these Greek letters is wholly not necessary.)

While more elaborate macroeconomic theories vary this equation in interesting and important ways, it is common to all of these theories that the real interest rate is higher when either (1) the time discount rate is high or (2) future growth is expected to be high.

We now provide some intuition for these claims.

Time discounting and mortality risk. Time discounting refers to how much people discount the future relative to the present, which captures both (i) intrinsic preference for the present relative to the future and (ii) the probability of death.

The intuition for why the probability of death raises the real rate is the following. Suppose we expect with high probability that humanity will go extinct next year. Then there is no reason to save today: no one will be around to use the savings. This pushes up the real interest rate, since there is less money available for lending.

Economic growth. To understand why higher economic growth raises the real interest rate, the intuition is similar. If we expect to be wildly rich next year, then there is also no reason to save today: we are going to be tremendously rich, so we might as well use our money today while we’re still comparatively poor.

(For the formal math of the Euler equation, Baker, Delong, and Krugman 2005 is a useful reference. The core intuition is that either mortality risk or the prospect of utopian abundance reduces the supply of savings, due to consumption smoothing logic, which pushes up real interest rates.)

Transformative AI and real rates. Transformative AI would either raise the risk of extinction (if unaligned), or raise economic growth rates (if aligned).

Therefore, based on the economic logic above, the prospect of transformative AI – unaligned or aligned – will result in high real interest rates. This is the key claim of this post.

As an example in the aligned case, Davidson (2021) usefully defines AI-induced “explosive growth” as an increase in growth rates to at least 30% annually. Under a baseline calibration where σ=1 and ρ=0.01, and importantly assuming growth rates are known with certainty, the Euler equation implies that moving from 2% growth to 30% growth would raise real rates from 3% to 31%!

For comparison, real rates in the data we discuss below have never gone above 5%.

(In using terms like “transformative AI” or “advanced AI”, we refer to the cluster of concepts discussed in Yudkowsky 2008, Bostrom 2014, Cotra 2020, Carlsmith 2021, Davidson 2021, Karnofsky 2022, and related literature: AI technology that precipitates a transition comparable to the agricultural or industrial revolutions.)

II. But: long-term real rates are low

The US 30-year real interest rate ended 2022 at 1.6%. Over the full year it averaged 0.7%, and as recently as March was below zero. Looking at a shorter time horizon, the US 10-year real interest rate is 1.6%, and similarly was below negative one percent as recently as March.

(Data sources used here are explained in section V.)

The UK in autumn 2021 sold a 50-year real bond with a -2.4% rate at the time. Real rates on analogous bonds in other developed countries in recent years have been similarly low/negative for the longest horizons available. Austria has a 100-year nominal bond – being nominal should make its rate higher due to expected inflation – with yields less than 3%.

Thus the conclusion previewed above: financial markets, as evidenced by real interest rates, are not expecting a high probability of either AI-induced growth acceleration or elevated existential risk, on at least a 30-50 year time horizon.

III. Uncertainty, takeoff speeds, inequality, and stocks

In this section we briefly consider some potentially important complications.

Uncertainty. The Euler equation and the intuition described above assumed certainty about AI timelines, but taking into account uncertainty does not change the core logic. With uncertainty about the future economic growth rate, then the real interest rate reflects the expected future economic growth rate, where importantly the expectation is taken over the risk-neutral measure: in brief, probabilities of different states are reweighted by their marginal utility. We return to this in our quantitative model below.

Takeoff speeds. Nothing in the logic above relating growth to real rates depends on slow vs. fast takeoff speed; the argument can be reread under either assumption and nothing changes. Likewise, when considering the case of aligned AI, rates should be elevated whether economic growth starts to rise more rapidly before advanced AI is developed or only does so afterwards. What matters is that GDP – or really, consumption – ends up high within the time horizon under consideration. As long as future consumption will be high within the time horizon, then there is less motive to save today (“consumption smoothing”), pushing up the real rate.

Inequality. The logic above assumed that the development of transformative AI affects everyone equally. This is a reasonable assumption in the case of unaligned AI, where it is thought that all of humanity will be evaporated. However, when considering aligned AI, it may be thought that only some will benefit, and therefore real interest rates will not move much: if only an elite Silicon Valley minority is expected to have utopian wealth next year, then everyone else may very well still choose to save today.

It is indeed the case that inequality in expected gains from transformative AI would dampen the impact on real rates, but this argument should not be overrated. First, asset prices can be crudely thought of as reflecting a wealth-weighted average across investors. Even if only an elite minority becomes fabulously wealthy, it is their desire for consumption smoothing which will end up dominating the determination of the real rate. Second, truly transformative AI leading to 30%+ economy-wide growth (“Moore’s law for everything”) would not be possible without having economy-wide benefits.

Stocks. One naive objection to the argument here would be the claim that real interest rates sound like an odd, arbitrary asset price to consider; certainly stock prices are the asset price that receive the most media attention.

In appendix 1, we explain that the level of the real interest rate affects every asset price: stocks for instance reflect the present discounted value of future dividends; and real interest rates determine the discount rate used to discount those future dividends. Thus, if real interest rates are ‘wrong’, every asset price is wrong. If real interest rates are wrong, a lot of money is on the table, a point to which we return in section X.

We also argue that stock prices in particular are not a useful indicator of market expectations of AI timelines. Above all, high stock prices of chipmakers or companies like Alphabet (parent of DeepMind) could only reflect expectations for aligned AI and could not be informative of the risk of unaligned AI. Additionally, as we explain further in the appendix, aligned AI could even lower equity prices, by pushing up discount rates.

IV. Historical data on interest rates supports the theory: preliminaries

In section I, we gave theoretical intuition for why higher expected growth or higher existential risk would result in higher interest rates: expectations for such high growth or mortality risk would lead people to want to save less and borrow more today. In this section and the next two, we showcase some simple empirical evidence that the predicted relationships hold in the available data.

Measuring real rates. To compare historical real interest rates to historical growth, we need to measure real interest rates.

Most bonds historically have been nominal, where the yield is not adjusted for changes in inflation. Therefore, the vast majority of research studying real interest rates starts with nominal interest rates, attempts to construct an estimate of expected inflation using some statistical model, and then subtracts this estimate of expected inflation from the nominal rate to get an estimated real interest rate. However, constructing measures of inflation expectations is extremely difficult, and as a result most papers in this literature are not very informative.

Additionally, most bonds historically have had some risk of default. Adjusting for this default premium is also extremely difficult, which in particular complicates analysis of long-run interest rate trends.

The difficulty in measuring real rates is one of the main causes, in our view, of Tyler Cowen’s Third Law: “all propositions about real interest rates are wrong”. Throughout this piece, we are badly violating this (Gödelian) Third Law. In appendix 2, we expand on our argument that the source of Tyler’s Third Law is measurement issues in the extant literature, together with some separate, frequent conceptual errors.

Our approach. We take a more direct approach.

Real rates. For our primary analysis, we instead use market real interest rates from inflation-linked bonds. Because we use interest rates directly from inflation-linked bonds – instead of constructing shoddy estimates of inflation expectations to use with nominal interest rates – this approach avoids the measurement issue just discussed (and, we argue, allows us to escape Cowen’s Third Law).

To our knowledge, prior literature has not used real rates from inflation-linked bonds only because these bonds are comparatively new. Using inflation-linked bonds confines our sample to the last ~20 years in the US, the last ~30 in the UK/Australia/Canada. Before that, inflation-linked bonds didn’t exist. Other countries have data for even fewer years and less liquid bond markets.

(The yields on inflation-linked bonds are not perfect measures of real rates, because of risk premia, liquidity issues, and some subtle issues with the way these securities are structured. You can build a model and attempt to strip out these issues; here, we will just use the raw rates. If you prefer to think of these empirics as “are inflation-linked bond yields predictive of future real growth” rather than “are real rates predictive of future real growth”, that interpretation is still sufficient for the logic of this post.)

Nominal rates. Because there are only 20 or 30 years of data on real interest rates from inflation-linked bonds, we supplement our data by also considering unadjusted nominal interest rates. Nominal interest rates reflect real interest rates plus inflation expectations, so it is not appropriate to compare nominal interest rates to real GDP growth.

Instead, analogously to comparing real interest rates to real GDP growth, we compare nominal interest rates to nominal GDP growth. The latter is not an ideal comparison under economic theory – and inflation variability could swamp real growth variability – but we argue that this approach is simple and transparent.

Looking at nominal rates allows us to have a very large sample of countries for many decades: we use OECD data on nominal rates available for up to 70 years across 39 countries.

V. Historical data on interest rates supports the theory: graphs

The goal of this section is to show that real interest rates have correlated with future real economic growth, and secondarily, that nominal interest rates have correlated with future nominal economic growth. We also briefly discuss the state of empirical evidence on the correlation between real rates and existential risk.

Real rates vs. real growth. A first cut at the data suggests that, indeed, higher real rates today predict higher real growth in the future:

To see how to read these graphs, take the left-most graph (“10-year horizon”) for example. The x-axis shows the level of the real interest rate, as reflected on 10-year inflation linked bonds. The y-axis shows average real GDP growth over the following 10 years.

The middle and right hand graphs show the same, at the 15-year and 20-year horizons. The scatter plot shows all available data for the US (since 1999), the UK (since 1985), Australia (since 1995), and Canada (since 1991). (Data for Australia and Canada is only available at the 10-year horizon, and comes from Augur Labs.)

Eyeballing the figure, there appears to be a strong relationship between real interest rates today and future economic growth over the next 10-20 years.

To our knowledge, this simple stylized fact is novel.

Caveats. “Eyeballing it” is not a formal econometric method; but, this is a blog post not a journal article (TIABPNAJA). We do not perform any formal statistical tests here, but we do want to acknowledge some important statistical points and other caveats.

First, the data points in the scatter plot are not statistically independent: real rates and growth are both persistent variables; the data points contain overlapping periods; and growth rates in these four countries are correlated. These issues are evident even from eyeballing the time series. Second, of course this relationship is not causally identified: we do not have exogenous variation in real growth rates. (If you have ideas for identifying the causal effect of higher real growth expectations on real rates, we would love to discuss with you.)

Relatedly, many other things are changing in the world which are likely to affect real rates. Population growth is slowing, retirement is lengthening, the population is aging. But under AI-driven “explosive” growth – again say 30%+ annual growth, following the excellent analysis of Davidson (2021) – then, we might reasonably expect that this massive of an increase in the growth rate would drown out the impact of any other factors.

Nominal rates vs. nominal growth. Turning now to evidence from nominal interest rates, recall that the usefulness of this exercise is that while there only exists 20 or 30 years of data on real interest rates for two countries, there is much more data on nominal interest rates.

We simply take all available data on 10-year nominal rates from the set of 39 OECD countries since 1954. The following scatterplot compares the 10-year nominal interest versus nominal GDP growth over the succeeding ten years by country:

Again, there is a strong positive – if certainly not perfect – relationship. (For example, the outlier brown dots at the bottom of the graph are Greece, whose high interest rates despite negative NGDP growth reflect high default risk during an economic depression.)

The same set of nontrivial caveats apply to this analysis as above.

We consider this data from nominal rates to be significantly weaker evidence than the evidence from real rates, but corroboration nonetheless.

Backing out market-implied timelines. Taking the univariate pooled OLS results from the real rate data far too seriously, the fact that the 10-year real rate in the US ended 2022 at 1.6% would predict average annual real GDP growth of 2.6% over the next 10 years in the US; the analogous interest rate of -0.2% in the UK would predict 0.7% annual growth over the next 10 years in the UK. Such growth rates, clearly, are not compatible with the arrival of transformative aligned AI within this horizon.

VI. Empirical evidence on real rates and mortality risk

We have argued that in the theory, real rates should be higher in the face of high economic growth or high mortality risk; empirically, so far, we have only shown a relationship between real rates and growth, but not between real rates and mortality.

Showing that real rates accurately reflect changes in existential risk is very difficult, because there is no word-of-god measurement of how existential risk has evolved over time.

We would be very interested in pursuing new empirical research examining “asset pricing under existential risk”. In appendix 3, we perform a scorched-earth literature review and find essentially zero existing empirical evidence on real rates and existential risks.

Disaster risk. In particular, the extant literature does not study existential risks but instead “merely” disaster risks, under which real assets are devastated but humanity is not exterminated. Disaster risks do not necessarily raise real rates – indeed, such risks are thought to lower real rates due to precautionary savings. That notwithstanding, some highlights of the appendix review include a small set of papers finding that individuals with a higher perceived risk of nuclear conflict during the Cold War saved less, as well as a paper noting that equities which were headquartered in cities more likely to be targeted by Soviet missiles did worse during the Cuban missile crisis (see also). Our assessment is that these and the other available papers on disaster risks discussed in the appendix have severe limitations for the purposes here.

Individual mortality risk. We judge that the best evidence on this topic comes instead from examining the relationship between individual mortality risk and savings/investment behavior. The logic we provided was that if humanity will be extinct next year, then there is no reason to save, pushing up the real rate. Similar logic says that at the individual level, a higher risk of death for any reason should lead to lower savings and less investment in human capital. Examples of lower savings at the individual level need not raise interest rates at the economy-wide level, but do provide evidence for the mechanism whereby extinction risk should lead to lower saving and thus higher interest rates.

One example comes from Malawi, where the provision of a new AIDS therapy caused a significant increase in life expectancy. Using spatial and temporal variation in where and when these therapeutics were rolled out, it was found that increased life expectancy results in more savings and more human capital investment in the form of education spending. Another experiment in Malawi provided information to correct pessimistic priors about life expectancy, and found that higher life expectancy directly caused more investment in agriculture and livestock.

A third example comes from testing for Huntington’s disease, a disease which causes a meaningful drop in life expectancy to around 60 years. Using variation in when people are diagnosed with Huntington’s, it has been found that those who learn they carry the gene for Huntington’s earlier are 30 percentage points less likely to finish college, which is a significant fall in their human capital investment.

Studying the effect on savings and real rates from increased life expectancy at the population level is potentially intractable, but would be interesting to consider further. Again, in our assessment, the best empirical evidence available right now comes from the research on individual “existential” risks and suggests that real rates should increase with existential risk.

VII. Plugging the Cotra probabilities into a simple quantitative model of real interest rates predicts very high rates

Section VI used historical data to go from the current real rate to a very crude market-implied forecast of growth rates; in this section, we instead use a model to go from existing forecasts of AI timelines to timeline-implied real rates. We aim to show that under short AI timelines, real interest rates would be unrealistically elevated.

This is a useful exercise for three reasons. First, the historical data is only able to speak to growth forecasts, and therefore only able to provide a forecast under the possibly incorrect assumption of aligned AI. Second, the empirical forecast assumes a linear relationship between the real rate and growth, which may not be reasonable for a massive change caused by transformative AI. Third and quite important, the historical data cannot transparently tell us anything about uncertainty and the market’s beliefs about the full probability distribution of AI timelines.

We use the canonical (and nonlinear) version of the Euler equation – the model discussed in section I – but now allow for uncertainty on both how soon transformative AI will be developed and whether or not it will be aligned. The model takes as its key inputs (1) a probability of transformative AI each year, and (2) a probability that such technology is aligned.

The model is a simple application of the stochastic Euler equation under an isoelastic utility function. We use the following as a baseline, before considering alternative probabilities:

• We use smoothed Cotra (2022) probabilities for transformative AI over the next 30 years: a 2% yearly chance until 2030, a 3% yearly chance through 2036, and a 4% yearly chance through 2052.

• We use the FTX Future Fund’s median estimate of 15% for the probability that AI is unaligned conditional on the development of transformative AI.

• With the arrival of aligned AI, we use the Davidson (2020) assumption of 30% annual economic growth; with the arrival of unaligned AI, we assume human extinction. In the absence of the development of transformative AI, we assume a steady 1.8% growth rate.

• We calibrate the pure rate of subjective time preference to 0.01 and the consumption smoothing parameter (i.e. inverse of the elasticity of intertemporal substitution) as 1, following the economic literature.

Thus, to summarize: by default, GDP grows at 1.8% per year. Every year, there is some probability (based on Cotra) that transformative AI is developed. If it is developed, there is a 15% probability the world ends, and an 85% chance GDP growth jumps to 30% per year.

We have built a spreadsheet here that allows you to tinker with the numbers yourself, such as adjusting the growth rate under aligned AI, to see what your timelines and probability of alignment would imply for the real interest rate. (It also contains the full Euler equation formula generating the results, for those who want the mathematical details.) We first estimate real rates under the baseline calibration above, before considering variations in the critical inputs.

Baseline results. The model predicts that under zero probability of transformative AI, the real rate at any horizon would be 2.8%. In comparison, under the baseline calibration just described based on Cotra timelines, the real rate at a 30-year horizon would be pushed up to 5.9% – roughly three percentage points higher.

For comparison, the 30-year real rate in the US is currently 1.6%.

While the simple Euler equation somewhat overpredicts the level of the real interest rate even under zero probability of transformative AI – the 2.8% in the model versus the 1.6% in the data – this overprediction is explainable by the radical simplicity of the model that we use and is a known issue in the literature. Adding other factors (e.g. precautionary savings) to the model would lower the level. Changing the level does not change its directional predictions, which help quantitatively explain the fall in real rates over the past ~30 years.

Therefore, what is most informative is the three percentage point difference between the real rate under Cotra timelines (5.9%) versus under no prospect of transformative AI (2.8%): Cotra timelines imply real interest rates substantially higher than their current levels.

Now, from this baseline estimate, we can also consider varying the key inputs.

Varying assumptions on P(misaligned|AGI). First consider changing the assumption that advanced AI is 15% likely to be unaligned (conditional on the development of AGI). Varying this parameter does not have a large impact: moving from 0% to 100% probability of misalignment raises the model’s predicted real rate from 5.8% only to 6.3%.

Varying assumptions on timelines. Second, consider making timelines shorter or longer. In particular, consider varying the probability of development by 2043, which we use as a benchmark per the FTX Future Fund.

We scale the Cotra timelines up and down to vary the probability of development by 2043. (Specifically: we target a specific cumulative probability of development by 2043; and, following Cotra, if the annual probability up until 2030 is x, then it is 1.5x in the subsequent seven years up through 2036, and it is 2x in the remaining years of the 30-year window.)

As the next figure shows and as one might expect, shorter AI timelines have a very large impact on the model’s estimate for the real rate.

• The original baseline parameterization from Cotra corresponds to the FTX Future Fund “upper threshold” of a 45% chance of development by 2043, which generated the 3 percentage point increase in the 30-year real rate discussed above.

• The Future Fund’s median of a 20% probability by 2043 generates a 1.1 percentage point increase in the 30-year real rate.

• The Future Fund’s “lower threshold” of a 10% probability by 2043 generates a 0.5 percentage point increase in the real rate.

These results strongly suggest that any timeline shorter than or equal to the Cotra timeline is not being expected by financial markets.

VIII. Markets are decisively rejecting the shortest possible timelines

While it is not possible to back out exact numbers for the market’s implicit forecast for AI timelines, it is reasonable to say that the market is decisively rejecting – i.e., putting very low probability on – the development of transformative AI in the very near term, say within the next ten years.

Consider the following examples of extremely short timelines:

1. Five year timelines: With a 50% probability of transformative AI by 2027, and the same yearly probability thereafter, the model predicts 13.0pp higher 30-year real rates today!

2. Ten year timelines: With a 50% probability of transformative AI by 2032, and the same yearly probability thereafter, the model predicts 6.5pp higher 30-year real rates today.

Real rate movements of these magnitudes are wildly counterfactual. As previously noted, real rates in the data used above have never gone above even 5%.

Stagnation. As a robustness check, in the configurable spreadsheet we allow you to place some yearly probability on the economy stagnating and growing at 0% per year from thereon. Even with a 20% chance of stagnation by 2053 (higher than realistic), under Cotra timelines, the model generates a 2.1% increase in 30-year rates.

Recent market movements. Real rates have increased around two percentage points since the start of 2022, with the 30-year real rate moving from -0.4% to 1.6%, approximately the pre-covid level. This is a large enough move to merit discussion. While this rise in long-term real rates could reflect changing market expectations for timelines, it seems much more plausible that high inflation, the Russia-Ukraine war, and monetary policy tightening have together worked to drive up short-term real rates and the risk premium on long-term real rates.

IX. Financial markets are the most powerful information aggregators produced by the universe (so far)

Should we update on the fact that markets are not expecting very short timelines?

Probably!

As a prior, we think that market efficiency is reasonable. We do not try to provide a full defense of the efficient markets hypothesis (EMH) in this piece given that it has been debated ad nauseum elsewhere, but here is a scaffolding of what such an argument would look like.

Loosely, the EMH says that the current price of any security incorporates all public information about it, and as such, you should not expect to systematically make money by trading securities.

This is simply a no-arbitrage condition, and certainly no more radical than supply and demand: if something is over- or under-priced, you’ll take action based on that belief until you no longer believe it. In other words, you’ll buy and sell it until you think the price is right. Otherwise, there would be an unexploited opportunity for profit that was being left on the table, and there are no free lunches when the market is in equilibrium.

As a corollary, the current price of a security should be the best available risk-adjusted predictor of its future price. Notice we didn’t say that the price is equal to the “correct” fundamental value. In fact, the current price is almost certainly wrong. What we did say is that it is the best guess, i.e. no one knows if it should be higher or lower.

Testing this hypothesis is difficult, in the same way that testing any equilibrium condition is difficult. Not only is the equilibrium always changing, there is also a joint hypothesis problem which Fama (1970) outlined: comparing actual asset prices to “correct” theoretical asset prices means you are simultaneously testing whatever asset pricing model you choose, alongside the EMH.

In this sense, it makes no sense to talk about “testing” the EMH. Rather, the question is how quickly prices converge to the limit of market efficiency. In other words, how fast is information diffusion? Our position is that for most things, this is pretty fast!

Here are a few heuristics that support our position:

1. For our purposes, the earlier evidence on the link between real rates and growth is a highly relevant example of market efficiency.

2. There are notable examples of markets seeming to be eerily good at forecasting hard-to-anticipate events:

   1. In the wake of the Challenger explosion, despite no definitive public information being released, the market seems to have identified which firm was responsible.

   2. Economist Armen Alchian observed that the stock price of lithium producers spiked 461% following the public announcement of the first hydrogen bomb tests in 1954, while the prices of producers of other radioactive metals were flat. He circulated a paper within RAND, where he was working, identifying lithium as the material used in the tests, before the paper was suppressed by leadership who were apparently aware that indeed lithium was used. The market was prescient even though zero public information was released about lithium’s usage.

Remember: if real interest rates are wrong, all financial assets are mispriced. If real interest rates “should” rise three percentage points or more, that is easily hundreds of billions of dollars worth of revaluations. It is unlikely that sharp market participants are leaving billions of dollars on the table.

X. If markets are not efficient, you could be earning alpha and philanthropists could be borrowing

While our prior in favor of efficiency is fairly strong, the market could be currently failing to anticipate transformative AI, due to various limits to arbitrage.

However, if you do believe the market is currently wrong about the probability of short timelines, then we now argue there are two courses of action you should consider taking:

1. Bet on real rates rising (“get rich or die trying”)

2. Borrow today, including in order to fund philanthropy (“impatient philanthropy”)

1. Bet on real rates rising (“get rich or die trying”)

Under the logic argued above, if you genuinely believe that AI timelines are short, then you should consider putting your money where your mouth is: bet that real rates will rise when the market updates, and potentially earn a lot of money if markets correct. Shorting (or going underweight) government debt is the simplest way of expressing this view.

Indeed, AI safety researcher Paul Christiano has written publicly that he is (or was) short 30-year government bonds.

If short timelines are your true belief in your heart of hearts, and not merely a belief in a belief, then you should seriously consider how much money you could earn here and what you could do with those resources.

Implementing the trade. For retail investors, betting against treasuries via ETFs is perhaps simplest. Such trades can be done easily with retail brokers, like Schwab.

(i) For example, one could simply short the LTPZ ETF, which holds long-term real US government debt (effective duration: 20 years).

(ii) Alternatively, if you would prefer to avoid engaging in shorting yourself, there are ETFs which will do the shorting for you, with nominal bonds: TBF is an ETF which is short 20+ year treasuries (duration: 18 years); TBT is the same, but levered 2x; and TTT is the same, but levered 3x. There are a number of other similar options. Because these ETFs do the shorting for you, all you need to do is purchase shares of the ETFs.

Back of the envelope estimate. A rough estimate of how much money is on the table, just from shorting the US treasury bond market alone, suggests there is easily $1 trillion in value at stake from betting that rates will rise.

• In response to a 1 percentage point rise in interest rates, the price of a bond falls in percentage terms by its “duration”, to a first-order approximation.

• The average value-weighted duration of (privately-held) US treasuries is approximately 4 years.

• So, to a first-order approximation, if rates rise by 3 percentage points, then the value of treasuries will fall by 12% (that is, 3*4).

• The market cap of (privately-held) treasuries is approximately $17 trillion.

• Thus, if rates rise by 3 percentage points, then the total value of treasuries can be expected to fall by $2.04 trillion (that is, 12%*17 trillion).

• Slightly more than half (55%) of the interest rate sensitivity of the treasury market comes from bonds with maturity beyond 10 years. Assuming that the 3 percentage point rise occurs only at this horizon, and rounding down, we arrive at the $1 trillion estimate.

Alternatively, returning to the LTPZ ETF with its duration of 20 years, a 3 percentage point rise in rates would cause its value to fall by 60%. Using the 3x levered TTT with duration of 18 years, a 3 percentage point rise in rates would imply a mouth-watering cumulative return of 162%.

While fully fleshing out the trade analysis is beyond the scope of this post, this illustration gives an idea of how large the possibilities are.

The alternative to this order-of-magnitude estimate would be to build a complete bond pricing model to estimate more precisely the expected returns of shorting treasuries. This would need to take into account e.g. the convexity of price changes with interest rate movements, the varied maturities of outstanding bonds, and the different varieties of instruments issued by the Treasury. Further refinements would include trading derivatives (e.g. interest rate futures) instead of shorting bonds directly, for capital efficiency, and using leverage to increase expected returns.

Additionally, the analysis could be extended beyond the US government debt market, again since changes to real interest rates would plausibly impact the price of every asset: stocks, commodities, real estate, everything.

(If you would be interested in fully scoping out possible trades, we would be interested in talking.)

Trade risk and foom risk. We want to be clear that – unless you are risk neutral, or can borrow without penalty at the risk-free rate, or believe in short timelines with 100% probability – then such a bet would not be a free lunch: this is not an “arbitrage” in the technical sense of a risk-free profit. One risk is that the market moves in the other direction in the short term, before correcting, and that you are unable to roll over your position for liquidity reasons.

The other risk that could motivate not making this bet is the risk that the market – for some unspecified reason – never has a chance to correct, because (1) transformative AI ends up unaligned and (2) humanity’s conversion into paperclips occurs overnight. This would prevent the market from ever “waking up”.

However, to be clear, expecting this specific scenario requires both:

1. Buying into specific stories about how takeoff will occur: specifically, Yudkowskian foom-type scenarios with fast takeoff.

2. Having a lot of skepticism about the optimization forces pushing financial markets towards informational efficiency.

You should be sure that your beliefs are actually congruent with these requirements, if you want to refuse to bet that real rates will rise. Additionally, we will see that the second suggestion in this section (“impatient philanthropy”) is not affected by the possibility of foom scenarios.

2. Borrow today, including in order to fund philanthropy (“impatient philanthropy”)

If prevailing interest rates are lower than your subjective discount rate – which is the case if you think markets are underestimating prospects for transformative AI – then simple cost-benefit analysis says you should save less or even borrow today.

An illustrative example. As an extreme example to illustrate this argument, imagine that you think that there is a 50% chance that humanity will be extinct next year, and otherwise with certainty you will have the same income next year as you do this year. Suppose the market real interest rate is 0%. That means that if you borrow $10 today, then in expectation you only need to pay $5 off, since 50% of the time you expect to be dead.

It is only if the market real rate is 100% – so that your $10 loan requires paying back $20 next year, or exactly $10 in expectation – that you are indifferent about borrowing. If the market real rate is less than 100%, then you want to borrow. If interest rates are “too low” from your perspective, then on the margin this should encourage you to borrow, or at least save less.

Note that this logic is not affected by whether or not the market will “correct” and real rates will rise before everyone dies, unlike the logic above for trading.

Borrowing to fund philanthropy today. While you may want to borrow today simply to fund wild parties, a natural alternative is: borrow today, locking in “too low” interest rates, in order to fund philanthropy today. For example: to fund AI safety work.

We can call this strategy “impatient philanthropy”, in analogy to the concept of “patient philanthropy”.

This is not a call for philanthropists to radically rethink their cost-benefit analyses. Instead, we merely point out: ensure that your financial planning properly accounts for any difference between your discount rate and the market real rate at which you can borrow. You should not be using the market real rate to do your financial planning. If you have a higher effective discount rate due to your AI timelines, that could imply that you should be borrowing today to fund philanthropic work.

Relationship to impatient philanthropy. The logic here has a similar flavor to Phil Trammell’s “patient philanthropy” argument (Trammell 2021) – but with a sign flipped. Longtermist philanthropists with a zero discount rate, who live in a world with a positive real interest rate, should be willing to save all of their resources for a long time to earn that interest, rather than spending those resources today on philanthropic projects. Short-timeliners have a higher discount rate than the market, and therefore should be impatient philanthropists.

(The point here is not an exact analog to Trammell 2021, because the paper there considers strategic game theoretic considerations and also takes the real rate as exogenous; here, the considerations are not strategic and the endogeneity of the real rate is the critical point.)

XI. Conclusion: outside views vs. inside views & future work

We do not claim to have special technical insight into forecasting the likely timeline for the development of transformative artificial intelligence: we do not present an inside view on AI timelines.

However, we do think that market efficiency provides a powerful outside view for forecasting AI timelines and for making financial decisions. Based on prevailing real interest rates, the market seems to be strongly rejecting timelines of less than ten years, and does not seem to be placing particularly high odds on the development of transformative AI even 30-50 years from now.

We argue that market efficiency is a reasonable benchmark, and consequently, this forecast serves as a useful prior for AI timelines. If markets are wrong, on the other hand, then there is an enormous amount of money on the table from betting that real interest rates will rise. In either case, this market-based approach offers a useful framework: either for forecasting timelines, or for asset allocation.

Opportunities for future work. We could have put 1000 more hours into the empirical side or the model, but, TIABPNAJA. Future work we would be interested in collaborating on or seeing includes:

1. More careful empirical analyses of the relationship between real rates and growth. In particular, (1) analysis of data samples with larger variation in growth rates (e.g. with the Industrial Revolution, China or the East Asian Tigers), where a credible measure of real interest rates can be used; and (2) causally identified estimates of the relationship between real rates and growth, rather than correlations. Measuring historical real rates is the key challenge, and the main reason why we have not tried to address these here.

2. Any empirical analysis of how real rates vary with changing existential risk. Measuring changes in existential risk is the key challenge.

3. Alternative quantitative models on the relationship between real interest rates and growth/x-risk with alternative preference specifications, incomplete markets, or disaster risk.

4. Tests of market forecasting ability at longer time horizons for any outcome of significance; and comparisons of market efficiency at shorter versus longer time horizons.

5. Creation of sufficiently-liquid genuine market instruments for directly measuring outcomes we care about like long-horizon GDP growth: e.g. GDP swaps, GDP-linked bonds, or binary GDP prediction markets. (We emphasize market instruments to distinguish from forecasting platforms like Metaculus or play-money sites like Manifold Markets where the forceful logic of financial market efficiency simply does not hold.)

6. An analysis of the most capital-efficient way to bet on short AI timelines and the possible expected returns (“the greatest trade of all time”).

7. Analysis of the informational content of infinitely-lived assets: e.g. the discount rates embedded in land prices and rental contracts. There is an existing literature related to this topic: [1], [2], [3], [4], [5], [6], [7].

   • This literature estimates risky, nominal discount rates embedded in rental contracts out as far as 1000 years, and finds surprisingly low estimates – certainly less than 10%. This is potentially extremely useful information, though this literature is not without caveats. Among many other things, we cannot have the presumption of informational efficiency in land/rental markets, unlike financial markets, due to severe frictions in these markets (e.g. inability to short sell).

Thanks especially to Leopold Aschenbrenner, Nathan Barnard, Jackson Barkstrom, Joel Becker, Daniele Caratelli, James Chartouni, Tamay Besiroglu, Joel Flynn, James Howe, Chris Hyland, Stephen Malina, Peter McLaughlin, Jackson Mejia, Laura Nicolae, Sam Lazarus, Elliot Lehrer, Jett Pettus, Pradyumna Prasad, Tejas Subramaniam, Karthik Tadepalli, Phil Trammell, and participants at ETGP 2022 for very useful conversations on this topic and/or feedback on drafts.

Appendix 1. Against using stock prices to forecast AI timelines

Link to separate post

Appendix 2. Explaining Tyler Cowen’s Third Law

Link to separate post

Appendix 3. Asset pricing under existential risk: a literature review

Link to Google Doc

Appendix 4. Supplementary Figures

Link to Google Doc