Mike Cafarella
Appearances
Oxide and Friends
Predictions 2025
All right. So one year, I'm... a little bit chagrined by last year's prediction for one year, which imagined a cyberpunk future in an unreasonable 12-month time span. That's probably not going to happen. So I want to make it a little more modest.
Oxide and Friends
Predictions 2025
I'm going to take the opposite side of Simon's and say that the strong agent vision is as ludicrous as everyone says, but weak agents, some weak version of this kind of squishy thing, is actually here to stay, by which I mean inference time, like, post-LLM inference procedures to improve or to have a whole sequence of LLM requests. I think that's actually going to be around for a long time.
Oxide and Friends
Predictions 2025
And it means that like previous LLM interactions that were a little bit lengthy, a little bit annoying, but basically okay, are now going to stretch to minutes long.
Oxide and Friends
Predictions 2025
So I totally agree. Calling it an agent is insane. Letting it have arbitrary, it's like a software module that has no expected termination time. And no like budget of anything. Anyway, that is crazy. But some of the agent programming frameworks exist basically to chain optional sequence. Like, hey, I'm going to I'm going to write some code on your behalf. Then I'm going to try to lint it.
Oxide and Friends
Predictions 2025
So, you know, I agree it's been around like in real world examples for some time, but I think in the last year we saw a kind of abstraction of that pattern for the first time. Totally, yeah. That they call mixture of agents, which I hate the name, but like the basic idea is that you farm it out to...
Oxide and Friends
Predictions 2025
either 10 different models or the same model with very high temperature settings, you get multiple candidate answers and then you try to integrate them. It definitely does better on some tasks, right?
Oxide and Friends
Predictions 2025
Yeah, so I guess what I'm saying is that million dollars, we're now going to burn it at inference time.
Oxide and Friends
Predictions 2025
If they try to spin off the foundry business, does the prediction change?
Oxide and Friends
Predictions 2025
Exactly. No, but the challenge here, though, is that the tech companies themselves can't know if they're training on your data. Like some data, some log shows up at Google and like one of however many people touched it. Google can't make a trustworthy claim even to itself that they didn't train a model on it.
Oxide and Friends
Predictions 2025
So, Mike, do you have a three-year? Okay. My three-year is that the hottest VC startup financing sector is manufacturing in three years. And here's the reason. So... So you've built huge companies out of like kind of comparatively piddling industries like retail and advertising. These are a much smaller fraction of GDP than manufacturing.
Oxide and Friends
Predictions 2025
Manufacturing is one of the few areas where we don't have some straddling tech colossus touching it yet. And also you think about where the AI stuff is. can really strut its stuff, you need a very large number of good but not perfect verdicts, and you need a process that can survive some fraction of them being wrong.
Oxide and Friends
Predictions 2025
So as everyone said, like buying my vacation airline tickets is a bad example, but like doing sensing on quality control for some widget going off the end of the assembly line is a great example of that, right? Where like the increase in sensing and perception could really strut at stuff.
Oxide and Friends
Predictions 2025
And, you know, there's also like various like national security issues that might be involved, but I don't even think you need that. I think just making stuff as like a great area to apply AI and one of the few areas that software hasn't totally beaten to the ground yet is why it's going to come back.
Oxide and Friends
Predictions 2025
Let me sharpen it a little bit then. For hard tech, I don't necessarily mean that they are building science fiction objects that did not exist before. I mean that they are competing with an overseas factory churning out chunks of steel on something.
Oxide and Friends
Predictions 2025
There was a time when Cosmo.com was the canonical example of a... You'll loss on every single transaction. And at some point it was replaced by... by MoviePass, maybe the $200 OpenAI product is now gonna push MoviePass into the dustbin of history and take its rightful place as the product that loses money on every transaction.
Oxide and Friends
Predictions 2025
You could basically DoorDash a Snickers bar for zero delivery cost.
Oxide and Friends
Predictions 2025
All right. My six-year, which I think is optimistic, a lot like Simon's, is the first gene therapy that uses a DNA sequence suggested by an LLM is actually deployed at least in a research hospital, maybe not wild. That is, yeah. Like a CGTA sequence that came from the model goes into a human body.
Oxide and Friends
Predictions 2025
you know, I would say that my rough reading of the models that have been designed for genetic sequence prediction is that like, they're able to achieve kind of remarkable things. I, I, I'm in particular thinking of this Evo model that was released kind of early in 24. I don't know if Simon or others are familiar with this thing. Um,
Oxide and Friends
Predictions 2025
To me, they do this experiment in that model, which is really jaw-dropping. Okay, so the core technical idea here is that the model architecture's a little bit different, because when you're predicting genetic sequences, the alphabet is small, but the sequences are much longer than in natural language, right? So the model architecture's a little bit different.
Oxide and Friends
Predictions 2025
But the experiment that they performed that was really stunning to me was the following. So imagine you have a genetic sequence, and this was just in single-celled organisms. They're not doing this on mammals or anything. Imagine you have a genetic sequence and you intentionally mutate it. So you've got a bunch of different versions of that sequence.
Oxide and Friends
Predictions 2025
And then you try to evaluate its fitness in two different ways. One is that you try to grow it in the lab and see how much it grows. The other is that you look at the probability of that sequence as evaluated by one of these trained models. And now let's imagine you take all of those sequences and you sort them according to those two scores.
Oxide and Friends
Predictions 2025
You sort them according to the observed fitness in the lab, like when you try to grow it in a petri dish, and you also sort it according to the inverse of the probability, meaning like High probable strings go on the top, low probability strings go on the bottom. And what's stunning is that those two sort orders are remarkably highly correlated.
Oxide and Friends
Predictions 2025
So like the ability to just stare at a genetic sequence and actually say something with maybe some predictive accuracy about its real world fitness to me is just absolutely stunning.
Oxide and Friends
Predictions 2025
Let's see. Could I come up with one? That's right. I mean, it feels like an easy parlay from where you got there. I don't know if the following is optimistic or pessimistic. The PlayStation 6 is the last PlayStation. There's never a PlayStation. Ooh.
Oxide and Friends
Predictions 2025
Well, if that was the goal, like mission accomplished. More good books out of it than anyone else. So yeah, sure.