LLMs as we know them are already on their way out. In this video, I break down five breakthroughs that will redefine AI over the next 18 months: Diffusion LLMs (with Stanford’s Stefano Ermon), Power Attention for massive context, hidden/latent-space thinking and private chains of thought, Google’s Nested Learning for continual learning, and the most disruptive shift yet — Continuous Thought Machines (with Jaime Sevilla) as an early break from the Transformer itself.
*https://www.youtube.com/watch?v=h-z71uspNHw
**https://300.ya.ru/v_IvKq4KXh
таймкоды
00:00:00 Введение
- Магистерские программы в области ИИ готовятся к изменениям.
- В ближайшие 18 месяцев ИИ изменится благодаря пяти прорывным разработкам.
- Видео представляет первый взгляд на будущее ИИ.
00:00:53 Диффузионные модели
- Профессор Стефано Херман из Стэнфорда — один из пионеров диффузионных моделей.
- Диффузионные модели превосходят автоагрессивные LLM в генерации изображений и видео.
- Современные LLM генерируют текст по одному слову за раз.
00:01:50 Принцип работы диффузионных моделей
- Диффузионные модели начинают с чистого шума и совершенствуют его итеративно.
- Процесс усовершенствования параллелен, что ускоряет генерацию.
- Стандартные модели ИИ повторяют процесс около десяти раз для генерации одного ответа.
00:03:10 Преимущества диффузионных моделей
- Диффузионные модели быстрее и дешевле благодаря параллельному процессу.
- Встроены функции исправления ошибок, что повышает качество результатов.
- Ожидается, что все LLM в будущем будут основаны на диффузии.
00:03:35 Скорость и эффективность
- Диффузионные модели требуют меньше итераций для генерации.
- При том же уровне качества они работают примерно в десять раз быстрее.
- Разрыв в скорости будет увеличиваться по мере улучшения моделей.
00:04:26 Гибкость
- Диффузионные модели могут изменять любое слово в любой позиции последовательности.
- Они позволяют размещать подсказки в любом месте текста.
- Это делает их более гибкими и удобными для редактирования.
00:05:25 Будущее диффузионных моделей
- Диффузионные языковые модели превосходят автоагрессивные.
- Планируется выпуск модели языка распространения рассуждений.
- Оптимизация качества, скорости и стоимости моделей продолжается.
00:06:19 Коммерческие модели
- Mercury — первая коммерческая модель распространения большого языка от Inception Labs.
- Google представила Gemini Diffusion в мае 2025 года.
- Dream Seven B — модель с открытым исходным кодом.
- 2026 год ожидается как год серьёзного конкурента из класса диффузии.
00:07:03 Будущее трансформаторов
- Трансформаторы теряют доминирование в архитектуре моделей.
- К концу 2025 года специалисты по гиперскейлингу будут работать над субквадратичными базовыми моделями.
- К концу следующего года трансформаторные модели почти не будут использоваться.
00:07:36 Прорыв в искусственном интеллекте
- Джейкоб Бакман, основатель Manifest AI, не смог участвовать в видео.
- Обсуждается значительный скачок в интеллекте и радикальное изменение работы искусственного интеллекта.
- Упоминается основополагающая статья о архитектуре Transformer, которая вызвала бум в области ИИ.
00:08:28 Архитектура Transformer
- Transformer изучает влияние слов друг на друга через миллиарды проб и ошибок.
- Введены три вектора: Q, K и V для каждого слова в последовательности.
- Q задает вопрос о влиянии слов на смысл, K отражает роль слова в предложении, V — истинное значение слова.
00:09:24 Применение Transformer
- Transformer может извлекать шаблоны из любых последовательностей лексем, включая картинки и звуковые волны.
- Механизм лежит в основе всех существующих моделей ИИ.
00:10:23 Платформа Kyo
- Kyo — первая платформа для кодирования, созданная для совместной работы.
- Позволяет легко создавать персонализированные мини-SAAS-проекты без знания кодирования.
- ИИ управляет процессом от пользовательского интерфейса до серверной части и уровня данных.
00:12:18 Проблемы стандартного внимания
- Стандартное внимание требует квадратичных вычислений, что ограничивает количество слов в операции.
- Линейное внимание использует математический трюк для масштабируемости, но теряет часть смысла.
00:13:12 Субквадратичные архитектуры
- Предпринимаются попытки создать архитектуру, сочетающую качество стандартного внимания и масштабируемость линейного внимания.
- Google Titan демонстрирует методы для увеличения количества токенов до десяти миллионов.
00:14:07 Power Attention от Manifest AI
- Power Attention превосходит стандартные трансформаторы при любых размерах контекста.
- Позволяет выбирать наилучший вариант внимания в зависимости от размера контекста.
- При небольшом контексте используется стандартное внимание, при большом — линейное представление.
00:15:02 Эффективность моделей
- Для построения наиболее интеллектуальной модели нужно уравновешивать количество вычислений, параметры и длину контекста.
- Современные модели сбалансированы в небольшой области, где длина контекста составляет около 8 килобайт.
- Эффективность стандартного внимания снижается при длительном контексте.
00:15:41 Кривая масштабирования контекста
- Добавление контекста улучшает предсказания модели.
- Разрыв между моделями в разных масштабах меньше, чем разрыв между началом и концом одной модели при добавлении контекста.
00:16:25 Функция power attention
- Функция power attention динамически настраивает контекстное окно, обеспечивая его сбалансированность.
- Это приводит к созданию более умных моделей с длинным контекстом при меньших вычислительных затратах.
- Ожидается появление субквадратичных архитектур в 2026 году.
00:16:49 Рекуррентные нейронные сети и трансформеры
- Рекуррентные нейронные сети RNNS привлекают внимание возможностью двустороннего потока информации.
- Трансформеры используют фиксированное количество шагов, что ограничивает возможности сети.
- RNNS страдают от ранней конвергенции, в то время как трансформеры вынуждены выбирать из ограниченного словарного запаса.
00:17:47 Мыслить в пространстве идей
- Новое направление исследований позволяет трансформаторам мыслить в пространстве идей, а не слов.
- Модель может использовать слова в особом порядке, изобретать новые лексемы и сжимать информацию высокоэффективными способами.
- OpenAI объявила это направление одним из основных способов улучшения своих моделей.
00:19:25 Контролируемая конфиденциальность
- Идея заключается в сохранении части внутренней логики модели свободной от контроля.
- Это позволяет модели мыслить на уровне, который мы не можем полностью контролировать, но сохраняет способность понимать её внутренний процесс.
- Такой подход может быть более безопасным в долгосрочной перспективе.
00:20:58 Непрерывное обучение
- Непрерывное обучение объединяет различные процессы, включая обучение во время логического вывода и запоминание.
- Алгоритм вложенного обучения Google отделяет обучающие данные от базовой модели, снижая риск неправильного усиления.
- Система предоставляет несколько обновляемых уровней памяти, позволяя ИИ обновлять информацию в реальном времени.
00:24:55 Будущее искусственного интеллекта
- Ожидается, что к концу десятилетия ИИ внесёт вклад в решение открытых математических задач.
- Автоматизация задач повысит продуктивность людей и приведёт к закрытию многих рабочих мест.
- Государствам потребуется больше вычислительной мощности для управления экономикой, связанной с ИИ.
- Доступ к центрам обработки данных станет важным для экономики, как сегодня доступ к интернету, электричеству или воде.
00:26:35 Введение в машину непрерывного мышления
- Трансформатор ещё не достиг своих пределов и может поддерживать тенденции роста.
- Машина непрерывного мышления основана на концепциях мышления и времени, которые современный ИИ игнорирует ради простоты.
- В этой модели мышление и время живут внутри модели как единая динамика.
00:27:30 Проблемы современных моделей
- Современные модели представляют сложные концепции неестественным образом, следуя набору местных правил.
- Они не видят общей картины и не могут обобщать концепции.
00:27:52 Пример спиральной задачи
- Многослойные персептроны RNN и tanh решают спиральную задачу, но не понимают концепцию спирали.
- Слой M представляет спираль в виде спирали, что позволяет правильно экстраполировать данные.
00:30:03 Принципы работы CTM
- В CTM существует несвязанное измерение времени, независимое от размера входных данных.
- Каждый нейрон в CTM — динамическая система, обрабатывающая историю сигналов.
- Сеть вознаграждается за быстроту и правильный ответ.
00:31:51 Результаты тестирования в лабиринте
- Другие архитектуры, такие как LSTM и трансформеры, не смогли продвинуться дальше 20 шагов в лабиринте.
- CTM успешно нашла путь длиной в 100 шагов и обобщила решение на задачи до 800 шагов.
00:32:46 Перспективы и ограничения CTM
- CTM — первая версия новой архитектуры, которая многообещающая, но пока не достаточно мощная для замены трансформеров.
- Основной недостаток CTM — последовательный характер обработки, который затрудняет распараллеливание.
00:34:12 Заключение
- Несмотря на недостатки, CTM может стать началом смены парадигмы в ИИ.
- Ожидается много результатов в ближайшие 18 месяцев.
In this video
Intro
0:00
LLMs as we know them today are already on their way out. In the next 18 months, AI is going to look very different. And
0:07
this video is your first look at what replaces them. There are five breakthroughs behind this change. Some
0:12
from Google Deep Mind and OpenAI, and some from smaller, lesserknown research labs. We’ll start from speed and
0:19
intelligence, then move up to how models think and learn. And we’ll end with the most disruptive one, the first serious
0:26
break from the transformer itself. Part of this video is sponsored by Crew AI. More on them later.
First Technology
0:34
This video has started with a simple question. Are we close to another major leap in AI or are we stuck with single
0:41
digits efficiency gains for now? That’s why I sat down with Professor Stefano at
0:46
the Stanford, one of the pioneers of diffusion models, the backbone of basically every image and video
0:51
generator you see today. Well, I’m super excited about diffusion language models. Uh I was one of the early pioneers of
0:58
diffusion models for image generation. Um when back then in 2019 uh the the
1:05
world of image generation was dominated by GANs generative adversarial network. We came out with this alternative
1:10
approach diffusion models um that showed promise initially in academic benchmarks
1:17
and eventually took over the entire field and now all the best models for
1:22
continuous modalities like images and videos are diffusion based and so what
1:27
I’m seeing is something very similar right now very clear evidence that diffusion language models are superior
1:34
to autoagressive models I’d heard about diffusion language models before but I didn’t really know
1:39
what made them special or why they might be a serious replacement for today’s LLMs. How are diffusion models different
1:47
from the LLMs that we have today? Interestingly, all existing or most of the existing LLMs are autogressive,
1:54
meaning that they generate text or code uh one word, one token at a time, left [clears throat] to right. On the other
2:01
hand, some of the best generative models for images, video, music are diffusion based where the object is generated by
2:09
an iterative refinement process where you start with a rough guess of what the answer should be and then you keep
2:15
refining it and crucially this refinement process is highly parallel. The neural network is able to modify
2:22
multiple tokens at the same time. So current AI models generate the responses one word at a time. They
2:27
produce a single word. Then they take the entire sequence including that new word back in as an input and attach
2:34
another word at the end. This loop repeats over and over until the model finishes its response. And if we include
2:40
the thinking process, a standard AI model that we all use today loops on the order of 10,000 times just to generate a
2:48
single response. Diffusion models, while technically also auto reggressive because they loop as well, generate the
2:54
responses in a very different way. They work the same way they generate an image. They start from pure noise which
3:01
in language is basically a nonsensical response and then they refine that response step by step until they reach
3:08
something reasonable. Why is that better? What we’re saying is that this generative process is significantly more
3:14
efficient. It’s faster. It’s cheaper. And because there is also built-in error correction, the network is trained to
3:21
fix mistakes and that’s how model is used at inference time. but also seeing very high quality results. What we see
3:27
is it’s a future where all LLMs are going to be eventually diffusion [clears throat] based because this is uh
3:34
far superior approach. So there are three main advantages to diffusion models that I didn’t really know at
3:40
least not all of them. Speed and efficiency. Although diffusion models loop as well, they usually need far
3:46
fewer iterations to reach a final answer because the process is more parallel. A reasonable number of steps for language
3:53
is often in the tens or at most low hundreds. In practice, for the same level of quality, diffusion models are
3:59
already about 10 times faster. And this gap is expected to grow as responses and
4:05
thinking processes get longer. Smarter outputs. Auto reggressive LLMs generate responses from left to right, which
4:11
means if they make a mistake, they have to build on top of that. Those first few words which account for only a tiny
4:17
fraction of that compute carry a disproportionate amount of weight and shape everything that comes after. Once
4:23
a token is generated, there is no opportunity to revise it. Generating one board at a time also introduces the
4:29
occasional risk of going completely off the rails as you’ve probably seen with Chad GPT where the model drifts off the
4:36
probability distribution and just gets stuck. Diffusion models on the other hand have the freedom to revise any word
4:42
at any position in the sequence. They can more naturally correct mistakes by working on the entire response
4:48
holistically. Third, flexibility. This is in my opinion the most underrated feature of diffusion models. While auto
4:55
reggressive models expect the prompt to be a prefix, diffusion LLMs allow the prompt to sit at any arbitrary position.
5:02
If you want to rewrite a paragraph in the middle of a document, diffusion models can just edit that spot. Today’s
5:08
LLMs struggle because they don’t naturally see what comes after the edit. So, they have to engage in this awkward
5:16
computation and often rewrite the parts that they are not supposed to touch. Given all of these advantages, where are
5:22
diffusion models right now and when should we expect their rise to the top? The most efficient approach will win.
5:27
And what we’re seeing is that diffusion language models are superior to autogressive models. We’re very excited
5:34
to um further [snorts] scale up this approach and this technology. There’s going to be a
5:39
reasoning diffusion language model coming out pretty soon from inception. Uh there’s still a lot of things that we
5:45
can optimize to further improve the quality, the speed, the cost of diffusion language models. It’s a very
5:51
new technology. Uh but you know in a very short time we’ve been able to catch
5:56
up with other regressive models despite the huge investments that went into into
6:02
that technology just like what we we’re seeing for image [laughter] uh diffusion models you know that have
6:09
been distilled into like very very efficient one step really quick generative models. We’re working on
6:15
similar ideas to make diffusion language models even faster. We’re confident that
6:21
there’s going to be uh that that’s going to be the winning paradigm and the future is going to be uh based on the
6:26
fusion language models. If you like these videos and you are part of the 93% of people who watch my
6:32
channel and aren’t subscribed, please hit the subscribe button. Faceless re-upload channels have a better ratio
6:39
than this. Mercury is considered the first commercialgrade diffusion large language model which is from Inception
6:45
Labs founded by Dr. on Google also introduced Gemini diffusion in May 2025
6:51
and there are other notable candidates like the open-source Dream 7B. 2026 is
6:56
probably the year that we are going to see a serious competitor from the diffusion class.
Second Technology
7:03
The second technology is one that we’ve talked about before. It it’s a bit ridiculous that we have two ways of
7:11
giving data to the model to govern the response we’re getting out of it and we’re putting a billion times more data
7:17
into one of them. The writing is on the wall for me as somebody working in this space. Transformers do not have a lot of
7:23
time left as the dominant architecture. By the end of 2025, this year every hyperscaler will at the very least be
7:29
working on a subquadratic foundation model. By the end of next year, nobody will be using transformer models.
7:35
They’re almost nobody. Jacob Bikman, the founder of Manifest AI, was supposed to be on this video, but unfortunately he couldn’t make it.
7:42
But still, I’ve got some updates. This breakthrough that we are going to talk about is a significant jump in
7:47
intelligence and a radical change in how AI is trained. It fixes one of the most
7:52
fundamental problems in current AI. A flaw baked in from the very beginning.
7:57
Attention is all you need is the seinal paper that triggered this entire AI boom. the paper that introduced the
8:04
transformer architecture. This algorithm was designed to learn how words influence each other through billions of
8:10
trials and errors. They introduced three fundamental vectors of Q, K and V for
8:15
each word in a sequence. For example, in a sequence like the cat sat on the mat. If we wish to understand the meaning of
8:21
SAT, we have a Q vector that is like a question and it asks who in this sentence should influence my meaning.
8:28
Then we have K which is like an answer to Q and it represents the word’s role in the sentence effectively saying here
8:35
is how much I should influence you. And finally we have V which is the actual meaning of the word. Now instead of
8:42
handwriting rules for what words influence each other, we calculate the dotproduct of Q and K for every single
8:48
pair of words in the entire sequence. Then we normalize those results to get an attention score. The higher the
8:55
score, the more influence those words have on each other. What’s the content of Q and K? We don’t actually know. And
9:02
that’s the point. This system is designed to capture the complex structure of language in a form that is
9:07
not reducible to a set of basic rules. For better or worse, QKV combination
9:12
isn’t limited to language processing. It can basically derive patterns out of any sequence of tokens. If it’s a picture,
9:19
it encodes how individual pixels relate and make a circle, an eye, or a cat. If
9:24
it’s a soundwave, it encodes how individual digital signals make a character, a word, or eventually a
9:30
coherent narration of a story. This mechanism is literally at the heart of every AI model out there. So, what’s the
9:37
flaw in attention that is dragging every AI model down? Hold that thought for a second because now you might be
9:42
wondering, how are these advancements relevant to us? They result in awesome products that weren’t possible just a
9:48
couple of years back, like our today’s sponsor, Creo AI. Creo is the world’s first vibe coding platform built for
9:54
your team, allowing you to easily create personalized minias without knowing how
9:59
to code. The great thing about Creo is that it was designed from the ground up for collaborative work. For example,
10:06
when we are working on a collaboration like this, it usually involves multiple rounds of reviews, feedback, and
10:11
adjustments from the brand, the agency, my team, and myself. To streamline the process, I started building an app
10:18
inside Creo. It’s incredibly simple. I just write what I need in natural language, and Curio handles everything
10:24
from the UI to the backend and even the data layer. You can control and customize the entire process, but I
10:31
prefer to keep it simple and let the AI manage all the details. Here’s the final result minutes later. A simple dashboard
10:38
where I can log in as an admin and upload all my assets. Here’s an image for instance. From there, I can share
10:44
the site with a client using the built-in Creo system controlling their access. So, they can only review the
10:51
assets and leave feedback. They visit the platform and they can see all the assets that I have shared with them.
10:57
They can either approve or leave a feedback and request adjustments. Meanwhile, the version shared with my
11:02
team gives them a lot more freedom. They have the ability to inspect the feedback history and even fully customize the
11:08
website itself. That’s how you can collaborate with your team on many SAS projects to increase productivity and
11:14
save both time and money. Now, that’s the foundation. There are a couple of more features that tie everything
11:19
together. Every Creo app is also automatically an MCP endpoint, so you can interact with them through Chad GBD,
11:26
CLA, or any MCP compatible agent. Each app, like the one we just built, comes
11:31
with a built-in AI agent that can guide you, your team, and your clients. For example, I asked my agent, why was the
11:39
image inside the December 15th project rejected the first time around? And the agent correctly identified the initial
11:45
feedback of the middle guy should wear sunglasses. On the client side, I can also ask, how many images are in the
11:51
project December 15th? And the agent correctly responds to my client. There is only one image with this specific
11:58
status. Remember, this agent was just automatically added to my project. I
12:04
didn’t need any extra setups. Creole also allows you to import your designs directly from Figma, provides a massive
12:10
library of MCP servers to get external data, and keeps a history of all of the changes you’ve made with the ability to
12:17
restore any version. So, if you’re ready to streamline your workflow and start building minias projects faster and more
12:23
efficiently, now is the perfect time to try Creo for yourself. Click the link in the description to start for free. And
12:29
if you decide to go pro, use the promo code in the description to get an extra 20% off. Thanks again, Creo, for
12:36
sponsoring this part of the video. The issue with a standard attention is here in this massive table. Because every
12:42
word should be compared to every other word, including itself. There is a harsh limit on how many words you can include
12:49
in this operation. It’s not a matter of resources because there are n byn number
12:54
of calculations which makes this a quadratic operation and there is no hope to keep up as the compute requirements
13:01
explode quadratically while you increase the number of words linearly. In response to this problem, some
13:07
researchers developed linear attention. The linear attention architecture uses a mathematical trick. Instead of comparing
13:14
each word to every other word, it first calculates a representation of the whole sequence. You could think of it as a
13:20
mathematical summary. Then it compares all of the words to this summary. That means if you have five words, you get
13:27
five relations. That’s a linear equation that requires much less computation and memory and is able to scale easily. But
13:34
because some of the meaning is lost in the process of calculating that summary, the output is much lower quality. Now
13:40
there are attempts to build architectures that keep the quality of a standard attention while enjoy the
13:47
scalability of linear attention and we call them sub quadratic architectures. Linear attention is also technically
13:53
subquadratic but just to differentiate Google’s Titans paper showcased some of the techniques they might use in the
13:59
future and the fact that they’ve publicly said that they can scale up to 10 million tokens is a strong signal
14:05
they are working on this in the background. While Google might be hesitant to reveal the fullest story,
14:10
there is another research group that is publicly sharing the new breakthroughs. Manifest AI has released a new mechanism
14:17
called power attention which already outperforms a standard transformers at all context sizes, especially at larger
14:24
context. Imagine that instead of choosing between the table or linear attention, you could always pick the
14:30
best option at any given moment. If the context is still a small, you use a standard attention. So you get full
14:37
precise attention. But once the context grows past a certain limit, instead of doing expensive attention over
14:44
everything, you compress the older part into a linear style representation. It’s like having infinite context. As long as
14:50
it’s efficient to remember every word, you do that. The moment that becomes inefficient, you create a mathematical
14:57
summary of what’s happened so far. This is a really interesting demonstration by Manifest AI. They show that to build the
15:04
most intelligent model possible for any given amount of compute, you need to stay on the diagonal line balancing the
15:10
number of parameters and the context length. Now, if you plot the models that are being trained today, you can see
15:16
there is only a small region in the middle that the models are balanced. And interestingly, the context length this
15:22
corresponds to is exactly the length that companies typically spend most of their compute on around 8K and 16k
15:29
context. If you look at the right side, the efficiency of a standard attention is way off balance for long context and
15:36
companies end up using the same short context window naturally. It’s a bit ridiculous that we have two
15:44
ways of giving data to the model to govern the response we’re getting out of it and we’re putting a billion times
15:51
more data into one of them. Why would we be doing that? Is it because we don’t get benefit from putting data into the
15:56
state? No, we get very concrete and well understood benefits. Uh this is what you call a context scaling curve or
16:03
context-wise loss curve. It basically shows that as you give the model more and more tokens of context, its
16:10
predictions improve. In this case, just measuring the the negative log likelihood, the loss um and
16:15
meaningfully, right? The gap between these two lines, which are models at very different scales, is not as large
16:20
as the gap between the end of one line when you’ve given it a lot of context and the beginning of that same line. Now, Power Attention fixes that problem
16:28
by dynamically adjusting the context window to remain always balanced. And this results in the smartest model and
16:34
the longest context for the amount of compute you’ve spent. 2026 is again very
16:40
likely the year we are going to see early glimpses of subquadratic architectures. The evidence is just too
16:45
overwhelming to ignore. They are just better. For the third technology, we need to
Third Technology
16:51
slightly zoom out and go from inside the machine to the loop outside of it.
16:57
Right before the transformer, there was a massive interest around recurrent neuronet networks. The appeal of RNN’s
17:02
is that the flow of information moves both ways and multiple times. All neurons inform each other forward and
17:09
backward until the network settles into an equilibrium just like the brain. This helps us to hold the thought, play with
17:16
and refine it efficiently until we reach a conclusion. But the problem is RNN’s tend to settle for the first good enough
17:23
answer or as the researchers call it, they suffer from early convergence. The transformer on the other hand made a big
17:30
compromise. Instead of using a naturally looping mechanism, it unrolled everything into a feed forward network
17:36
with a fixed number of steps. You start at the input, push it through a stack of
17:41
layers, and no matter how simple or complex the task is. By the time you reach the final layer, you must produce
17:47
a word before the output layer, everything lives in a latent space, a highdimensional mathematical space that
17:54
can represent extremely rich patterns. But at the very end, the network is forced to collapse all of that richness
18:00
down to a single choice from a limited vocabulary. You can see how looping thousands of times while repeatedly
18:07
compressing deep internal representations into a single token can lead to brittle limited reasoning. It
18:13
just throws away an enormous amount of potential at each step. Now, if the answer isn’t going back to RNN’s or
18:20
sticking with the looping transformer, what is it? There is a new research direction that lets the transformer to
18:26
think in the space of ideas not words instead of forcing the model to constantly pick from a fixed vocabulary
18:32
and even worse to do it in a clean human readable language with proper grammar
18:38
which restricts the model even further. We let it think freely. There are probably many ways to achieve this and
18:44
I’m heavily simplifying here but the core idea is what if we never force the
18:49
model to think in human readable form at all. it would likely start using words in special arrangements, invent new
18:56
tokens, and compress information in highly efficient ways. Even the original Deep Seek R10 paper points out that if
19:03
you don’t force the model to think in a human readable way, it naturally drifts into mixing languages, making up words,
19:09
and using strange symbols. You can even imagine going a step further, dropping words entirely during thinking, and
19:16
letting the model operate in rich vector-like representations. Open AAI has announced this research direction as
19:22
one of the main ways they plan to improve their next models. The idea is to keep parts of the model’s
19:29
internal reasoning free from supervision. So don’t look at it during
19:34
training and thus let it remain representative of the model’s internal
19:40
process. One reason is that we have seen very promising empirical results. Uh this is a technology we employed a lot
19:46
internally. And secondly uh it is scalable and in the sense that
19:51
explicitly we make the scalable objective not adversarial to our ability to monitor the model. And so longterm we
19:57
believe that by preserving some amount of this controlled privacy for the models uh we can retain the ability to
20:03
understand their inner process and we believe this can be a very impactful technique uh as we move towards this
20:09
very capable longunning system. There is a trade-off here. OpenAI’s view is that if you want a model that can
20:16
think and reason at a level we can’t fully follow, you have to accept some opacity and give it the freedom it
20:22
needs. And here is the interesting part. Even if it doesn’t sound like it, Open AI believes this approach can actually
20:29
be safer. Readable chain of thought often push the models into unfaithful thinking, saying one thing while doing
20:36
another. private thought might be more secure in the long run because the model’s internal reasoning stays aligned
20:43
with what it truly believes. They are already developing techniques to then later interpret those unreadable
20:49
thoughts. This one also feels like a 2026 story, especially if GP6 timing
20:54
rumors are true. Now, the fourth innovation is probably
Fourth Technology
21:00
the one that you’ve been waiting for the most. Continue learning. We have to preface this by saying continual
21:07
learning might be one phrase but it bundles up a bunch of different processes. There’s learning at inference
21:12
time. There’s memory and there is the continual part. We are not even sure if this will come from one giant
21:18
breakthrough or if it’s more like a destination that we can reach by improving everything else. But one
21:24
specific innovation that made the entire industry turn its head was Google’s nested learning algorithm. This approach
21:31
isn’t exactly the kind of continual learning you might be imagining like a model that deeply learns about you as an
21:38
individual. It’s more of a larger scale continual learning system that unlocks
21:43
some very cool capabilities at the company level. As of now, Chad GPT, for
21:48
example, can’t learn something from you and then use it later when it’s talking to me. We are already seeing a few
21:55
companies like Perplexity and Cursor experiment with continue learning models that improve by interacting with users.
22:03
This idea has been around for a while and leading labs like OpenAI and Google have been interested in it too. But
22:08
there has been at least one critical issue. As open’s VP of research has pointed out, cursor uh is trying to train some models
22:17
online with their with their users in the loop and it’s theoretically possible to train models in GPT or every other
22:24
product just just responding to the users and reinforce through whatever whatever rewards you get in there. But
22:30
this is not what I am aware at least like not not what opening is doing at the moment. And it is it’s can be great
22:38
but it can be also dangerous because you are not really very much controlling what you are reinforcing in that loop
22:44
and what’s what could what could happen. So yeah at least until until we have a really good safeguards. I don’t think I
22:50
don’t think we should try to do that in anything like as as as complex and large scale as just GPT.
22:55
Nested learning from Google showed a way around part of this problem. Using this algorithm they can separate the learned
23:02
data from the core model itself. The model won’t remember the exact content of every conversation, but the important
23:08
signals can bubble up to the top, cutting through the noise of millions of interactions to extract the most
23:14
valuable information. At least that’s the goal. That reduces some of the risk because the original model remains
23:20
intact, while a separate memory or learning layer accumulates knowledge from public interactions. Another really
23:26
interesting advantage of this architecture is that it introduces multiple updatable memory layers.
23:32
Imagine interacting with an AI that has a real-time updatable sense of what’s trending right now, what’s happened this
23:38
week, and what belongs in the long-term memory without searching around. Of course, the system uses a surprise
23:45
mechanism to decide what’s worth remembering and what isn’t. If that sounds familiar, it’s because these are
23:51
the same researchers who worked on the Titans algorithm, and nested learning is essentially an upgraded version of that
23:56
idea. Up to this point, every breakthrough that we have talked about is an upgrade
Fifth Technology + Future AI Trends
24:03
on top of the transformer. But some of the constraints we are fighting are not bugs. They are part of the design
24:09
itself. So to move past them, we have no choice but to move past the transformer
24:14
entirely. Ironically, this final innovation, the first serious attempt to escape the transformer, comes from one
24:21
of the creators of the transformer himself, Leon Jones, one of the original authors of Attention is All You Need.
24:28
But before that, I wanted to ask the question, what if the transformer remains the dominant or even the only AI
24:35
architecture around? For that, I went to Haimvia, the director of Epoch AI, a nonprofit research institute studying
24:42
the future of artificial intelligence. They have published some of the most widely cited analysis on the future of
24:48
AI and they are a highly respected source in the field followed by people like Andre Karpathy, Daniel Kagelo and
24:54
Dores Patel. In our work at APOAI, we have been tracking the trend of inputs
25:00
in artificial intelligence and the capabilities it has unlocked. We do expect these trends to continue for the
25:05
next [clears throat] 5 years. It is starting to be able to solve questions at the level of at the level of
25:10
researchers in mathematics. And I do expect that before the end of the decade, so by 2030, we will have
25:17
artificial intelligence have contributed to the solution of an open conjecture of mathematics that uh will be recognized
25:24
by mathematicians as a great contribution to the field that will help them uh do their own research on not
25:30
only artificial intelligence but pretty much any topic that you can think of. I do think that this could result in uh
25:37
many people having access to more quality knowledge and many tasks being uh progressively automated in ways that
25:44
help people be more uh be more productive so that we can produce more with less and spend more time with our
25:50
uh with our loved ones. I expect that before the end of the decade uh many jobs will be disrupted and we will have
25:57
an unprecedented levels of unemployment that we will need to handle. artificial intelligence right now it takes many
26:04
resources but in the future will need even more. I will argue that any state that wants to benefit from artificial
26:11
intelligence needs to be thinking seriously about how it is going to achieve the computational uh the
26:17
computational capacity that’s going to be needed to run their economy from here to 10 years. It is possible that access
26:24
to data centers is going to be as important for a state’s uh economy as
26:31
today is access to the internet, the electricity or even water. It sounds like the transformer hasn’t
26:37
hit its limits yet and it can probably sustain these growth trends for a lot longer. But something even more capable
26:45
might be joining the party. The final breakthrough that we are going to talk about and it has some very surprising
26:51
results is the continuous thought machine. What is this thing actually trying to do and how is it different
26:57
from a transformer? The continuous thought machine is built on two fundamental concepts that current AI has
27:03
mostly stripped away for the sake of simplicity. Thinking and time. In this design, thinking and time live inside
27:09
the model as natural unified dynamics. And you’ll see why that matters a lot. When deep learning took off, the messy
27:16
parts of time and continuous thought were stripped away for simplicity and focusing on static pattern recognition
27:22
instead. Now that this static intelligence clearly works, it’s starting to look like the right moment
27:28
to bring some of that original complexity back. Leon Jones explains that today’s models can represent very
27:34
complex concepts, but they do it in a way that is extremely unnatural. It’s like the current architecture never
27:40
truly sees the bigger picture. It just blindly follows a huge collection of local rules. Here’s a beautiful
27:46
demonstration of how today’s AI models are masking their limitations with brute force computation. So there’s actually
27:53
quite an obscure paper that is my poster child for this. There’s an image of it’s
27:58
solving, you know, the classical spiral data set of needing to separate the two classes in the spiral.
28:04
Yes. and it has the decision boundary for a for both a classic RNN uh multi-layer
28:12
perceptron and a tanh multi-layer perceptron. And you can see they both
28:18
solve it, right? Technically, they both solve the problem because they they they classify all the points correctly and
28:24
get a very good test score on this on this very simple data set. And then they
28:31
show you the decision boundary for the for the M layer that they built in this paper. And it’s a spiral.
28:38
The layer represented the spiral as a spiral. Sh shouldn’t we should you know
28:44
if the data is a spiral, shouldn’t we represent it as a spiral? And then if you look back at the decision boundaries
28:51
for for the Spiral and the classic ReLU multi-layer perceptron, it’s clear that you just have these tiny
28:58
little peace-wise linear separations. Um, and that’s what I mean. Yes.
29:05
If you know, if you train these things enough and you push these little
29:11
peace-wise linear boundaries around enough, it can it can fit the spiral and get a
29:19
high accuracy. But there’s no feeling when I look at those that that image
29:24
that the relu version actually understands that it is a spiral,
29:31
right? And when you represent it as a spiral, it actually extrapolates correctly because the spiral just keeps
29:38
going out. If you truly understand the concept of a spiral, you can project and
29:43
apply it in new ways. Change the scale, change the dimensions, extend the pattern further. It is still a spiral.
29:50
This is not literally true, but this visual example gives you an intuition for what’s off. Current models can
29:56
approximate complex concepts yet they still fail to own those concepts in a
30:02
way that generalizes. So how does the continuous thought machine try to fix this? In the CTM there is a decoupled
30:08
time dimension meaning there is a time variable that works independent of the size of the input. So you might have a
30:14
short sentence that needs minutes of thinking. The other part is the neuron level models. In a standard AI, a neuron
30:21
is simple. It takes an input, runs it through an static function and it outputs a single value. In CTM, every
30:28
single neuron is its own dynamical system with internal weights. It processes a history of signals. It’s not
30:34
just interested in the current signal alone, but a pattern of signals that it has received through let’s say the last
30:40
five or 10 internal texts. The result of this combination is the emergence of this incredibly rich time dependent
30:47
activity inside the network. And they are rewarding network for two things. First being quick and second reaching
30:54
the correct answer. It turns out while it is very hard to train this model to converge on anything useful, when you do
31:01
get it right, the network naturally develops dynamic computation. When it needs to classify a simple image, it
31:07
makes a snap decisions. But when it’s trying to solve something complex like a maze or classify a very ambiguous image,
31:15
it takes its time and keeps thinking until it reaches a high confidence answer. Another very interesting
31:21
emergent property is that because the model is self-reflective and tracks its own confidence over time, it actually
31:28
develops an internal reliable confidence variable that naturally solves one of the most insidious problems in current
31:35
AI. Today’s models can’t genuinely tell you how confident they are or how much
31:41
you should trust a given response, but the continuous thought machine has a native very reliable confidence
31:47
variable. Now, with all of that, what are the results and benchmarks? Different architectures were dropped
31:53
into a maze where they don’t have a map or a god view. They are trapped and they have to find their way by just trying
31:59
single steps of left, right, up, and down. This requires the model to think in steps. try about a hundred moves and
32:06
gradually build a map of the maze in its head. The other architectures, the LSTMs and the feed forward networks like
32:13
transformers, they all failed and they couldn’t move past 20 steps. They just
32:18
couldn’t maintain a coherent internal map. But the continuous thought machine, which was trained to find a 100 step
32:24
path to the end of the maze, successfully completed the challenge. Although the model was trained on 100
32:30
step mazes, it could generalize and nearly perfectly solve paths up to 800
32:36
steps. That shows it has actually learned a general procedure for solving mazes, not just memorized patterns for a
32:42
specific number of steps. This is a truly revolutionary architecture. It suggests that many of the surface level
32:48
problems we face come from the underlying architecture itself. So what should we expect from this architecture
32:54
if it’s truly better than transformers? The transformer is an amazing architecture and it has such a pull on
33:00
the industry that it is quite difficult to break free of it because better is not enough. It has to be crushingly more
33:06
powerful. Leon Jones’s explanation of how much inertia there is inside companies is just brilliant. There’s
33:13
actually better, right? There is actually already architectures that have been shown in the research to work
33:20
better than transformers. Okay? but not better enough
33:26
in order to move the entire industry away from such an established architecture where you’re familiar with
33:33
it. You know how to train it. You know how it works. You know how the internals work, right? You know how to fine-tune
33:39
them. You have all this software is already set up for training transformers, fine-tuning transformers,
33:45
inference. So, if you want to move the industry away from that, being better is not good
33:51
enough. It has to be obviously crushingly
33:56
better. Transformers were that much better over RNN’s. Okay, transformers
34:02
where you just applied it to a new problem and it just was so much faster to train and you just got such higher
34:09
accuracy that you just had to move. So CTM is just a version one of this new
34:15
architecture and it is very promising but no one knows if the capability gap is big enough to convince an
34:22
industry-wide shift. Although I should also point out that CTM has a important drawback because of the sequential
34:28
nature of its processing. It’s not really parallelizable. The result of one step is needed before you can take the
34:35
next step and in a world dominated by GPUs that makes CTM a hard cell for most
34:40
companies at the moment. Please follow me on X. I don’t really post there, but it helps a lot with securing guests.
34:46
Diffusion models, subquadratic attention, latent space thinking, continue learning, and continuous
34:53
thought machines, or more generally, transformer replacement. They mark the beginning of a real paradigm shift, and
34:59
we’ll see a lot of the results in the next 18 months. Thanks for watching. Take care.