Dr. Poppy Crum

We aren't developing new resources.

We've got the same cells that are, or I mean, there's neurogenesis, of course, but it's how those are getting allocated.

And, you know, just one quick comment from what we said before when we talk about the homunculus.

The homunculus is an example of a map in the brain, a cortical map.

And maps are important in the brain because they, you know,

allow cells that need to interact to give us specificity, to make us fast, to have, you know, tight reaction times and things, you know, because you've got shorter distance and, you know, things that belong together.

Also, there's a lot of motility in terms of, you know, what those cells respond to potentially dependent on our input.

So the homunculus might be one map, but there are maps all over our brain.

And those maps still have a lot of cross input.

So what you're talking about is, are you having areas where we didn't use to allocate and differentiate

in the specificity of what those cells were doing that are now quite related to the different ways my brain is having to interpret a text message and the subtlety and the nuance of that that actually now I'm

I get faster at.

I have faster reaction times.

I also have faster interpretations.

So am I allocating cells that used to do something else to allow me to have that?

Probably.

But I'm also building, you know, where, like, think about me as a multisensory object that has, you know, I have to integrate information across sight, sound, smell.

to form a holistic object experience, that same sort of integration and pattern is happening now when we communicate in ways that it didn't used to.

So what does that mean?

It means there's a lot more repeatability, a lot faster pattern matching, a lot more integration that is allowing us to go faster.

It goes in lots of different directions.

One thing I did want to say, though, with smartphones specifically and sort of everything, in audio, our ability to carry our lifetime of music and content with us has been because of huge advances in the last 25, 30 years.

and maybe slightly more, around compression algorithms that have enabled us to have really effective what we call perceptual compression, lossy perceptual algorithms, and things like MP3 and, you know, my past work with companies like Bilby.

But whenever you're talking about what's the goal of content compression algorithms, it's to translate the entirety of the experience, the entirety of a signal in, you know,

Thanks, Andy.

with a lot of the information removed, right?

It's great to be here.

But in intelligent ways.

When you look at the way someone is communicating with acronyms and the shorthand that the next generations use to communicate, it is such a rich communication, even though they might just say, LOL.

I mean, it's like, or they might, you know, it's actually a lossy compression that's triggering a huge cognitive experience, right?

Lossy means that in your encoding and decoding of that information, there is actually information that's lost when you decode it.

But hopefully that information is not impacting the perceptual experience.

Imagine I have a song and I want to represent that song.

I could take out, to make my file smaller, I could take out every 500 milliseconds of that.

it would sound really horrible, right?

Or I could be a lot more intelligent and instead basically, you know, if you look at early models like MP3, they're kind of like computational models of the brain.

They stop, you know, they might stop at like the auditory nerve, but they're trying to put a model of how...

Our brain would deal with sound, what we would hear, what we wouldn't.

If this sound's present and it's present at the same time as this sound, then this sound wouldn't be heard, but this sound would be.

So we don't need to spend any of our bits coding this sound.

Instead, we just need to code this one.

And so it becomes an intelligent way for the model and the algorithm of deciding what information needs to be represented and what doesn't to –

create the same, the best perceptual experience, which perceptual meaning what we get to take home.

I think one of the things that's important then, why I think whenever I used to have to teach some of what it means to...

represent a rich experience with minimal data.

You think with minimal information, some of the acronyms that exist in like mobile texting, they've taken on a very rich life in internal life.

Yeah, well, those are simplistic ones, but I think people can have communication now that we can't understand entirely.

Sometimes, but I have to figure it out then.

But yes.

But the point is, that is an example of a lossy compression algorithm that actually has a much richer perceptual experience, right?

And it often needs context, but it's still, you know, you're using few bits of information to try to represent a much richer feeling in a much richer state, right?

And, you know, if you look at

different people, they're going to have, you know, bigger physiological experience dependent on, you know, how they've grown up with that kind of context.

It's good.

It's just different.

Absolutely.

But it's maintained because of the neural connections that are built in those individuals, right?

And that generation.

You get more frequency, and then it comes with different levels of expectation on those.

My daughter's at camp right now, and we were only allowed to write letters for two weeks.

How did that get over?

It's happening.

I mean, I lost their home in a flood years ago.

And one of the only things I saved out of the flood, which is this, and I just brought these back because I got them from my brother, is this communication between one of my ancestors, you know, during the Civil War, like they were courting.

And that was all saved, these letters back and forth between the women.

And, you know, it's like 1865.

Yeah.

I do.

I do.

I had them in my computer bag until I flew up here.

But they were on parchment.

And even though they went through a flood, they didn't run.

And it's this very different era of communication.

And it's wonderful to have that preserved because that doesn't translate, right, through and without that history.

In any case, I'm a huge advocate for integration of technology, but it's, for me, the world is data.

And I do think that way.

It's, you know, and I look at the way my daughter behaves.

I'm like, okay, well, what data is coming in?

And why did she, you know, respond that way?

And, you know, there's an example I can give, but, you know, we were talking about neuroplasticity.

It's like we are the creatures of sort of three things.

One is, you know,

Our sensory systems, how they've evolved, and be it by the intrinsic noise that is degrading our sensory receptors or the external strain.

My brain is going to have access to about the same amount of information as someone with hearing loss if I'm in a very noisy environment.

And so suddenly you've induced, you've compromised the data I have access to.

And then also our sort of experientially established priors, right?

Our priors being, if you think about the brain as sort of a Bayesian model, things aren't always deterministic for us like they are for some creatures.

Our brain's having to take data and make decisions about it and respond to it.

And we should get into, I mean, some of the things that make us the most effective in our environments and just in interacting in the world is how fast and effective we are with dealing with those probabilistic situations, those things where your brain...

It's like probabilistic inference is a great indicator of success in an environment.

And, you know, be it a work environment, be it just, you know, walking down the street.

And that's how do we deal with this, like, data that doesn't just tell us we have to go right or left, but there's a lot of different inputs.

And it's our sort of situational intelligence in the world.

And we can break that down into a lot of different ways.

In any case, we are the products of our sensory systems, our priors, which are the statistics and data we've had up until that moment that our brain's using to weight how it's going to behave and the decisions it makes, but also then our expectations, the context of that that have shaped where we are.

And so there's this funny story.

Like my daughter, when she was two and a half, we're in the planetarium.

at the Smithsonian, and we're watching, I think, one typical film you might watch in a planetarium.

We started in L.A., zoom out on our way to the sun, and we passed that sort of, you know, quintessential NASA image of the Earth.

And it's totally dark and silent, and my daughter, as loud as she possibly could, yells, Minions.

What's going on?

I'm like, oh, yes, of course.

Her experientially established prior of that image is coming from the Universal logo.

And, you know, she never, you know, that says universal.

Yeah, no, I love it.

It was totally valid.

But it was this very, you know, honest and true part of what it is to be human.

Like each of us is experiencing very different, you know, having very different experiences of the same physical information.

And we need to recognize that.

I do think we're much more plastic than we talk about or we realize in our daily lives.

But it is driven by our exposures and our priors and our sensory systems.

It's sort of that trifecta.

and our expectations at the moment.

And once you unpack that, you really start to understand and appreciate the influence of technology.

Now, I am a huge advocate for technology improving us as humans, but also improving the data we have to make better decisions and the sort of insights that drive us.

At the same time, I think sometimes we're penny wise, pound foolish with how we use technology.

And the quick things that make us faster can also make us dumber and take away our cognitive capabilities.

And where you'll end up with those that are using the technologies might be to write papers all the time or maybe, well, and we can talk about that more, are putting themselves in a place where

they are going to be compromised trying to do anything without that technology and also in terms of their learning of that data, that information.

And so you start even ending up with bigger differentiations and cognitive capabilities by how you use a tool, a technology tool to make you better or faster or not.

And just to your point about creating robots, the more we create robots, there's neuroplasticity that comes with using robots as humans when we use them in partnerships or as tools to accelerate our capabilities.

One of my sort of things I've always done is teach at Stanford.

We also have that in common.

But my class there is called neuroplasticity and video gaming.

And I'm a neurophysiologist, but I'm really a technologist.

I like buildings.

I like innovation across many domains.

And while that class says video gaming, it's really more – well, video games are powerful in the sense that there's this sort of closed loop environment.

You give feedback.

You get data on your performance.

But you get to control that and know what you randomize, how you build.

And what our aim is in that class is to build technology and games with an understanding of the neural circuits you're impacting and how you want to what you want to train.

I'll have students that are musicians.

I'll have students that are computer scientists.

I'll have students that are.

you know, some of Samford's top athletes.

I've had a number of their top athletes go through my course.

And it's always focused on, OK, there's some aspect of human performance I want to dissect, and I want to really amplify the sensitivity or the access to that type of learning in a closed-loop way.

just for anyone that isn't familiar with the role or the history of gaming in the neuroscience space, you know, there's been some great papers in the past.

Take a gamer versus a non-gamer, just to start with, someone self-identified.

A typical gamer actually has what we would call more sensitive, and this is your domain, so you can counter me on this anytime, but, you know, contrast sensitivity functions.

And, like, a contrast sensitivity function is, you know, ability to see things

edges and differentiation in a visual landscape.

They can see faster and they're more sensitive to that sort of differentiation than someone who says, I'm not a video game player or self-identifies that way.

Right.

So neuroplasticity, where I resonate with it a lot, is trying to understand, and this is what I've done a lot of in my career, is thinking about building and developing technologies, but with an understanding of how they shape our brain.

Well, you're spot on because then you take someone who, that self-identified non-gamer, make them play 40 hours of Call of Duty and now their contrast sensitivity looks like a video game player and it persists.

You know, go back, measure them a year later.

But, you know, 40 hours of playing Call of Duty and I see the world differently, not just in my video game.

I actually have foundational shifts in how I experience the world that give me greater sensitivity to my situational awareness, my situational intelligence.

Yeah.

Yeah, because that's a low-level processing capability.

I love intersecting those when you can.

But what's even, I think, more interesting is you also – and these were some – this was a great study by Alex Puget and Daphne DeBellier where it's not just the contrast sensitivity.

It's let's go to that next level where we were talking about Bayesian, like, probabilistic decisions where things aren't deterministic.

And for a video game player – and I can train this –

They're going to make the same decisions as a non-video game player in those probabilistic inferential situations, but they're going to do it a lot faster.

And so that edge, that ability to get access to that information is phenomenal, I think.

And when you can tap into that, that becomes a very powerful thing.

So like probabilistic inference goes up when I've played 40 hours of Call of Duty.

But...

Then what I like to do is take it and say, okay, here's a training environment.

I had a couple of Stanford's top soccer players in my course this year and we got – our focus was, okay, what data do you not have and how can we build a closed-loop environment and make it something so that you're gaining better neurological –

access to your performance based on data like my acceleration, my velocity, not at the end of my two-hour practice, but in real time and getting auditory feedback so that I'm actually tapping into more neural training.

So we had

sensors, you know, like on their calves that were measuring acceleration velocity and able to give us feedback in real time as they were doing, you know, a sort of somewhat gamified training.

I don't want to use gamified.

It's overused.

But let's say...

It felt like fun environment, but it's also based on computation of that acceleration data and what their targets were.

Everything we engage with in our daily lives, whether it's the statistics of our environments and our contexts,

It's feeding them different sonic cues so that they're building that resolution.

When I say resolution, what I mean is...

Especially as a novice, I can't tell the difference between whether I've accelerated successfully or not.

But if you give me more gradation in the feedback that I get with that sort of that closed-loop behavior, I start to – my neural representation of that is going to start differentiating more.

So with that, that's where the auditory feedback.

So they're getting that in real time.

And you build that kind of closed-loop environment that helps build that –

create greater resolution in the brain and greater sensitivity to differentiation.

or the technologies we use on a daily basis are shaping our brains in ways through neuroplasticity.

First, I'll say this.

Number one is having good swimming coaches.

Parents who are data centric and really like building technologies are sometimes maybe can be red herring distractions, but hopefully not.

So, for example, you go and train with elite athletes.

And if you go to a lot of swimming camps or training programs, it's always about work with cameras and what they're recording you.

They're assessing your strokes.

But the point is what – I mean –

You can use, and I did this, you know, knowing the things that the coaches, or frankly, you can go online and learn some of those things that matter to different strokes.

You can use perplexity labs, use some of these.

Yeah, yeah.

And you can build, quickly build a computer vision app that is giving you data analytics on your strokes and in real time.

In this case, I'm using mobile phone, so I'm doing everything above.

Right, or breaststroke or butterfly.

Some more than others, some we know as we age are very dependent on how attentive and engaged we are as opposed to passively just consuming and changing.

There's a lot of core things that maybe you want to care about.

Backstroke and freestyle.

And I am not.

We used to run.

I know you're a good runner, but I'm a runner.

I'm a rock climber, less a swimmer.

But, you know, things like the roll or how high they're coming above the water, what's your – you know, what's your velocity on a – you know, you can get actually very sophisticated once you have the data, right?

And, you know, what's your velocity on entrance?

How much – you know, how far in front of your head is your arm coming in?

How – you know, what is – maybe there's – again, maybe there are things that you – you know, are obvious, which is you want to know, you know, how –

consistent are your strokes and your cadence across, you know, the pool.

So you don't just have your speed, you suddenly have access to what I would call, and you'll hear me use this a lot, better resolution, but also a lot more analytics that can give you insight.

Now, important thing here is, you know, my 10-year-old is not going to, I'm not going to go tell my 10-year-old that she needs to change her velocity on this.

Yeah.

But it gives me information that I can at least understand and help her know how something is going and how consistent she is on certain things that her coaches have told her to do.

You know, and what I love about the idea is, look, this isn't just for you.

The ease of getting access to the type of data and information that would previously – and, I mean, I do code in a lot of areas, but you don't have to do that anymore to build these apps.

In fact, you shouldn't.

You should leverage, you know, AI for development of applications.

these types of tools.

But we are in a place where everyone, I believe, needs to be thinking more about how the technologies they're using, especially in the age of AI and immersive technologies,

You'd use AI to build an app that would allow you to do that so that you would have then access to that, whatever the data is that you want to do.

Yeah, so in that case, you're trying to do better stroke analytics and understand things as you move forward.

You could do the same thing for running, for gait.

You could do, you know, in a work environment, you can understand a lot more about where vulnerabilities are, where weaknesses are.

There are sort of two different places where I see this type of AI acceleration and tool building really having major impact.

It's on sort of democratizing data analytics and information that would normally be reserved for the elite.

to everyone that's really engaged.

And that has a huge impact on improving performance because that kind of data is really useful in understanding learning.

It also has applications for when you're in a work environment and you're trying to better understand success in that environment, in some process or skill of what you're doing, you can gain different analytics than you otherwise would.

in ways that become much more successful, but also give you

new data to think about with regard to what I would call a digital twin.

And when I use the word digital twin, the goal of a digital twin is not to digitize and represent a physical system in its entirety.

It's to gain, use different interoperable, meaning data sets coming from different sources to gain insights, you know, digitized data of a physical system or a physical environment or physical world, be it a hospital, be it airplanes, be it my body, be it my fish tank.

to give me insights that are continuous and in real time that I otherwise wouldn't be able to gain access to.

how they are shaping or architecting our brains as we move forward.

You go to any Neuroscience 101 medical school textbook, and you'll see a few pages on something called the homunculus.

Now, what is the homunculus?

It's a data representation, but it'll be this sort of funny-looking creature when you see it.

But that...

picture of this sort of distorted human that you're looking at is really just a data representation of how many cells in your brain are helping, are coding and representing information for your sense of touch, right?

Whether you're building apps or you're building a tool, you're using it as a tool that's helping you optimize your cognition and find your weaknesses, but also give you feedback on your performance and accelerate your learning in this, right?

But you're still putting in the effort to learn.

And I think even the ways that I'm using it, computer vision with mobile devices, AI is a huge opportunity and tool.

And that image though, and this is where things get kind of funny, that image comes from Wilder Penfield back in the 40s.

Using the cameras and the data that you've collected to have much more sophisticated input is huge.

But in both of those cases, you're shaping cognition.

You're shaping – you're using data to enrich what you can know.

And AI is just incredibly powerful and a great opportunity in those spaces.

The place where I think it is –

And I sort of separate it into literally just two categories.

Maybe that's too simplistic.

Am I using – and this is true for any tool, not just AI.

But am I using the tool, am I using the technology in a way to make me smarter about – and let me have more information and make me more effective but also cognitively more effective, gain different insights?

Or am I using it to replace –

replace a cognitive skill I've done before to be faster.

And it doesn't mean you don't want to do those things.

I mean, GPS in our car is a perfect example of a place where we're replacing a cognitive tool to make me faster and more effective.

And frankly, you take away your GPS in a city you drive around and we're not very good.

He recorded the somatosensory

Absolutely.

No.

And I mean, they had more gray matter in their hippocampus.

And we know that.

And you look at them today and they don't have to have that because the people in their back seats have more data, have more information, have eyes from the sky.

cells of patients just before they were to have surgery for epilepsy and such.

I mean, satellite data is so huge in our success in the future.

And, you know, it can anticipate the things that locally you can't.

And so it's been replaced.

But it still means when you lose that data, you don't expect yourself to have the same spatial navigation of that environment without it, right?

But you're still trying to gain insight and data and information that's making me a more effective human.

And since we don't have pain receptors in our cortex, you could have this awake human and be able to touch different parts of their brain and ask them to report what sensation they felt on their bodies.

There was a recent paper from MIT that I actually, it was, it is very much what I spend a lot of my time talking about and thinking about.

The upshot of the paper first was that people, there's a lot less mental process or cognitive process that goes on for people when they use LLMs to write papers.

And they have, they don't have the same transfer and they don't really learn the information.

Surprise, surprise.

So there were three different categories, people who had to write the papers, you know, just with or using their brain only.

And that would be case one.

Case two would be I get to use search engines, which would be sort of a middle ground.

Again, these are, you know, rough categories.

And then a third would be I use LLMs to write my paper.

And they're looking at, you know, sort of what kind of transfer happened, what –

They were measuring neural response, so they were using EEG to look at neural patterns across the brain to understand how much neural engagement happened during the writing of the papers and during the whole process and then what they could do with that, what they knew about that information down the road.

It's a really nice paper, so I don't want to diminish it in any way by summarizing it.

But what I think is a really important upshot of that paper and also just how we talk about it that I liked was I talk a lot about cognitive load always.

And you can measure cognitive load in the diameter of your pupil and body posture and how people are thinking.

And so he mapped that part of their cortex, and then that's how we ended up with the homunculus.

It's really how hard is my brain working right now to solve a problem or just in my context.

And there are a lot of different cues we give off as humans that tell us when we're under states of different load and cognitively and whether we are aware of it or not.

And there's something called cognitive load theory that breaks down sort of what happens when our brains are under states of load.

And that load can come from sort of three different places.

It might be coming from intrinsic energy.

what you would call intrinsic information, and this is all during learning.

The intrinsic load, cognitive load, would be from the difficulty of the material I'm trying to understand.

Really, some things are easy to learn, some things are a lot harder, and that's intrinsic load.

Extraneous load would be the load that comes from how the information is presented.

Is it poorly taught?

Is it poorly organized?

Or also even the environment.

If I'm trying to learn something auditorily and it's noisy, that's introducing extraneous cognitive load, right?

And you'll see, you know, it'll have bigger lips.

It's not the information itself, but it's because of everything else happening with that data.

It'll have, you know, smaller parts of your back and the areas where you just don't have the same sensitivities.

And then the third is germane cognitive load.

And that's the load that is used in my brain to build mental schemas, to build, to organize that information, to really develop a representation of what that information is that I'm taking in.

And that germane cognitive load, that's the work, right?

And if you don't have germane cognitive load, you don't have learning, really.

And what they found is basically the germane cognitive load is what gets impacted most by using LLMs, which is, I mean, it's a very obvious thing.

Like that's...

Using LLMs, you're not engaging the mental effort to build cognitive schema, to build neural schemas and sort of the mental representation of the information that you can interact with it later and you have access to it later.

And this is really important because without that, you won't be as intelligent on that topic, that's for sure, down the road.

Let me give two examples.

I have a doctor, I have a lawyer.

And both of them use LLMs extensively for searches, say, or for building information.

In one case, it's for patient aggregation of patient data.

Well, fast forward to today.

In another case, it's for history of case files.

And that is the GPS that's happening in those spaces because those are the tools that are quickly adopted.

Where you have someone that is maybe came from a different world, has learned that information, has gone and worked with data in a different way, worked their representation.

When you look at that homunculus, one of the things I always will ask people to think about is, you know, what's wrong with this image?

That information is going to be better extrapolation.

It's going to be better at generalization.

It's going to be better at seeing patterns that would exist.

The brain that has done everything through LLMs is going to be in a place where they will get the answer.

for that relevant task or using the tools they have, but you're not the same level of richness and depth of information or generalization or extrapolation for those topics as someone that has learned in a different way.

There's a generational

difference in understanding, not because they don't have the same information, but there is an acknowledgement that there's a gap, even though we're getting to the same place as fast.

And that's because of the learning that's happened.

Absolutely.

The cognitive load.

Like you've got to do the work.

Your brain has to.

And, you know, what was beautiful about your descriptions, Andy, is when you were talking about how you were using it, which I love, you know, to test yourself, find your vulnerabilities is, you know, and actually in the paper on MIT, which I think, again, these are things that are somewhat obvious, but we just have to, I think we have to talk about them more, is people with higher competency on the topic use the tools in ways that still engage more germane.

You know, this is an image from 1940 that is still in every textbook.

cognitive load, but helped accelerate their learning.

It's, you know, where is the biggest vulnerability and gap?

It's when it's especially in areas and topics where you're trying to learn a new domain fast or you're under pressure and you're not putting in the germane effort or you're not using the tools that you have access to that AI can enable.

You're not using them to amplify your cognitive gain, but instead to deliver something faster, more rapid, and then walking away from it.

And any Stanford student will look at it and they'll immediately say, well, the thumb should be bigger because we do this all day long.

And I've got more sensitivity in my fingers because I'm always typing on my mobile device, which is absolutely true.

A couple of comments.

I love the peach pie example.

Making bread is another example of that.

And in the 90s, everyone I knew when they graduated from high school got a bread maker that was shaped like a box and created this loaf of bread with a giant rod through it.

And it was the graduation gift for many years.

Or maybe they'll say something like, well, the ankles are the same size and we drive cars now a lot more than we did in the 40s.

you know, you don't see those anymore.

And, you know, if you even look at what happened with like the millennial generation in the last, you know, in the last five years, especially during the pandemic, suddenly bread making and sourdough, that became a thing.

What's the difference?

You know, you've got bread, it's warm.

It's, you know, with the bread maker, it's fresh.

And it is not at all desired relative to bread that takes a long period of time and is tactile and in the process and the making of it.

And, you know,

you know, is clearly much more onerous than the other in its process of development.

I think the key part is it's in the appreciation of the bread, the process is part of it.

And that process is development of sort of the germane knowledge and the commitment and connection to that humanness of development, but also the tactile commitment, the work that went into it is really appreciated in the same way that that peach pie for...

one, comes with that whole time series of data that wasn't just about my taste, but was also smell, also physical, also visual, and saw the process evolve and build a different prior going into that experience.

And that is, I think, part of richness of human experience.

Or maybe if I live in a different part of the world, I drive on one side versus the other.

Will it be part of

The richness of how humans interact with AI?

Absolutely.

Or interact with robots?

Absolutely.

So it's what are the relationships we're building and how are they, you know, how integrated are these tools, these, you know, companions, whatever they may be in our existence will shape us in different ways.

What I am particularly, I guess, bullish on and excited for is the robot that optimizes my health, my comfort, my intent in my environment, be it in the cabin of a car, be it in my rooms, my spaces.

And in a few years, we probably won't be driving and those resources get optimized elsewhere.

And I think we already have some of these.

Like it's the point where HVAC systems actually get sexy, right?

Not sexy in that sense, but they're actually really interesting because they are the heart of, you know.

Heating, ventilation.

But you think about a thermostat, you know, a thermostat right now is optimizing for an AI thermostat optimizing for my behavior.

But it's trying to save me resources, trying to save me money, but it doesn't know if I'm hot or cold.

It doesn't know, to your point, my intent, what I'm trying to do at that moment.

And this speaks more to a lot of the things you've studied in the past.

It doesn't know what my optimal state is for my goal in that moment in time.

Right.

But it can very easily, frankly, you know, it can talk to me, but it can also know how my state of my body right now and what is going, you know, if it's 1 a.m.

and I really need to work on a paper, you know, my house should not get cold, but it also should be very, it should, for me, it shouldn't.

So what the homunculus is, is it's a representation of how our brain has allocated resources to help us be successful.

I know for some people it should.

Home, vehicle, yes.

Because it needs to treat me as a dynamic time series.

It needs to understand the context of everything that's driving my state internally.

There's everything that's driving my state in my local environment, meaning my home or my car.

And then there's what's driving my state externally from my external environment.

And we're in a place where those things are rarely treated interacting together for the optimization and the dynamic interactions that happen.

But we can know these things.

We can know so much about the human state from non-contact sensors.

And those resources are the limited cells we have that support whatever we need to flourish in our world.

Well, you want to have your own choice about how you sleep.

Yeah.

Given the date, it's helping you have that.

And the beauty of that is when you develop expertise, you develop more support, more resources go to helping you do that thing.

But they also get more specific.

They develop more specificity so that, you know, I might have suddenly a lot more cells in my brain devoted to helping me.

Can AI teach it to us?

I would say AI is part of the story.

But before we get AI, we need better, more data.

Not just me, right?

So maybe I am very focused right now.

But without my belief, and this is my perspective, is imagine I'm very focused right now.

I need to know the context of my environment that's driving that.

Like what's in that environment?

Is it internal focus that's gotten me there already?

What is my environment?

What is that external environment?

So the understanding my awake state for me is very dependent on the data and interactions that happen from these different environments.

Let me give an example.

Like if I'm in my home or I'm in a, say I'm in a vehicle, right?

And you are measuring information about me and you know I'm under stress or you know I'm experiencing joy or heightens attention right now.

You know, I'm a violinist and my—

Some different states you may want to have my home or my system react to mitigate.

Potentially, it will make adjustments, but not necessarily right for you.

That's an important part is optimizing for personalization and how a system responds.

And, you know, it can make any home, an HVAC system or the internal state of a vehicle is going to adjust, you know, sound, background sound, music.

It's going to adjust, you know, whatever weather it can, haptic feedback, temperature, lighting, you know, any number of, you know,

well, my left hand, my right hemisphere, my somatosensory cortex, I'm going to have a lot more cells that are helping me feel my fingers and the tips of everything so that I can be fluid and more virtuosic.

position of your chair, dynamics of what's in your space, all of these different systems in my home or my other, what if my vehicle, or some other system can react, right?

But the important thing is how you react is going to shift me.

And the goal is to not measure me, but to

actually intersect with my state and move it in some direction, right?

Right.

Measurement or modification.

Measurement is critical.

But measurement not just of me, but also of my environment and understanding of the external environment.

This is where things like Earth observation and understanding, we're getting to a place where we're getting image...

really good image quality data from the satellites that are going in the sky at much lower distances so that you now have faster reaction times between technologies and the information they have to understand and be dynamic with them, right?

Sure.

Weather predictions, car environment, things happening.

The traffic problem gets resolved when you have autonomous vehicles in ways that don't have like the human side of things.

It does.

Only autonomous vehicles, you don't have traffic in the ways that you do with human behavior.

It is that injection from the human system that is interrupting all the models.

I think the world right now, we think about wearables a lot.

Wearables track us.

You have smart mattresses, which are wonderful for understanding.

But that means I have more cells, but they're more specified.

So there's so much you learn from a smart mattress and ways of also both measuring as well as intervening to optimize your sleep, which is the beauty.

And it's this nice, incredible period of time where you can measure so many things.

But, you know, in our home, so I use the example of a thermostat, right?

It's pretty, you know, frankly dumb about what my goals are or what I'm trying to do at that moment in time, but it doesn't have to be.

And there are, you know, there's a company, Passive Logic, I love them.

They actually have, I think, some of the smartest digital twin HVAC systems, but...

They're giving me more sensitivity.

You know, their sensors measure things like sound.

They measure carbon dioxide, your carbon, your CO2 levels.

Like when we breathe, we give off CO2, you know.

So imagine, you know, there's a dynamic mixture of acetone, isoprene, and carbon dioxide that's constantly exchanging when my, you know, when I get stressed or when I'm feeling, you know, depressed.

They're giving me more data that's differentiated.

happiness or suspense in my state.

And that dynamic sort of cocktail mixture that's in my breath is both an indicator of my state, but it's also something that it's just the spaces around me

And that's what my brain needs, and that's what my brain's responding to.

you know, have more information to contribute about how I'm feeling and can also be part of that solution in ways that I don't have to have things on my body, right?

So I have sensors now that can measure CO2.

You can watch my TED Talk.

I have given examples.

We brought people in when I was at Dolby and had them watching Free Solo, you know, the Alex Honnold movie where they're climbing El Capo.

So carbon dioxide is heavier than air.

So we could measure carbon dioxide from just tubes on the ground and you could get the real-time differential of CO2 in there.

And so when we think about that,

No.

But I mean, I like to say we broadcast how we're feeling, right?

And we do that wherever we are.

And in this, you could look at the time series of carbon dioxide levels and be able to, you know,

know what was happening in the film or in the movie without actually having it annotated.

You could tell where he summited, where he had to abandon his climb, where he hurt his ankle.

Absolutely.

You know, my practice as a musician versus my practice playing video games, all of these things influence our brain and influence our plasticity.

There's another study, I forget who the authors are, and they've got different audiences watching Hunger Games.

And different days, different people, you can tell exactly where Katniss's dress catches on fire.

And, you know, it's like we really are sort of, you know, it's like digital exhaust of how we're feeling.

But, you know, and our thermals, we, you know, radiate the things we're feeling.

I'm very bullish on the power of, you know, our eye and representing our cognitive load, our stressors.

Our eye, yes, like the diameter.

Our, yes.

Yes, yes, yes.

You know, back when I was a physiologist, I always, you know, I've worked with a lot of species on, you know, understanding information processing internally in cells, but also then I would very often use pupilometry as an indicator of, you know, perceptual engagement and experience.

Yeah, more arousal, cognitive load, or obviously lighting changes.

But the thing that's changing from-

20 years ago, 15 years ago, it was very expensive to track the kind of resolution and data to, you know, leverage all of those autonomic nervous system, you know, deterministic responses, because those ones are deterministic and not probabilistic, right?

Those are the ones that it's like the body's reacting, even if the brain doesn't say anything about it.

And, but today we can do that with it.

I mean, do it.

Well, we can do it right now with, you know,

open source software on our laptops or our mobile devices, right?

And every pair of smart glasses will be tracking this information when we wear them.

So it becomes a channel of data.

And, you know, it may be an ambiguous signature in the sense that there's, you know, changes in lighting, there's changes, am I aroused or am I...

Now, where things get kind of interesting to me and sort of my obsession on that side is every time we engage with a technology, it's going to shape our brain, right?

Yeah, no, and it's a lot of the consumer devices and just computation we can do from, you know, whether it's cameras or exhalant or, you know, other data in our environments that tell us about our physical state and some of these situations that you're talking about, a lot of the, I mean, why isn't it happening?

It's both, you know, our environments, but our environments are changing.

A lot of the reasons are simply the regulatory process is antiquated and not up to keeping up with the acceleration of innovation that's happening, you know, getting things through the FDA and,

even if they're deemed, you know,

in the same ballpark and supposed to move fast, the regulatory costs and processes is really high.

And you end up many years down the road from when the capability and the data and technology actually should have arisen to be used in a hospital or to be used in a place where you actually have that kind of appreciation for the data and use.

Those are shaping who we are.

The consumer-grade technology

devices for tracking of data of our biological processes are on par and in many cases surpassed the medical grade devices.

And that's because they just have, but then they will have to bill what they do and what they're tracking in some way that is consumer, you know, is not making the medical claims to allow them to be able to be

You know, I think you can look at people's hearing thresholds and predict what city they live in.

continue to move forward in those spaces.

But there's no question that that's a big part of what holds back the availability of a lot of these devices and capabilities.

Absolutely, yes.

I'm an advocate for fewer things on, you know, not having all this stuff on our bodies.

There's so much we can get out of the computer vision side, you know, from, you know, the cameras in our spaces and how they're supporting us in our rooms, in our, the sensors on our, in our, you know, I brought up HVAC systems earlier.

So now you've got.

effectively a digital twin and sensors that are tracking my metabolic rates just in my space.

They're tracking carbon dioxide.

They're tracking sound.

You're getting context because of that.

You're getting intelligence, and now you're able—

to start having more information from what's happening in my environment.

The same is true in my vehicle.

You can tell whether I'm stressed or how I'm feeling just by the posture I have sitting in my car, right?

And you need AI.

This is AI interpretation of data.

But what's driving that posture might be coming from also an understanding of what else is happening in that environment.

So it's suddenly this – with contextual intelligence, AI-driven understanding of what's happening in that space that's driving the state of me and how do I –

I keep leaning to the side because I'm thinking about the way I move and sit.

It's a proxy for what's actually happening inside me.

And then you've also got data around me coming from my environment.

What's happening if I'm driving a car or what's happening in my home in the weather.

And not just threats that might be outside and noise that's happening not inside the space, but things that give context to have more intelligence with the systems we have.

I'm a huge believer in you don't – we aren't anywhere until we have integration of those systems between the body, the local environment, and the external environment.

And we're finally at a place where AI can help us start integrating that data.

In terms of wearables, though –

So obviously, some of the big companies, we've got the watch we have on our hand has a lot of information that is very relevant to our bodies.

The devices we put in our ears, you may not realize, but, you know, a dime-sized patch in your concha, we can use, we can know information.

heart rate, blood oxygen level, because the electrical signature that your eye produces when it moves back and forth, we can know what you're looking at just from measuring a signature, measuring your...

electrooculogram in your ear.

We can measure EEG, electroencephalograms.

You can also get, you know, eye movements out of electroencephalograms, but you can get attention.

You can know what people are attending to based on signatures in their ear.

So our earbuds, you know, that becomes sort of a window to our state.

And you've got a number of companies working on that right now.

You know, so do we need to wear lots of different sensors?

No.

Do we need to have the sensors, the data we have, whether it's on our bodies or off our bodies, be able to work together and not be proprietary to just one company, but to be able to integrate great with other companies?

That becomes really important.

You need integrative systems so that the...

The data they have can interact with the systems that surround you or surround my spaces or the mattress I'm sleeping on, right?

Because you've had a lot of specialty of design come from different developers.

And that's partly been a product of, again, the FDA and the regulatory pathways because of the cost of development here.

It tends to move companies towards specialization unless they're very large.

But where we're at today is we're getting to a point where you're going to start seeing a lot of this data get integrated, I think.

And by all means, hopefully we're not going to be wearing a lot of things on our bodies.

I sure as heck won't.

The more we put on our bodies, it affects our gait.

It has ramifications in so many different ways.

When I got here, I was talking to some of the people that work with you, and they're like, well, what wearables do you wear?

And I actually don't wear many at all.

And I have worn rings.

I've worn watches at different times.

But for me, the importance is the point at which I get insights that I am a big believer in.

As little on my body as possible when it comes to wearables.

One interesting company that I think is worth mentioning is Pison.

And Pison, again, they've got a form factor that's like a Timex watch or they're partnered with Timex, but they're measuring –

Are you familiar with PISM?

Okay.

So they're measuring psychomotor vigilance.

So, you know, really trying to understand it's like an ENG, electro-neural modulation, and they're trying to understand fatigue and neural attentiveness in a way that is, you know, continuous and useful for, say, high-risk operations or...

training, whether it be it in sport.

But what I like about it is it's actually trying to get at a higher level cognitive state from the biometrics that you're measuring.

And that to me is an exciting, really exciting direction is when you're actually doing something that you could make a decision

In part, it can be both, right?

about how I engage in my work or how I engage in my training or my life based on that data about my cognitive state and how effective I'm going to be.

And then I can start associating that data with the other data to make better, to have better decisions, better insights at a certain point in time.

And that becomes, that's really your digital twin.

Because cities have sonic inference types of noise, things that are very, you know, very loud cities, but also what's creating that noise, right?

That's often unique, the inputs, the types of vehicles, the types of density of people or...

Data and understanding the quantification that you're working towards is really important.

Those are summary statistics, effectively, that maybe they're good on some level to aim for if it means that people move more.

All for it, right?

And it's something that...

If I didn't move as much before and I didn't get up and I didn't do something and this is making me do it, that's awesome or that's great.

But it's also great when now through like a computer vision app, I can understand it's not just 10,000 steps, but maybe there's a small battery of things I'm trying to perform against.

And even the construction in those environments, it is changing what noise exists.

that are helping shape me neurally with the feedback and the targets that I'm getting so that there's more nuance towards achieving the goal I'm aiming for, which is what I'm all about from a neuroplasticity perspective.

So I just don't like the word gamification.

I believe everything should be fun.

Training can be fun and gamified in some ways.

Again, my life has been predominantly in industry, but I've always—I love teaching, and I've always been at Stanford to really—

How do I use technology and merge it with the human system in a way that does help optimize learning and training in a way that is from a sort of neural circuit first perspective?

How do we think about the neural system and use this more enjoyable, understandable target to engage with it?

One of my favorite examples, though, is there was a period—it was right around 2018, and from 2018 to 2020, into the pandemic, where, you know, there became—the students, I noticed, had a much more—

That's shaping, you know, people's hearing thresholds.

there were a lot of projects, their final project, they can build whatever they want.

And, you know, they've had to do projects where they build neural brain computer interfaces.

They've had to build projects in VR.

They've had to build AR projects.

They've had to build projects that, you know, use any sort of input device.

You know, they have to use different sensor-driven input devices.

And that's all part of what they develop.

And

Around 2018, 2020, I started to see almost every project had a wellness component to it, which I loved.

At the lowest level, it's also shaping their sensitivities.

I thought that was—and it was a very notable shift in, like, the student body, and maybe you've seen that, too.

But I still got this—like, one of my favorite games today was this VR game where I'm, you know—

In a morgue, I wake up, I've got to solve an escape room, I've got zombies that are coming out of me and they're climbing out of the morgue and they're getting closer and there's people breathing on my neck and they're like, you know, and everything.

And it's a wellness app.

If you're used to hearing, you know, certain animals in your environment and they come with, you know—

Go figure.

It was their idea of, look, this is what I feel like.

I've got to—because I'm also measuring my breath and heart rate.

And I've got to keep those biological signatures.

Like everything about how the zombies in solving my escape room problems, they're going to get closer to me if my breath rate goes up, if my heart rate goes up.

Exactly.

Exactly.

Yes.

But it was in that environment and it was, you know, realized for them how they felt.

But yeah, and you can do it in much simpler ways.

But at least I'm a huge fan of how do we use the right quantification to develop the right habits, the right skills, the right experience.

acuity or resolution in a domain we might not, or an area where we might not be able to break it into the pieces we need, but it's going to help us get there because my brain actually needs to now learn to understand that different, you know, that sophistication.

you should be heightened to a certain response in that, you're going to develop increased sensitivity to that, right?

I'm happy to walk through a cheat sheet later after, you know, for building out like a computer vision app if, you know, for quantifying some of, you know, some of these more personalized domain related things that people might want to do.

Whereas if it's really abnormal, you know, I hear chickens.

Yes, absolutely.

And we're in an era where everyone, all you hear about is AI and AI tools, and there are tools that absolutely accelerate our capabilities as humans.

But, you know, we gave the examples of talking about some, you know, some of the LLMs.

I mean, I sat next to, of course, we went to Cal.

I sat next, I was at a film premiere and I was sitting there, I was sitting next to a few students who happened to be from Berkeley.

I have a neighbor who has chickens in the city.

And they said to me, you know, they were computer science students and double engineering.

And one of them, when he knew what I talk about or care about, he's like, you know, I'm really worried my peer group, like my peers can't start a paper without chat GPT.

And

It was a truth, but it was also a concern.

So they understand the implications of what's happening.

And, you know, that's on one level.

We're in an era of agents everywhere.

Yes.

And, you know, I think Reid has said that there's, you know, a number of people have said, you know, we won't.

we'll be using agents, AI agents for everything at work in the next five years.

Yes.

And some of those things we need to use, agents will accelerate, they will accelerate capability, they will accelerate short-term revenue, but they also will diminish workforce, you know, cognitive skill.

And as a user of agents in any environment, as a

an owner of companies employing agents.

You have to think hard about what the near-term and long-term ramifications.

Doesn't mean you don't use your agents in places where you need to, but you need to, without the germane cognitive load, there is a different dependence now that you have to have down the road, but also you have to think about how do you

How do you engage with the right competence to keep your humans that are engaged with developing their cognitive skills and their germane cognitive, their mental schemas to be able to support your systems down the road?

There's the semantic context and then just the sort of spectrum, right?

Not at all in the ways that I think are most relevant.

Maybe, you know, there are some, you know, side cases that think about that.

And so, like, first, two things to think about.

One, when I talk about digital twins to companies and such, I like to frame it on how it's being used, how the immediacy of the data from the digital twins.

So,

And the intensity of that spectrum, meaning when I say spectrum, I mean the different frequency amplitudes and what that shaping's like.

Let's go back 50 years.

An example of a digital twin that we still use, air traffic controllers.

When an air traffic controller sits down and looks at a screen, they're not looking at a spreadsheet.

They're looking at a digitization of information about physical objects that is meant to...

give them fast reaction times, make them understand the landscape as effectively as possible.

We would call that situational awareness.

I've got to take in data about the environment around me, and I've got to be able to action on it as rapidly, as quickly as possible to make the right decisions that mitigate any potential impacts.

things that are determined to be problems or risks, right?

And so that's what you're trying to engage a human system.

The visualization of that data is important or doesn't have to be visualization, the interpretation of it, right?

And it's not the raw data.

It's, again, it's how is that data –

you know, represented.

You want the key information in a way that the salient, most important information, in this case, you know, about planes is able to be acted on by that human or even autonomous system, right?

We're both into reefing.

And, you know, I built an aquacultured reef in my kitchen partly because I have a child and I wanted her to understand.

Yeah, yeah.

I love it myself, so don't get that wrong.

It wasn't just all true.

But to understand sort of the fragility of the ecosystems that happen in the ocean and things we need to worry about, care about, and all.

And that affects how your neural system is changing, even at the lowest level of what, you know.

And...