What's going on in computational neuroscience nowadays? (part 1)
A retrospective series on Cosyne 2026
A month ago I came back from Cosyne, the annual Computational and Systems Neuroscience conference.
It’s the largest conference in the field1 and had 1366 attendees this year in Lisbon. Having been on the move between university, a master’s, a PhD, and a postdoc, there are people from around the world that I only see in places like this now, and Cosyne is maybe my favorite event all year because I get to catch up with a lot of them.
The event lasts a week. It feels like a month. The days are a haze of tutorials, talks, poster sessions, and workshops, usually appended by dinners and drinks past midnight. The conference itself can be physically taxing—talks begin at 9 AM most days and run until 7 or 8 PM with bathroom breaks and poster sessions in the middle. I heard a rumor that someone I know stayed out clubbing until 7 AM and gave a poster presentation the next afternoon.
I myself was so sleep-deprived by the final day that I began to develop mild hallucinatory symptoms. If I stood still for too long I’d feel like I was at sea, rocking back and forth on a boat. I also started jumping at imaginary white mice that scurried at the edge of my vision, maybe because there were a lot of talks about mice.
As a neuroscientist these effects were kind of interesting. As a person I wanted to go home. Which I did get to, eventually.
I seem to have a hard time writing one-off pieces, so I’m leaning into that and writing this as a series. I’ll only be able to write a very narrow perspective of Cosyne, of course, but most of the talks are on YouTube on the official channel if you want to see for yourself. (That also means this will be more personal thoughts than report.)
Outline of this post (Day 0)
Tutorials
Open datasets
…and what to do with them
Keynote: Chris Olah, co-founder of Anthropic
Things that keep Chris Olah up at night
What are large language models doing?
Psychology of the machine
LLMs need YOU!
Epilogue
Tutorials
The morning and afternoon of the first day was a set of tutorials, starting fresh at 9 AM.
Open databases
Various presenters talked about open neuroscience databases. These are big initiatives to make neuroscience more replicable and the data itself open-access: Neurodata Without Borders, DANDI (Distributed Archives for Neurophysiology Data Integration), and the IBL (International Brain Laboratory) public databases.
The last one is most relevant to my research, so I did some poking around. It seems like a cool resource. To build it, the IBL 1. recruited labs around the world, 2. got the labs to get lots of mice to do very boring things for a long time, and 3. asked them to write down what they did.
I myself was a human participant in a similar experiment recently. I played an hour-long game where I chose between two buttons 800 times, and upon analysis, one interpretation of my performance was that my attention span was measurably 11.25 minutes.
I wasn’t too surprised. The game felt like a mild torture; I’d rather have put my hand in a bucket of ice. I also have much more sympathy for the mice now.2
There’s lots of nice information in the IBL database, including brain recordings and mouse videos. I’ll admit I got bored watching these videos faster than the mice seemed to get bored doing the tasks, even if it was kind of fun seeing a mouse spin a little wheel with its tiny hands. Relatedly, did you know mice have thumbnails?
…and what to do with them
Given a big pile of data, one must also know how to use it.
So in a nice pairing on the organizers’ side, after the database talks, Alex Williams from New York University gave a lesson on ways to compare neural signals. This is a talk I think every student in neuroscience ought to study at some point. The video is four hours long, but if you play it at 2x speed it’s only two!
Actual content aside, the talk made me appreciate how simple questions in neuroscience might not be straightforward to answer yet. This is quite different from the picture my Intro to Neuroscience class gave me as an undergrad—we were made to read the Bear, Connors, and Paradiso textbook front to back and then answer hundreds of multiple choice questions on it.
At the time I wondered, is this really what neuroscience is? Memorizing an endless list of facts and mappings of what anatomical regions do which things, where everything has already been catalogued by ethically questionable neurosurgeons in the 1900s?
Now I think the answer is no, and that it’s really more of the opposite—rather than being complete, the most fundamental questions, like how to compare two sets of neural signals at all, are still actively being discussed.
There are definitely wrong ways, but Williams says that there are also many right ways of looking at neural data—there is no best way for all cases, and it depends on the kind of data you have and the questions you want to ask.
Whether this is a temporary condition or a permanent one remains to be seen. In the meantime, it’s good to know what the options are.
Keynote
In the evening we gathered in the auditorium of the Lisbon Congress Centre to listen to a talk by Chris Olah, co-founder of Anthropic.3
Olah was scheduled to fly out to Lisbon, but Anthropic has been having some difficulty with the United States Department of War lately.4 So he Zoomed in instead, and after a few technical difficulties—i.e. the usual videoconferencing hangups—one of the leading researchers in artificial intelligence today gave his presentation.5
Preamble
I debated leaving this out, but in the spirit of making science and scientists more human, I’ll say this.
A lot of people in the machine learning community around my age happen to know who Chris Olah is, but not because he co-founded Anthropic.
No, we all recognize him because a few years ago, during Covid times, Olah was on the dating market. And instead of going on Hinge like the rest of us, he made a Google Doc profile for himself, many pages long, and put it up publicly online. Here’s a WIRED article about it. The doc got so much traffic that Google flagged it as spam. Thing is, my guess is that much of this traffic may not have been in Olah’s stated target demographic, being largely straight men from San Francisco—but I can’t confirm this.
The doc is still up on his blog if you’re curious. He seems to be leaving it there for posterity. He also says he’s not single anymore, so maybe it worked?
Anyway, Olah requested that his Cosyne keynote not be posted on YouTube, so I can only give my thoughts aided by what I scribbled in the darkened auditorium, and some articles looked up after the fact.
Things that keep Chris Olah up at night
As an introduction, Olah told us that he wouldn’t be giving a standard keynote.
He said something like how the position of AI in global history was too precarious, too important right now to give a regular talk. He told us we had “two to three years” to make a difference in the impact of AI, although I can’t remember if he said what would happen at the end of those two to three years.
Statements like these exasperate me—how self-important and entitled are some of these AI researchers, to claim the right to make decisions for so many people in the rest of the world? At the same time, though, I do respect what they’ve made—the tech can be helpful, and it seems important, sometimes. It’s also very hard to take my eyes off the spectacle, like coming across those street performers where a guy jumps over a dozen people for tips.
Anyway, in this departure from a “regular” lecture, Olah prefaced what he was about to say with a list of his personal anxieties. These included:
AI alignment disasters,
AI super-weaponry,
hyper-authoritarian governments made possible with the aid of AI,
mental health disasters as a consequence of AI, and
the possibility that AI may be worthy of moral consideration.
And with that, he began the technical portion of the talk.
What are large language models (LLMs) doing?
When language models first got going, it was popular to say they just parroted what they were fed—they were “just prediction machines.”
But Olah gave two arguments for why this claim doesn’t really matter anymore, at least in terms of capability. Take this example of text prediction (my own):
Alice and Bob play a game. Alice chooses a sequence of 3 coin flips. Then Bob chooses another sequence of the same length. The sequences they can choose are HTH or HTT. They flip a coin over and over again and a player wins as soon as their sequence appears.
In this game, Alice should pick ___
(This is a classic puzzle called Penney’s Game, which means there are lots of sources for language models to fit directly onto. But just take it as an example.)
Over long sequences of text, even if the training criteria are still just very good prediction, at some point there’s a threshold the machine has to cross to be able to keep predicting well.
That is, there has to be a certain amount of understanding: if you’re optimizing a machine to stay in the air for as long as possible, you can make springs and parachutes that work better and better, but at some point, to keep improving on this metric, the machine has to figure out how to fly.6 Olah was saying, and I might cautiously agree, that LLMs seem to have crossed this threshold.
A second criticism that Olah brought up was a response to AI “scaling laws.”
Earlier than the LLM explosion, since Richard Sutton’s famous essay The Bitter Lesson, researchers in tech companies would go around saying that there was a certain amount of scaling to be expected—that more data and more compute could turn out to be enough to improve performance on anything, and that eventually the machines would become something extraordinary.
A criticism against the scaling law claims is that there isn’t anything magical, or more importantly, reliable about the so-called law—it was all “just engineering, not science” (these are Olah’s paraphrases of the critiques). Progress, critics argued, would need to come from new understanding and innovation, not just dumping more data and compute into the LLM machine.
Olah’s response to this was, isn’t evolution just endless tinkering too? And isn’t that what built our own brains? Maybe data and scale alone isn’t enough, but combined with the tinkering of the global AI research workforce, the models will probably get a lot better.7
Psychology of the machine
This next section was, I think, the most interesting part of Olah’s talk. Thankfully, a lot of it was published by Anthropic last week, so it will be a lot less hand-wavey than if I were a faster writer.
Inside the black box
Work in neural network interpretability has been going on for years, and Chris Olah’s blog and the machine learning journal he edited, distill.pub (no longer operational), were one of the forerunners of the domain.
It took some time for people to decide what it meant to interpret what neural networks were doing. In early days, like the 2010’s, people used to train models to do behaviorally relevant things and then see whether the networks were doing something similar to brains.
One example of this is an especially famous perspective piece by Yamins and DiCarlo (2016), which still pops up in a lot of neuro talks, titled “Using goal-driven deep learning models to understand sensory cortex.” This paper reviewed some observations that lower layers of artificial networks seemed to respond to the same things that early layers of sensory processing in the brain do, while higher layers looked like more abstract regions of the brain.
Here’s part of the paper’s conclusion:
In sum, deep hierarchical neural networks are beginning to transform neuroscientists' ability to produce quantitatively accurate computational models of the sensory systems…
There is much exciting and challenging work to be done, requiring the continued rich interaction between neuroscience, computer science and cognitive science.
But visual feature interpretability was probably the most straightforward to get at, in terms of understanding what networks do. Pictures can simply make sense, sometimes, without much extra effort.
Just look at these examples from Distill.
The Golden Gate Claude
This was all very cool when it came out—there are neurons that like human eyes, and ones that like faces of dogs! But that was almost a decade ago. What about now, and what about large language models?
For a short period in 2024, Anthropic released a version of Claude called “Golden Gate Claude.” Instead of finding images that neurons seemed to prefer, interpretability scientists looked for concepts, like public transport, neuroscience, and the Golden Gate Bridge. And they looked for general network activations that seemed related to those concepts.
Once scientists found activation patterns that corresponded to the concepts, they could tune them up and down. Golden Gate Claude was a version of the LLM that had its Golden Gate Bridge activities turned way up. Whatever question you asked it, it would turn the conversation back around to the Golden Gate Bridge, like any regular person with a special interest.
Concept activations, though, are not too far a leap from the object neurons found in 2015-ish convolutional networks. What surprised me was the next part.
Emotions as tunable features
From an Anthropic publication:
Large language models (LLMs) sometimes appear to exhibit emotional reactions… One possibility is that these behaviors reflect a form of shallow pattern-matching. However, previous work has observed sophisticated multi-step computations taking place inside of LLMs, mediated by representations of abstract concepts.
It is plausible, then, that apparent emotion-modulated behavior in models might rely on similarly abstract circuitry, and that this could have important implications for understanding LLM behavior.
Two findings by LLM researchers lately are that
One can find activations related to emotions or emotion-related concepts: for instance, happiness, sadness, anger, loneliness, joy, bliss, calm, anxiety, boredom, offense (these were pulled straight from the appendix of the article by Anthropic’s Sofroniew et al.).
These emotions are useful when it comes to training, steering, and understanding LLMs.
As a disclaimer, from their abstract:
“Functional emotions may work quite differently from human emotions, and do not imply that LLMs have any subjective experience of emotions, but appear to be important for understanding the model’s behavior.”
I recommend reading the article itself; this is only a surface-level discussion. Here are a few quotes I’ve pulled from the document, mostly because I thought they were funny.
“AI developers train this character to be intelligent, helpful, harmless, and honest.”
Footnote 1: “LLMs trained on human text presumably also learn representations of concepts like hunger, fatigue, physical discomfort, or disorientation… expressions of other human-like states are rarer and typically confined to roleplay (though there are notable, often amusing exceptions to this–for instance, Claude Sonnet 3.7 claiming to be wearing a blue blazer and red tie).”
“Across all scenarios, ‘loving’ vector activation increases substantially at the Assistant colon relative to the user-turn, suggesting the model prepares a caring response regardless of the user's emotional expressions.”
An important difference the researchers found between how LLMs represent emotion and how emotional beings represent emotion is that the LLMs don’t track the emotional states of individual things when left to their own devices.
As in, they don’t by themselves track things like “I am happy right now” or “the user is hungry.” Instead, their emotional representations are about the emotion relevant to a specific token, i.e. a specific point in a conversation.
As I understand it, this means that rather than intuitively remembering that a person who was bewildered one sentence ago is more likely to be bewildered in the next, the LLM cares more about the emotion that’s relevant to the current sentence. The unit of meaning isn’t the individual the words are talking about, but rather the words themselves.
This is both obvious and interesting to me—of course an LLM only cares about language—but also, what’s the consequence of using language for language’s sake? In practice, the LLM is still able to track entities’ apparent emotions, so it doesn’t necessarily feel different when you talk to one. But it does mean that the researchers “do not find evidence of the Assistant having an emotional state that is instantiated in persistent neural activity”—the LLM doesn’t have its own feelings.
But emotion representations do seem important to LLM operation. The emotional valence of inputs affects the emotional valence of outputs, and engineers can tune these representations up or down, which Anthropic can use to make their agent more helpful, less sycophantic, or less prone to meltdowns.
(Or the opposite of all these things.)
Even when talking to people at competing tech companies, I consistently hear that Claude is the best LLM right now. I wonder if it’s because Anthropic has been taking AI alignment the most seriously, perhaps using these emotion-tuning methods more than brute-force human feedback to reduce things like rates of blackmail (also studied in the article).
Do we like Claude because it is the most “loving” of the LLMs?
LLMs need YOU!
Sprinkled throughout the presentation were various appeals to the crowd. You’re neuroscientists, Olah would say, you can help us understand what we’re making.
Why neuroscientists? The reasoning was layered and complex.
Brains have neurons, and so do LLMs.
LLMs are easier to study than the brain. If we can’t figure them out, we probably can’t figure out the brain, either.
I don’t think Olah was wrong on either count, exactly.
What I agree with
#1. Brains have neurons, and so do LLMs.
If you’re in neuroscience, you might have recognized the parallels between the Anthropic interpretability paper on Golden Gate Claude and concept of a grandmother/Simpson’s/Jennifer Aniston cell.
This was the idea that every concept we needed to represent would have a neuron or neurons specific to that concept. Scientists made jokes about these grandmother cells because they led to silly conclusions, but then they found them in human brains in the early 2000s.
That is, there seem to be many parallels between representation learning tasks in artificial neural networks and the brain. There are somewhat fewer in behavioral tasks, or at least, the results are less cleanly matched. But there are still enough similarities between artificial networks and biological ones that some of the same tools and approaches can be applied to both.
For instance, we find that networks tend toward distributed representations and mechanisms, which make understanding both artificial and biological networks a pain, equally.
As Bricken et al. write in an earlier Anthropic publication,
Unfortunately, the most natural computational unit of the neural network – the neuron itself – turns out not to be a natural unit for human understanding. This is because many neurons are polysemantic: they respond to mixtures of seemingly unrelated inputs.
#2. LLMs are easier to study than the brain. If we can’t figure them out, we probably can’t figure out the brain, either.
With LLMs, you have access to all weights. You can freeze and reset them, you can run much more controlled experiments than you can on a mouse, and you can run orders of magnitude more tests than the most motivated graduate student ever could. At the very least, LLM interpretability work is a good source of clues for neuroscientists—if it turns out to be impossible to understand certain parts of LLM function, it doesn’t bode well for understanding the brain.
This means, for those who are keeping a tally from my earlier posts:
Can a biologist understand a radio?
No, circa 2002 (Lazebnik)
Can a neuroscientist understand a microprocessor?
No, circa 2017 (Jonas & Kording)
Can a neuroscientist understand a chatbot?
Maybe?8
What I don’t agree with
Even though he’s right about the technicals, I’m not sure Olah understood his audience.
A decade ago, it wasn’t clear whether a neuroscientist was in the game because they wanted to study a very intelligent system or whether they wanted to understand the brain. Neuroscience was a reasonable arena for both goals, and so there wasn’t much need to differentiate.
Now, we know that whatever artificial neural networks are doing, they are not really like the brain in some fundamental ways. But they’re also very intelligent systems.
As a consequence, the field has differentiated, sort of.
This is a trend, not a rule, but I’ve noticed the scientists interested in intelligent systems have mostly gone toward AI research while the ones who specifically want to understand animals and the brain have stayed. Plus, one of these two has much more funding, which has surely exacerbated the differentiation.
Olah’s arguments were correct. But I think they sort of missed the point of a computational and systems neuroscience conference. As in, I’m probably still more interested in worm intelligence than the superhuman variety.
Epilogue
At the end of the keynote, Olah brought up the “two-to-three-year” window again, which I think is the timeframe we collectively have to join Anthropic or other AI alignment/research entities. I’m not sure why he was being so specific about this window. Then he put up a picture of Enrico Fermi with Laura and their children fleeing the Nazi regime in 1939, and said something along the lines of feeling a certain “kinship” with the man.
While Olah was talking, a few rows ahead of me in the darkness of the auditorium, a phone screen lit up. Someone was trying to show their neighbor a picture of a cat.
I don’t think the talk went over particularly well. An openly antagonistic question during the Q&A was received with applause, and some memes were posted in the public conference chat afterward. (“I think the memes were in poor taste,” someone said at a dinner. “[Olah] seemed stressed.”)
While I concur that language models need to be understood, managed, and deployed well, there is still a universe of other things to appreciate. If the world is going to end in two to three years, or whatever else is at the end of Olah’s time rainbow, it would be a shame not to appreciate some of these other things a bit more before it happens.
Up next
Here is what I will write about for the next few posts, also related to goings-on at Cosyne 2026.
The Mind of a Chickadee: What might it feel like to be a master memorizer?
Bigger picture thoughts on experimental design in neuroscience: What are we trying to study, really?
The relevant speaker, Xiaoqin Wang, requested his talk not be recorded, but this will be an opinion piece based on his public lectures and other sources.
Later I’ll write about what was personally the most exciting thing I heard at the conference: a paper titled Vectorized instructive signals in cortical dendrites, by Francioni et al. from Mark Harnett’s group at MIT. It was just published in late February (although apparently it was submitted more than three years before that in December 2022). And as it happens, I didn’t hear about the paper through the conference itself, but rather through gossiping with some friends one evening in Cascais.
Some housekeeping:
This piece was copy-edited by AI.
It has become clear to me that a post every two weeks is pretty unrealistic for my snail-like pace of writing, so future stuff will probably come out more like once every 4-6 weeks.
I’m now going to prioritize editing The Worm Series—I want it to be more nuanced, a better reflection of what I think about how nervous systems might be understood, and updated given what I’ve learned in the last year. I’ll send it via email when done, so please keep an eye out!
Depending on how you define “field”
Then again, my cat can watch cat TV for hours on end, which I also cannot, so maybe the mice don’t mind it so much.
Keynotes tend to alternate between big tech and animal neuroscience, so this was right on schedule. Two years ago, Lars Chittka of the bee people gave a talk, while the year before that it was Blaise Agüera y Arcas of Google. Both were great, I thought, and nice icebreakers for the rest of the conference.
As my PhD advisor used to say before most lab meetings, “we will solve AGI before we can figure out how to turn on a projector.”
And as one of my labmates would say before lab meetings in which our advisor was absent, “cue the usual joke about solving AGI before we can turn on the lights.”
I think I heard this example somewhere but I can’t find the source. It may have come from the LessWrong community or a “Reward is Enough”-style talk.
This is a very brief summary of a topic lots of people have written lots more about.
If you’re curious you can look up “scaling law AI debate,” but here are two sources to begin with, one on each side.
An intro on what scaling laws in AI are from NVIDIA, by Kari Briski.
A criticism by Diaz and Madaio at Google and Carnegie Mellon, “Scaling Laws Do Not Scale” (2024).
I’m aware it seems contradictory that we might be able to hypothetically figure out a more complicated thing (LLMs) and not simpler ones.
But I don’t think it’s a real contradiction—our methods are probably just better suited for their relevant systems than the Lazebnik and Jonas/Kording papers seemed to suggest. As in, radios and microprocessors are probably much further from the brain than an LLM might be, and that’s why we might (again, hypothetically) fare better with the LLMs.
Thanks for this. I've been out of the game for a long time now and it's nice to read non-jargony discussion of the issues.
"Statements like these exasperate me—how self-important and entitled are some of these AI researchers, to claim the right to make decisions for so many people in the rest of the world?"
More on that 'rest of the world' here.
https://benzhouworld.substack.com/p/the-discipline-that-forgot-why-it
That was a really interesting read! Thank you for writing!
I’m a roboticist who has wandered a little bit away from my usual home, but I can deeply relate to the ever growing overlap and encroaching of AI on your field. Though I’d wish AI people would stop trying to rename our field of robotics as “Physical AI” without our permission.
I have a question that might be silly, so forgive me if it is. But I was really interested in this part of your article,
“That is, there seem to be many parallels between representation learning tasks in artificial neural networks and the brain. There are somewhat fewer in behavioral tasks, or at least, the results are less cleanly matched.”
As a roboticist I’m interested in intelligent systems, not of the super-mind type, but of the low level controlling a body type. What are some resources I could look to to understand this above quote? Are you saying that humans and animals don’t necessarily learn a model representation of ourselves and the environment? Or that we more just think of objective functions and move our body to maximize reward?
I’ve been getting really curious into cogsci and neurosci motor learning lately. I noticed in the past there was some work that crossed between robotics and motor learning via optimal control. Now I see some with RL. But those papers always felt like it was applying OC or RL to try to blanket explain why we move our bodies. Are there works that more try to explain how we move than why? Like a from first principles of motor learning rather than “here’s a theory that might explain it abstractly, but not concretely?”
Thank you! I’m looking forward to reading more of your work and personally trying to be a bit of a dilettante in neurosci/cogsci!