Sponsor note: I am extremely happy to announce my first commercial, service-oriented sponsor: rush.cloud. I’ve been doing these podcasts entirely through kind philanthropic donations, which is very nice, but I’d ideally like to be helping someone when they sponsor me. And now I have that! So, if you are at all involved doing preclinical drug discovery and would benefit from computational tools, you should check out their platform + beautiful website here: rush.cloud.
Youtube:
Spotify:
Apple Podcasts:
Introduction
Ellen Zhong is perhaps one of the only people in the ML x bio field to have created an entirely new subfield of research during her PhD: the application of deep-learning to cryo-EM particle images.
If you aren’t familiar with that field, I luckily have a 8,000~ word article covering it, which walks through a lot of Ellen’s papers. If you don’t have time to read something that grossly large, the general breakdown of the problem is as follows: cryo-EM can give you thousands of 2D views of a 3D protein from many different angles, from that data, can you discover what that 3D structure is?
Ellen, who is now a computer science professor at Princeton University, has spent her academic career investigating that question, and now has an entire lab at Princeton (E.Z. Lab) focused on that and related ones. Including, as the title mentions, the possibility of doing performing cryo-EM structure determination at ultra-high scales.
In this podcast, we talk about her research, what she did during her recent sabbatical at Generate:Biomedicines, her recent interest in areas beyond cryo-EM (cryo-ET and NMR specifically), and more!
Timestamps
[00:00:00] Introduction
[00:02:43] What does it mean to apply ML to cryo-EM?
[00:04:28] Ab initio reconstruction and conformational heterogeneity
[00:15:41] Can we do multiplex cryo-EM structure determination?
[00:22:19] Datasets in cryo-EM
[00:26:25] Why isn’t there a foundation model for cryo-EM particle analysis?
[00:33:07] How much practical usage is there of these cryo-EM models amongst wet-lab cryo-EM researchers?
[00:40:34] Where can things still improve?
[00:46:57] Has deep learning done something in cryo-EM that was previously impossible?
[00:48:22] Ellen’s experience in the cryo-EM field
[00:53:40] Deep learning in cryo-EM outside of structure determination
[00:57:32] 3D volume reconstruction versus residue assignment in cryo-EM
[01:00:26] What did Ellen do during her sabbatical at Generate Biomedicines?
[01:07:07] Ellen’s research in cryo-ET
[01:13:54] Ellen’s research in NMR
[01:21:05] How did Ellen get into the cryo-EM field?
[01:26:57] Why did Ellen go back to graduate school?
[01:32:17] What makes Ellen more confident about trusting an external cryo-EM paper?
Transcript
[00:00:00] Introduction
Abhi: Today I’ll be talking to Dr. Ellen Zhong, a computer science professor at Princeton University. Ellen’s research focuses on applying machine learning to protein structures derived using cryogenic electron microscopy, also known as Cryo-EM. In fact, she was one of the first people to ever apply deep learning here, and her lab at Princeton remains at the forefront of this field.
Today we’ll be talking about her research, what the future of protein structure determination looks like, and her recent sabbatical at Generate Biomedicines. Thank you for coming onto the podcast, Ellen.
Ellen: Yeah, thanks for having me.
Abhi: So my first question is, I gave a very brief overview of what your own personal research interests are, but I’d love for you to give your own take as to what you’re most interested in these days.
Ellen: Sure, yeah. So, right now I’m a professor of computer science at Princeton University and our group works on various molecular machine learning problems. So really interesting problems at the intersection of AI and biology, and specifically we focus on structure determination. So can we actually analyze actively collected experimental data, in particular cryo-electron microscopy or Cryo-EM, to solve the actual 3D locations of the atoms of these macromolecular machines and be able to understand how they work and discover new kinds of proteins and their dynamics.
Abhi: When you first got involved in this field, like back in your PhD days... it seemed like you were the first person to ever take Cryo-EM particle images and apply deep learning there. When you were doing that back then, did it feel like you were taking a really big risk or was it... you were kind of sure of yourself, like this was the right direction?
Ellen: I guess at the beginning of my PhD I worked on a couple of different things and then I learned about the Cryo-EM problem, and I thought this 3D reconstruction problem was really interesting. And in particular, this problem of reconstructing continuous motions was unsolved at the time.
And actually I was sent this paper by Roy Lederman and Amit Singer, who are mathematicians at Princeton University, about this mathematical introduction to this problem. And after reading that paper, I was like, “Whoa, this is a really cool problem.” And I guess I didn’t know at the time that the rest is history, but it was very much like, “This feels like a problem, a very juicy problem that I can work on for the next five years or something like that.”
And I didn’t really think about anything else. But when I, at the time, was learning more about modern machine learning techniques, it seemed very natural to apply these deep learning models, which can learn super complex distributions from large-scale datasets, to this Cryo-EM problem. But yeah, it was just super fun.
[00:02:43] What does it mean to apply ML to cryo-EM?
Abhi: When it comes to actually applying machine learning to Cryo-EM particle images, I have a vague sense of what that means. Would you be able to give a brief overview of what it actually means to take the Cryo-EM particle images you get out of an electron microscope and throw ML at it?
Ellen: Yeah. So the specific problem that was the focus of both my PhD work and is very much a central research question in our group right now is this 3D reconstruction problem. So the electron microscope, you take a bunch of 2D projection images of your vitrified or flash-frozen biological sample.
So you have a bunch of 2D pictures of your protein. And then there’s the algorithmic challenge of reconstructing the 3D structure from all these 2D particle images. And so it doesn’t have to be approached with deep learning approaches. There’s a lot of classical algorithms.
There’s state-of-the-art tools that are not using neural networks to do this 2D to 3D reconstruction problem. And the central computational challenge is inferring both the 3D structure, but also the unknown camera poses and orientations of each of the molecules. And the specific new challenge we’re interested in is what happens when the different proteins are in different conformations.
So not only do we want to reconstruct the 3D structure, like a static picture of the protein, but we want to reconstruct a movie of the dynamics of these macromolecules. Which, you know, to me that seemed like the coolest thing, right? Structural biology was... it’s already so interesting to be able to see these 3D structures, but can we actually see how they work and how they function from experimental data?
[00:04:28] Ab initio reconstruction and conformational heterogeneity
Abhi: I’ve written about Cryo-EM before, and it seems like there were three distinct phases to the field, at least the part of the field that you work in, where at the very beginning you’re focused on ab initio reconstruction, where you have no prior knowledge as of what the actual protein structure looks like.
Then conformational heterogeneity, where there are multiple conformations in the particle images. How do you tease out all the possible conformations? Before I get to the third one... for ab initio reconstruction, is that a solved problem? Like you no longer need to have any prior knowledge?
Ellen: For well-behaved samples. So, because this is always collecting active new experimental data, you need to solve this 3D reconstruction problem for every single protein. And so for these new complexes that people had no idea what they looked like before, there’s so much upstream work both to make sure the sample is expressing well and is well-behaved, make sure the ice is very thin and you can have high-quality images.
And then there are existing methods for ab initio reconstruction. And so assuming that your input data is reasonable, then you can do this pose estimation in 3D reconstruction to get the structure. But that’s very much still an open problem.
Abhi: Is it an open problem in the sense of like sample prep can mess it up? Or is it an open problem in that, even if the sample prep is perfect, there are some proteins which, like CryoDRGN, which is the method you created to solve ab initio reconstruction... it still doesn’t work.
Ellen: The... I would say both. So like, definitely if the sample’s not good, the algorithms are not going to solve that for you. But the part where it becomes interesting whether you can jointly design the experiment and the algorithms are when you have lots of dynamics in your protein or maybe a complex mixture of multiple things. If you just have a very purified protein that’s very much like a rock or something like that, or a ribosome, which is one of the canonical examples in Cryo-EM, then it is shockingly easy and absolutely fantastical how we can solve the structures of these molecules in like... in like a day.
Abhi: Before I move on to conformational heterogeneity, one thing I wanted to ask was, I saw that you posted on Twitter recently about CryoBoltz, which is trying to use existing protein structure predictions... it’s almost as a starting point for the Cryo-EM modeling. If you have that, do you even need ab initio reconstruction, or are the prediction models are actually pretty good and...
Ellen: Yeah, that’s like another super interesting question for the field right now. So I feel really lucky that currently we have AlphaFold 2, AlphaFold 3, all these really powerful protein foundation models that can just predict structures from sequence. So like, why do we even need to do experiments?
And the answer is, okay, these prediction models are just predictions. There’s a lot of proteins where you can’t actually... a lot of specifically large complexes, which are the functional machines in vivo that do these essential biological processes. So for those you can’t predict the entire structure. If we want to understand function, if we want to actually be able to see what’s going on and the mechanism of these essential molecules, then we still need experimental data. And especially for training the next generation of these models, we’re going to need to understand what are the interesting actual ensembles or dynamics and larger complexes to be able to have these structure prediction models reach the next level.
Abhi: Okay. I’m going to have more questions about that later, but... But moving on to conformational heterogeneity, that was the subject of CryoDRGN 2... which released in 2021, is that correct?
Ellen: So the... yeah, I should give a whole talk or something like that. But the Cryo-EM reconstruction... so CryoDRGN is our model for reconstructing... it’s a VAE-based neural model for reconstructing these distributions of structures. And at this point we have so many extensions of CryoDRGN that tackle more challenging settings of the problem or in situ data with Cryo-ET, but in our group, we’re just all calling it CryoDRGN. Okay. To have it in one software package that people know where to access the model and where to use the method.
The original CryoDRGN paper... the original machine learning paper was very much trying to tackle the whole enchilada of ab initio reconstruction of complex mixtures or of distributions of structures. And then when we were creating the software tool for the actual structural biology community, then I realized, “Wait, wait, wait. That’s actually too hard.” And already if we just tackle this conformational heterogeneity problem where you assume you already have the camera poses and you just want to infer residual heterogeneity, we can already discover interesting things like missed structures, like new conformations, continuous motions, which was the original motivation for the method. That was CryoDRGN, I guess V1. In CryoDRGN 2, we revisited this ab initio reconstruction problem...
Abhi: Oh. So conformational heterogeneity came first and then ab initio came second?
Ellen: Yeah. Okay. So the original... well, the original ICLR paper in 2019-2020 was trying to do everything. Okay. So it was like... I guess at the time I didn’t know what the scope was for a machine learning conference paper. I just want to solve everything. Yeah. So that was doing ab initio heterogeneous reconstruction. And then later when we developed the tool, we just focused on the conformational inference. And then I guess in my research group, in the last couple of years, we’ve released an extension, CryoDRGN-ET, to Cryo-ET data and CryoDRGN-AI, which is ab initio... not artificial intelligence. And those two methods are designed for modern Cryo-EM datasets to actually tackle ab initio reconstruction of super large, complex Cryo-EM data.
Abhi: So, yeah. Okay. I guess I mixed the two up, in terms of timeline for the conformational heterogeneity bit, is that entirely solved? Like given a protein that is very flexible and displays many conformations, if a Cryo-EM person took images of that protein, applied the CryoDRGN software package to it, could they reliably grab out all the conformations or are there still ongoing issues?
Ellen: I would say, yeah. None of these problems, I would say, are solved. Okay. Right. These are all very much open challenges that any new dataset could present completely new challenges. There are now... I mean, it’s an exciting time because there’s a lot of new methods, both neural network-based and also classical signal processing-based algorithms or computational numerical linear algebra methods to tackle this heterogeneous reconstruction problem.
But yeah, I would say one of the main challenges is that this heterogeneity problem in Cryo-EM is not super well-defined, right? It really depends on what the practitioner, what the structural biologist wants at the end of the day. Like, do they want an ensemble of structures? Do they want an ensemble of atomic models, which is what CryoBoltz is trying to do? Is this atomic modeling part of...
Abhi: Sorry, what? Like in my head, the output of Cryo-EM... the output of this is one singular ground truth. What is the full ensemble of truths out there?
Ellen: So I break down this Cryo-EM problem into... there’s a couple of stages of image processing. There’s the pre-processing of the raw micrographs, and you know, there’s a lot of steps to that. And then there’s this 2D to 3D reconstruction problem. So that’s already with segmented single particle images in 2D to resolve the 3D structure or an ensemble of structures. Those structures... so “structure” is an overloaded word... in the Cryo-EM setting, that means a density volume.
Abhi: Okay. Right.
Ellen: So it’s just the electron scattering potential of the molecule as felt by the electron beam in the cryo-electron microscope.
And then assuming you have a sufficiently high-resolution volume that’s not filled with artifacts or too low resolution, then typically what’s done is you manually build in the atomic model. As of recently, now there’s some tools based on these modern deep learning and protein-based models that can do atomic modeling, but it’s still very much a manual art to actually placing the atom locations to then deposit into the PDB.
Abhi: That was surprising. I kind of always thought... I guess it makes sense that the output of Cryo-EM is an electron density field, because what else could you possibly gather from shooting electrons at something and seeing what comes out on the other side on the electron detector. But I also assumed that assigning atoms is a trivial process. Why is it... intuitively, why is it hard?
Ellen: So it’s very trivial if the volume is high resolution. Yeah. So if you have an atomic resolution volume, then you can just see where the nuclei are and place the atoms. But that’s extremely hard. And for most, a high-quality Cryo-EM structure will be maybe three angstrom resolution, which is enough to resolve maybe secondary structure features and some side chains. And then based on our existing prior knowledge on the geometry of protein structures or nucleic acid structure and composition, then you can mostly unambiguously place the atomic models.
But also, this is just a messy, hard scientific problem. The structures are not necessarily uniform in resolution. The parts that are moving will be lower resolution because we’ve blurred it together in the standard reconstruction problem.
Abhi: So like if there’s a bit floppy over here, it’ll just smear the electron density. Yeah. Okay.
Ellen: Yeah. You’ll smear it together, assuming you’re doing a homogeneous 3D reconstruction. So the whole promise of CryoDRGN and these heterogeneous methods is that we can actually model the ensemble and then we’re not averaging together all these different conformational states.
[00:15:41] Can we do multiplex cryo-EM structure determination?
Abhi: Yeah. So, we’ve discussed ab initio reconstruction and conformational heterogeneity.
Ellen: There’s a lot of jargon.
Abhi: One of the... I think one of the craziest things you’ve written about was a paper at the end of NeurIPS 2024 called Hydra.
Ellen: Yeah. Multi-headed dragon.
Abhi: Yeah. Which is a method that... I think the phrase used in the paper is “compositional heterogeneity,” where you have multiple proteins on the same micrograph, and you are trying to multiplex the structure determination of all of them at once.
Ellen: Yeah.
Abhi: This was back in December 2024. To me as an outsider, that just feels clearly extraordinary. You’re able to 2x to 3x how many proteins you can shove through this incredibly expensive and manual process.
Ellen: Yeah.
Abhi: Is it as revolutionary as it seems on face value? Like how much work is there?
Ellen: Yeah, so it’s a very hard problem and that’s very much still what we’re working towards. Like, can we do high-throughput structure determination, not only of ensembles and these dynamic atomic models, but just shove multiple proteins into the sample, into the electron microscope, and then simultaneously solve their structures?
So that’s very much something that we’re still working towards. Hydra, we demonstrated maybe three or four different structures on real data at the same time. And it just becomes a much more challenging optimization problem with more and more proteins if we’re using classical-based approaches where we’re trying to infer both now the identity of the protein and the pose or the 3D orientation relative to the camera pose... the orientation of the particle within the ice and also the conformational state.
So I think there’s probably some inherent limit to these classical-based approaches, and so very much as a moonshot in the group right now, we have a bunch of new projects that are like, “What happens if we want to solve a thousand structures at once? What happens if we want to analyze a cellular lysate or a Cryo-ET sample, which is like a slice out of a cell, and actually solve all of the...
Abhi: Dozens of biomolecules in there, maybe hundreds?
Ellen: And that’s... I would say, it’s both a good thing and a bad thing, but it’s very much still a moonshot, right? And it’s very exciting times.
Abhi: In the paper that was at NeurIPS, there were three proteins determined. Is three as of then kind of the upper limit, or is it more like we haven’t tested up to five or 10 yet and we just chose three?
Ellen: I would say it very much depends on the experimental sample. So definitely people are doing structure determination from native extracts and things like that... in actual structural biology papers, and using classical-based reconstruction methods you can simultaneously solve a handful of different structures.
The unfortunate... the status quo though is that it’s very manually driven and it’s very... an expert structural biologist or cryo-electron microscopist is going in and arbitrarily subsetting your dataset to find reasonably pure classes where it’s mostly just one complex or not. And it becomes this very user-driven process.
And so even in Cryo-EM papers where you’re focusing just on a single structure, if you look in the supplement and you look for one of the data processing figures...
Abhi: Sorry, what’s SI?
Oh, supplementary.
Ellen: Yeah. If you look in the supplement, there’s always going to be one of these figures that show the image processing pipeline, and it’s just this crazy flowchart of all the different steps and all the different user-chosen subsets that are taken to get to the final particle stack.
And so, very much, one of the interesting computational challenges is, can we automate that process? Can we make it either one-shot from an algorithm that is trying to do all of the processing or the optimization in one go, or, these days, can we have an autonomous or an agentic approach to tackling it? So there’s so many interesting directions right now.
Abhi: What is currently the... if I just come at it from an incredibly naive perspective, if the problem is, “Oh, there are so many... let’s say you’re trying to do a hundred proteins in one shot and there are so many different conformations each of these unique proteins have, how do you subset them into classes?” My instinct is just, “Oh, expand the micrograph and have more shots...”
Ellen: Just collect more data.
Abhi: Yeah, just collect more. Is that... is it just... yeah, what’s the primary bottleneck in getting to a hundred-shot Cryo-EM?
Ellen: Yeah. I think the number of images that we’re taking right now is... people can collect a lot of data. The microscopes are becoming very automated. So I would say data collection is not necessarily a problem. I think people are not actually actively working on it from the experimental side, which is interesting.
And there are also experimental challenges too. There’s computational challenges. You really need to do both. I’m excited to do that. And actually along these lines, my group and the Flatiron Institute and CCP-EM, which is this structural biology consortium in the UK, are putting together this challenge, we’re calling it CAHRA: Community-wide Assessment of Heterogeneous Reconstruction Algorithms, where we’re collecting datasets of complex mixtures and just seeing how people do with existing methods or workflows.
Abhi: Up until Hydra came out, or maybe methods like it, did datasets of heterogeneous mixtures not exist?
Ellen: No, I do think that’s one of the challenges for the field is people are just working on their bespoke protein complexes and then they deposit the data, but it’s not targeted towards algorithms development or pushing the capabilities of maybe extreme compositional heterogeneity. That’s something that we want to work towards.
[00:22:19] Datasets in cryo-EM
Abhi: When people deposit their data in the Cryo-EM field, do they deposit... obviously they put the electron density structure on the PDB... or is there some practice of also giving the ice crystal images?
Ellen: Yeah, so we deposit... people deposit both the atomic models and the density volumes. And something that is common but not ubiquitous is depositing the raw electron microscope data to EMPIAR, which is another publicly accessible database. And that’s where things get a lot messier. ‘Cause the raw data is usually much, much larger, like terabytes of raw imaging data. And the processing is challenging. And so... I think that’s a major challenge.
Abhi: So even if we lived in a world where everyone deposited their raw data onto these platforms, even then it would be such a headache to deal with the data that it’s not necessarily clear that it would translate to all that much utility.
Ellen: Yeah. And also, yes, both that, and there’s no standard metadata and things like that. So there’s maybe just some system-wide logistical challenges and the fact that the data quality is very heterogeneous. So sometimes you have really high-quality micrographs and other times it’s not that great.
Abhi: How much is the generation of the micrograph almost a skill issue on the person who actually generated the data versus... there are just some proteins that are very difficult to characterize well, and the data will always be low quality.
Ellen: I would say both. Okay. Right. Yeah. This is why experimental wet lab biology is just wild and really challenging. It’s definitely both. The hands matter, right, in terms of making the sample. And then it’s totally the case that some protein complexes are just really sticky. They adhere to the air-water interface. The images just look weird and, you know, there’s a lot of sample preparation challenges that are very system-specific. So in that way, it’s similar to X-ray crystallography where you just need to do brute force guess-and-check to get your protein to crystallize.
Abhi: Do you think we’ll ever live in the universe where Cryo-EM and maybe NMR will be the only determination methods that anyone does? Or will there always be a place for X-ray?
Ellen: Oh, I think there’s always going to be a place for X-ray.
Abhi: Okay.
Ellen: Yeah. I think Cryo-EM is very good at large complexes and the sample prep is easier ‘cause you don’t need to crystallize. But if you want to do rational-based, med-chem-driven, structure-based drug discovery, you probably just want to crystallize your protein target with your small molecule.
Abhi: Oh, there’s no... even if we had the greatest whatever version of Cryo-EM in the world, it probably would not be able to resolve a small molecule.
Ellen: It’s not the best for small things because the smaller the target, the less signal you’re going to get... because the less electron scattering you’ll get from the microscope.
Abhi: Has there ever been a paper that has tried to actually detect whether you can find a small molecule from a Cryo-EM image?
Ellen: Yeah. So you can definitely solve small molecule structures. It just becomes literally exponentially harder. Okay. Because you need more and more data to get higher and higher resolution... because you’re just fighting against the noise and there’s an exponential decay in the signal as a function of resolution or frequency. So you just need more and more data to solve higher and higher resolution structures. So if you really want to see atomic-level detail of small molecules, you just need a ton of data.
[00:26:25] Why isn’t there a foundation model for cryo-EM particle analysis?
Abhi: Okay. So we went through ab initio reconstruction, conformational heterogeneity, compositional heterogeneity. I think an interesting quirk of perhaps all of the models that you created is that there is not a single universal pre-trained model for any of this. They’re rebuilt for every new protein that comes in. Why? This just feels alien compared to most machine learning problems. Why was it initially formulated that way when you first came up with the idea and why has it stayed like that?
Ellen: Yeah, I think that’s a really interesting question. And it is very different from the rest of bio-ML, right? Where you have a large, maybe pre-trained model that you can now... like AlphaFold, you train on the PDB, and now you can use it to predict new structures. And in Cryo-EM, and these CryoDRGN-based models, we’re always training a new model from scratch. So when people are analyzing data, it’s some structural biologist who’s training this deep learning model, which is super cool, to analyze their data.
And there’s two things. One is... we’re always... the problem is that we want to infer the structure. We want to infer... we want to solve this inverse problem of what is the actual structure from the experimental data. So it’s very much this active problem of inferring the signal from the experimental data and...
Abhi: I guess instinctively, surely there is translatable information from one protein structure determination problem to another. Has anyone tried to build a pre-trained foundation model for all of this? Or maybe they have and it just doesn’t work that well in practice?
Ellen: Yeah, I think there’s definitely aspects of the problem that make it really hard to train a general model. The first is that the images are extremely noisy. And so we’re already trying to do this very hard inverse problem of what is the 3D structure given these noisy 2D images. And traditionally, the field has been very averse to bringing in prior knowledge because then you can really easily trick yourself.
Abhi: Oh yeah. Okay.
Ellen: And the thing we’re interested in is solving the *new* structure. Right. We’re not interested in pattern matching existing structures. And so, because of the high amount of noise, you can really easily just overfit to noise, align all the noise, and get whatever you want from the data.
Abhi: But I would imagine... yeah, like pre-training people would’ve said that also pre-AlphaFold. Is it... how real is the paranoia that you may potentially hallucinate something as part of the electron density that doesn’t actually exist?
Ellen: So I think it comes back to what is exactly that you want to get out of this experiment. Like, what is it that you want to get out of this multimillion-dollar microscope. And there’s definitely... I do think something our group is working on are these generalizable models that can actually learn across datasets. It’s a very hard problem. It’s unlike any other domain of machine learning. So I think it’s interesting to work on.
It has all these caveats and disclaimers of, “Okay, if you do this, the prediction from maybe this AlphaFold-informed reconstruction model... is it actually the structure from the data? Or is it a structure from this extremely knowledgeable prior that knows everything about proteins that currently exists in humanity?” And I think it’s more interesting to be sure that this is what’s in your data.
Abhi: That’s fair. This is so expensive, so time-consuming that whatever the output of it has to be correct.
Ellen: Right. And like, why else are we spending millions of dollars on these microscopes? Like just tell me where the atoms are. Don’t tell me what AlphaFold already knows. And I think from the discovery, from the actual scientific perspective, that’s where we can discover new things. Like new things that we don’t know currently about structural biology or protein structure, or maybe antibody CDR loops... things that AlphaFold can’t predict or these structure prediction models can’t model that well.
So the whole promise of structure determination is to be able to determine these new structures. And so the generalizable foundation models for Cryo-EM, I think, would be very useful for speeding up the process and would be very useful as... I think it’s an interesting research direction, but I’m very much of the mind that we want to let the data speak... and to let the data show experimentally what’s going on with these proteins.
Abhi: I’m not sure how much you’d be able to talk about what you think the next generation of methods would look like. But if you have any insight into where things will go beyond Hydra and CryoDRGN, I would love to hear your thoughts.
Ellen: Yeah. So I do think it’s interesting right now that there’s a lot more methods for heterogeneity and there’s a lot more deep learning approaches that have different kinds of architectures and different kinds of inductive biases for the type of heterogeneity or things like that. I do think at this moment, this heterogeneity problem is very heterogeneous. And so it’s kind of dependent on what people want to get out of the experiment.
Abhi: Sorry, what do you mean by that?
Ellen: Like, do you want... right now there’s all these different... now we have this potpourri of different methods. You can get atomic... like maybe dynamic atomic models. You can get simpler, linear subspace methods that will give you maybe larger-scale motions. So I guess at this point, what I tell the Cryo-EM community is that now people have a lot of choices, right, in terms of what reconstruction methods they want to use. And so, it’s on them to understand what the priors are in these different methods. Like, is this a method that only models conformational heterogeneity? So if your data has any compositional heterogeneity... so these are mouthfuls... then you’re going to...
Abhi: it’s easier to shoot yourself in the foot.
Ellen: Yeah, so maybe we’re in this risky time. I don’t know.
[00:33:07] How much practical usage is there of these cryo-EM models amongst wet-lab cryo-EM researchers?
Abhi: Maybe that lends itself well to the question I had, especially while I was writing a Cryo-EM article. How much use do these methods have today? Amongst non-machine learning people who... they don’t really believe in machine learning at all. Their job is pure structure determination. How much do they use stuff like CryoDRGN?
Ellen: Yeah, that’s a great question. So I’m always delighted when people are actually using these methods. And so there’s the Cryo-EM methods community and machine learning community which are actively working on new methods, and that’s maybe more cutting edge. And then there’s this much, much larger community of actual practicing structural biologists. And I would say there’s definitely overlap in the methods, but it takes longer time to adopt.
And something that I am very satisfied with during the whole CryoDRGN development was actually the educational part of... when we’re training these deep learning models and learning this low-dimensional latent space describing different conformations, like what does that even mean? Right? So there was a period of just going on a speaking tour and talking to a bunch of research institutes and research groups to tell them just how these methods work and that they’re not magic, they’re not hallucinating things. There’s art to training them for sure, but how do you even interpret the outputs?
So, I would say they are used, but there’s for sure a challenge in terms of understanding the outputs of these models and really taking advantage of them to extract all you can from your data.
Abhi: I don’t know too much about the culture amongst Cryo-EM people, but I know a little bit about the culture amongst medicinal chemists and it seems like molecular simulation is really useful in that area. But if you are doing molecular simulation in medicinal chemistry, you’re usually a pure computational chemist who thinks and breathes computational chemistry. And if you’re very much into doing things in the real world, you don’t know how to do molecular simulation at all. So there’s one group at the company devoted to just pure molecular simulation, another group devoted to giving results to the molecular simulation people. I guess this is all to set up the question of: amongst the people who are actively users of these models, of these techniques, how often is it that they need to deeply understand machine learning and that they are kind of brought up in the culture of computation?
Ellen: Yeah. I think there’s two ways. I think this bifurcates. If the methods get good enough, if the models get good enough, then it becomes truly democratized, right? And then you don’t need to understand how they work. And you can just black-box it, use the methods, use AlphaFold, look at the outputs, and as long as you know enough about how much to trust it, then it’s great.
And I think that’s for sure a direction that’s worth going towards: having robust enough models with reasonable defaults that people can just run and not need to bother themselves with all the details of how it works. On the other hand, if you understand how these models work, you can get so much more out of it, right? You can debug, you can really get a lot more out of your data.
And so I do think right now... I hope that people are in this place, and maybe these days with AI-based tools it becomes a lot easier to learn about all these different areas that one needs to be adept in to use these models. But I think Cryo-EM structure determination is still very much an area, it’s a niche where you need to deeply understand so many different aspects of this pipeline vertically, and if you do that, you have this superpower. And I think that’s still very much where we are. And I think maybe that limits the accessibility of the method, but it’s where you can actually make magic happen.
Abhi: Yeah, What do you view as the biggest barrier to adoption? Is it entirely just education, the models getting better, or some secret third thing?
Ellen: I think it’s definitely... usability is huge and also interpretability of the output.
Abhi: Aren’t the outputs immediately interpretable because it’s speaking the exact same language as the practitioner? What’s the interpretability issue?
Ellen: I think the... I mean... yeah, this is... so now that we have... now that we live in this world where you can reconstruct these molecular movies... the challenge is now what? Like, what do you do with the molecular movies? How do you actually then analyze the ensemble? And depending on the question that the structural biologist is asking, then maybe they just need the two end states or something. Or maybe they actually want the movie. But I think this is actually where it becomes perhaps overlapping with the MD community just in terms of the way they analyze the ensembles. But how do you extract the insights from the distribution of structures?
Abhi: Yeah. I mean, I guess you alluded to this earlier about how there’s currently a ton of methods each with their own biases and failure modes. Do you think we’re slowly tending towards an AlphaFold-esque or AlphaFold 3-esque era where there’s a single model that does it all and it’s push-button?
Ellen: Maybe the field is moving a little bit towards that because of the software packages that exist. So I guess the other thing about whether people use the models or not, and I think this is probably true for computational biology in general, is if you want people to use your methods and tools, they have to be easy to use.
Abhi: Sure.
Ellen: Just from an actual UI perspective and software design perspective. And so right now we’re trending towards these universal tools based on the ones that are actually easy to use... for people who don’t know how the command line works and things like that.
Abhi: Do you think where the CryoDRGN software package is today is... people who have never used GitHub before can just use it just fine?
Ellen: Hopefully. I think that was definitely something that I cared a lot about is the usability of the tool. And I have gotten good feedback on it, but, you know, it’s still a command-line-based tool. It’s still training a deep learning model. And I guess on my end, I had to learn so much Cryo-EM to get into Cryo-EM. There’s so much jargon in this field. There’s so much to know. And so... yeah, I think it’s a good life skill to force these structural biologists to learn command-line-based tools... and to learn maybe Jupyter Notebooks or Colab-based things. And so I think this is a win-win.
[00:40:34] Where can things still improve?
Abhi: Are there some niches even within Cryo-EM that there are no useful deep learning tools available for today? And it’s very much like the art that... whatever they’ve been doing for the last 30 years, it’s probably the best thing.
Ellen: That’s an interesting question. You know, neural network-based models have definitely taken over a lot of parts... have eaten a lot of parts of that computational pipeline, but definitely not all. I think the image processing pipeline, there’s a lot of different stages and a lot of them are just simple function-fitting classical tools. And there are parts where neural-based methods have... particle picking is one where it’s still very much a hard problem to identify the protein within the larger micrograph, the larger field of view.
Abhi: Like, what part of this ice image actually contains the protein?
Ellen: Okay. Yeah. So when you put your sample in your microscope, you take a picture. And so the picture is called the micrograph and there’s, depending on the concentration of your sample, all the individual particle images of your protein floating around in the ice. And then you have particle picking algorithms, basically segmentation algorithms, that identify... that detect the locations of the particles and segment them out for this 2D to 3D reconstruction problem.
And that’s still a very hard problem. You know, you have... very immediately there were these computer vision-based, CNN-based tools to particle pick. There’s still classical, just cross-correlation template matching tools that are used. And, you know, depending on that particle picking algorithm, do you see the rare views, do you see the rare conformations? And that’s still a hard problem. And so people are using all these different tools, but definitely just going back to the data a lot to reprocess and...
Abhi: Well, I guess the way that you’re phrasing it right now makes it sound like the simpler methods work fine. Is that the takeaway? That deep learning or the inclusion of deep learning here has not yet demonstrated extraordinary results beyond the usual ones? Is that fair to say?
Ellen: Yeah, I think... I mean, I think there’s... it depends on... there’s pros and cons. So the deep learning-based tools work, but they require a little bit of fine-tuning actually.
Abhi: Okay.
Ellen: Or you need to retrain a new model every time. And actually, these days, the microscopes are super automated, are really fast, and the image processing is now a huge bottleneck in terms of solving structures. And so if you want speed, you just use maybe the simpler-based tools.
Abhi: Okay. Oh, sorry, go ahead.
Ellen: Oh yeah, just the simpler tools.
Abhi: The one other question I wanted to ask was, beyond machine learning, how much innovation do you expect to happen in the physical processes of collecting the data?
Ellen: So that’s also very much still actively being developed in the field. So like there’s people at Berkeley working on these crazy laser phase plates to just fundamentally increase the amount of signal in the collected images, and that will just solve all the downstream problems. Like if we could magically get less noise or be able to irradiate our sample with more electrons before blowing it up, then that solves all of our downstream problems. So that would be great. So people are very much still working on the hardware, which is why this field is so rich in terms of the different areas that you need to understand.
Abhi: I was going to... if someone is at OpenAI and working on pre-training, maybe three years ago, all they needed to care about was, “Oh, how do I distribute my data across multiple GPUs better?” But now maybe things are getting so close to the edge of what’s possible that they now need to start deeply understanding kernels. They need to start paying attention to every software update that Nvidia turns out. How much do you personally stay on top of the hardware innovations that’s going on?
Ellen: So I try. I think one of the privileges of my job right now is that I can do the research, but I can also... I also, you know, both get access and can talk to all these people in all these other areas. And I think it is really important to stay abreast of all these other directions to understand which ones will completely change our field.
Abhi: When I look at the field of at least protein structure prediction, it feels as if to me there is not much room left for machine learning advances. And most of the innovation will just come from being able to collect the data better and faster.
Ellen: That’s a hot take.
Abhi: Yeah. Potentially it’s wrong, but I’m curious on your side... how much alpha do you think there is on improving... churning the ML crank faster and faster versus just trusting the people at Berkeley will solve the problem by better data collection? Like how much room is left to go with stuff like CryoDRGN?
Ellen: I see. I think there’s still lots of interesting moonshot problems on the ML side. And one of the main directions that we’re interested in is can we innovate on the algorithms enough to be able to change how data is collected? Right. So one major trend of the field is moving towards less pure samples. So before it was completely pure. And now with Hydra and these kind of extreme compositional heterogeneity, we are moving towards dirtier lysates or just cellular fractions or the in situ slices. So that’s definitely a trend. And that’s only going to be possible if the algorithms can either keep up or can the algorithms themselves motivate new ways of collecting data.
[00:46:57] Has deep learning done something in cryo-EM that was previously impossible?
Abhi: I think... all my examples are coming from the protein structure prediction field. But... I think it took a few years post-RFdiffusion being released, I actually saw for the first time that, “Oh, these models have given us research papers... like clinical-stage research papers... that otherwise could not have existed if this model did not exist.” Are there some Cryo-EM-determined structures out there that exist and are deposited that you suspect would not have come about had stuff like CryoDRGN not existed?
Ellen: Hmm. Oh, I don’t know. I feel like it’s too...
Abhi: I guess this is a strong claim to make.
Ellen: Yeah. It’s too strong of a claim to make. I mean, the hope of these tools is yes. Right? Like, and in the original publication, the satisfying part was re-analyzing previously published datasets and finding structures that were missed.
Abhi: Okay. That’s something.
Ellen: So that was awesome. Yeah. That was super cool, especially because at the time I didn’t even have the taste to understand what I was doing in terms of reanalyzing the data. And I was like, “Oh, this is weird. I guess it was not there.” And then it was like, “Wait, this was not there before. That’s cool.” Yeah. So I think that... yeah, that was, I guess, super gratifying.
[00:48:22] Ellen’s experience in the cryo-EM field
Abhi: Going back to when you began in this field entirely and you need to come up to speed on Cryo-EM and all the jargon that that entailed... what was the biggest, almost like culture shock that you had coming into the Cryo-EM field versus... I’m assuming you were primarily like a physics ML person prior to this.
Ellen: Well, I was actually an MD person, or molecular dynamics person prior, so yeah, very new to Cryo-EM. And I talk about... so there’s a bunch of mathematicians working on the Cryo-EM problem and those papers are a lot easier to read because there’s less jargon. So that was maybe my initial foray, but then, you know, just reading all the papers in the field and just trying to make sense of the jargon was the biggest shock originally, just as a graduate student trying to figure things out.
My first time actually talking to... meeting people in this field... was also just interesting in terms of... I guess the jargon was the main thing. Of just like, “Okay, what is... what is pseudosymmetry?” You know, “Is it symmetric or is it not symmetric? Like what the heck is pseudosymmetry?”
And so... and I remember one of the first times I gave a talk on CryoDRGN to a primarily structural biology community, they were really kind of suspicious of deep learning. Yeah. So I think there was a major sea change in the structural biology community because of AlphaFold.
Abhi: Okay.
Ellen: So that was a pretty major shift... a vibe shift... of whether people trust deep learning-based approaches or not. And so definitely before that, there was a lot of skepticism, but there was a lot of excitement too, because heterogeneity was the next frontier of the technology and “how do we model continuous motions?” And so I think, yeah, in the beginning people were extremely excited and I was trying to not overhype it, you know, of just like, “Don’t get too excited.” This is not going to solve all your problems. Like, if it’s garbage in, if your sample is really, really challenging, it’s not going to necessarily solve all the problems.
Abhi: Do you often find that, at least in... actually I’m not going to make any broad statements about proteins again. At least within your group and perhaps other Cryo-EM ML research groups, how much of the innovation comes from talking to Cryo-EM people and listening to their ideas and their research ideas? Versus looking at the ML literature. Like, I’d see... I think to take a concrete example, there feels like a pretty strong connection between your work and the NeRF work... Neural Radiance Fields. How much of it is... is that where you’re taking inspiration from the ML literature versus taking inspiration from traditional Cryo-EM?
Ellen: Yeah. So I think now... I think my group has people who are very much computer vision, machine learning people, computer scientists, people who are more mathy. We have a biochemist, a biophysicist. So I really love how interdisciplinary or multidisciplinary the group is because you need to draw from all these different kinds of areas.
Originally, CryoDRGN was... I had no idea about NeRF. And so at some point... CryoDRGN preceded NeRF. And then when I saw... at some point somebody was like, “Oh, you should check out this Neural Radiance Field.” I was like, “Whoa, that’s super cool.” And I had no knowledge of computer vision. And I remember my PhD advisor at the time being like, “Oh, you should think about whether this type of architecture can be useful for other domains.” I was like, “Nah.”
Abhi: You could have been a graphics researcher.
Ellen: I could have done graphics. Yeah. So I guess... yeah, definitely it’s good to stay abreast of what’s going on in machine learning. What are the trends, what are the latest and greatest architectures and training paradigms and activation functions and things like that. But I think it’s extremely important to stay focused on the problem at hand and focused on what are the challenges associated with a specific problem... and whether the techniques can be cargo-culted or not.
Abhi: Gotcha.
Ellen: And most times they can’t, but... related things.
Abhi: So I’m assuming you aren’t going to SIGGRAPH every year to see what inverse problems people in the graphics field are working on?
Ellen: No, I’ve never been to SIGGRAPH, but I sometimes go to the computer vision conferences. And I think it’s nice to establish crosstalk between all these different communities, because the problems are so related. But yeah, the thing to fight against is just chasing the trends in these other areas and applying them to Cryo-EM. I don’t think that’s the way to go about it.
[00:53:40] Deep learning in cryo-EM outside of structure determination
Abhi: Outside of structure determination...
outside of applying machine learning to particle analysis, you mentioned that there is machine learning being applied to actual particle picking itself. What do you view as the most promising direction outside of particle analysis? Is there anything that you view as particularly interesting outside of what the Ellen Zhong lab is working on?
Ellen: Oh, well I was going to interpret your question as outside of the 3D reconstruction problem.
Abhi: Sure. Yeah.
Ellen: Which is also something that our group is working on right now. But outside of... so really we’ve been focusing mostly on this 3D reconstruction, so just 2D to 3D image processing. The direction that we’re getting into now is this CryoBoltz paper. Yeah, so the atomic modeling. And I think that’s something that’s really interesting. It’s an unsolved problem.
It’s largely done manually still. It’s unclear how to do it for low-resolution maps, but how do we actually build the atomic models into the Cryo-EM density volumes? Especially when we have an ensemble or distribution of structures from CryoDRGN. That’s currently unsolved. How much should we rely on priors from structure prediction models? Building in the atomic models...
Abhi: How much do you personally trust them?
Ellen: I think we should use them. We should definitely use them for everything they got. I think now the challenge is validation. Like once it spits out a structure, how do we actually validate at scale? If we have a thousand Cryo-EM volumes that we’re building atomic models in, how much information do we share across those thousand structures? And now how do we automate the validation procedure? Because any practicing structural biologist, when they deposit an atomic model, they’re going through every single residue or every single atom, residue by residue. And... so should we still be doing that? That’s an open question.
Abhi: Earlier when I asked about why pre-trained foundation models aren’t used in Cryo-EM, you said you want to actually recapitulate the data as you measured it and not potentially hallucinate something. But it does sound like if you’re going in the CryoBoltz direction, it is on the table to hallucinate something.
Ellen: Yeah.
Abhi: How much have you seen that happen in practice?
Ellen: That’s a great question. I think one... I guess the way that I’m approaching it right now is separating the reconstruction problem from the atomic modeling problem.
Abhi: Okay.
Ellen: People are definitely trying to bring them together and having reconstruction models that are deformations of an atomic model. And that’s where I think you can get hallucinations. And I have seen... there was this really interesting case study that one of our collaborators at Princeton showed us, where if you have a homogeneous dataset... so synthetic dataset, you can create a synthetic dataset of just a static structure... and it’s extremely noisy. And if you fit one of these heterogeneous reconstruction models, you get... you hallucinate conformations. Especially if it’s one of these models that only models conformational heterogeneity. You just get these flexing motions that are definitely not in the data. And so yeah, that happens. Right. And that’s something that is a worry.
[00:57:32] 3D volume reconstruction versus residue assignment in cryo-EM
Abhi: I actually hadn’t mentally separated this 3D volume reconstruction problem as something you do first and then you assign residues. I assumed... I guess now looking back at the actual papers, it does seem like they are two sequential steps. Is it obvious... going back to proteins, there’s a similar question of, “Should you design the structure first and then the sequence, or maybe you do both at the same time?” And the canonical thought has always been, “Oh, you want to design them at the same time because they rely on each other.” Is there any similar undercurrent of thinking within the Cryo-EM field where you want to be able to do both at the same time because it just helps you succeed at each of them individually?
Ellen: Yeah. So I think by separating them out, you are separating the 3D structure determination from the sequencing of the actual protein. And obviously, maybe you needed to sequence something to make the sample in the beginning... but not necessarily, especially if you’re just extracting from the wild.
Abhi: Yeah.
Ellen: And so it’s been used... Cryo-EM has been used in this bottom-up fashion just to discover new proteins, to discover new complexes, new interactions. And then it becomes a hard problem of how do you sequence the thing that you’ve solved the structure for. And I think now in this post-AlphaFold era where we have structural hypotheses, it becomes less of a completely crazy problem. But I think that is one of the exciting things. And then in an actual design context, you could imagine high-throughput structure determination of an ensemble of different design sequences and then seeing afterwards which ones bind or something like that.
Abhi: Yeah. How much... actually, this is a question I probably should have asked a long time ago... but when you’re actually assigning atoms to this three-dimensional model, how often is it the case that you know what residues are on the table to start off with versus you don’t know anything about the sequence?
Ellen: I think most of the time you’re trying to solve the structure of something whose sequence you know.
Abhi: Okay.
Ellen: Yeah. You want to determine this new protein structure, this is the sequence that you’ve purified out. But especially if it’s from an endogenous source, you never know what else is going to be there.
Abhi: I imagine like lysates...
Ellen: Yeah. If it’s from a lysate, like whether there are ligands bound, whether there are other complexes that are co-purified. And so that’s the thing that from the machine learning standpoint is an interesting axis: is how much do you rely on priors and how much do you allow for discovery?
[01:00:26] What did Ellen do during her sabbatical at Generate Biomedicines?
Abhi: Okay. That makes sense. And I know that you posted on Twitter a while back that you recently took a sabbatical from Princeton to go to Generate Biomedicines for a bit.
Ellen: Yeah.
Abhi: And I learned this morning that that sabbatical has now ended.
Ellen: Yeah.
Abhi: What were you working on there? And I guess... so for some context, Generate Biomedicines bought this 70,000 square foot facility, I think in 2023, and just filled it with Cryo-EM machines. So clearly there’s an overlap as to why they actually need your exact skillset.
Ellen: Yeah.
Abhi: What were you working on there?
Ellen: Yeah. So at Generate... so that was an amazing sabbatical. Generate has four cryo-electron microscopes, which is double the number of Princeton University. And all these challenges that we talked about in the past of, “Okay, the datasets come from all these different research groups around the world.” I mean, they’re aggregated in these databases, but the quality is kind of different. And Generate, you know, that’s a single organization where they can actually standardize everything and pipeline everything for high-throughput structure determination.
At Generate, we were working on structure determination and solving the ensembles of antibodies... the conformational ensembles of antibodies, and trying to ask interesting questions about the role of antibody structural dynamics in binding, in affinity maturation, and things like that. So that was super cool, but very much from... “Can we just actually observe?” Right. So... I learned a lot about antibodies. It was amazing. But yeah. So... is there a logic or grammar to antibody CDR loops?
Abhi: Why... what’s the utility of structure there?
Ellen: Hmm. Yeah. Great question.
Abhi: ‘Cause in my head, CDRs are floppy. Is that just completely incorrect?
Ellen: In my head, they are too, but no one knows. Right? Okay. Because no one’s observed the floppiness because we haven’t had tools to be able to observe conformational dynamics or conformational ensembles until, I guess, Cryo-EM. And it’s still a very challenging sample prep problem, especially if you’re trying to scale it up. But now can we actually just observe from experimental data what the actual floppiness of the CDRs are? Is that actually the case? Or are some more rigid than others? What’s the sequence determinants of that? And obviously, there’s lots of therapeutic relevance if you understand... if you can now rationalize the structures of the loops.
Abhi: How much are you able to share about how floppy they actually are?
Ellen: Oh, well, very much still a work in progress. But it is publication-oriented.
Abhi: Okay. Gotcha.
Ellen: Which is why I’m really excited. And definitely why I was interested in doing the sabbatical there.
Abhi: What is the clinical relevance of it? Like, let’s say you are able to determine that the CDR loop is floppy, it binds in this specific way. What do you tell a therapeutics team as the result of this process?
Ellen: Yeah. I feel like the therapeutic angle is above my pay grade or outside of my actual area of expertise. But I would imagine that the floppiness has some functional relevance. And if you can better understand, characterize, or design the function of the binder, maybe you would get fewer off-target effects or you would just actually have more useful functional data about your antibody or about your therapeutic.
Abhi: Did the experience make you desire building therapeutics or did you go the other direction, thinking “this kind of sucks”?
Ellen: Yeah. No, no, no. It was actually extremely inspiring. I would say my prior has always been more on the academic side of, “Oh, basic science. There’s so much we don’t understand about biology. I think working more upstream on the fundamental things is more... I don’t know, just interesting.” And then actually being there and realizing that these are real drugs that are actually helping people was very inspiring. I was like, “Oh, yeah.” So that was cool. That was super cool.
Abhi: The... when I saw that they acquired all of these Cryo-EM machines... I did not know it was only four. In my head, it was dozens.
Ellen: Four is a lot!
Abhi: Yeah, four. Four seems to be a lot. My initial thought was like, “Oh, they want to... they’re the ones who made that protein structure model Chroma. Maybe they want to scale up protein structure determination using these.” Was that also on the roadmap or is that...
Ellen: Yeah. I guess... so I’ve known some of the people at Generate for a long time. I think Gevorg Grigoryan, the CTO, John Ingraham, the head of ML there... their papers I read when I started as a PhD student. I was like, “Whoa, these are super cool protein ML papers.” And I think their hypothesis... okay, who knows? I don’t know. But I think definitely structure prediction is a huge capability, or generative models over protein structure. And then can we use those models for design? But obviously... I think very much this area, you don’t want to stay purely in silico, because if you actually want to make real therapeutics, you want experimental data, ideally in high throughput. And so I think probably Cryo-EM was a major part of that and that investment was... ahead of its time, I would say.
Abhi: Yeah. There’s one other therapeutics company that has made a really big bet on Cryo-EM. I think it’s Gandeeva Therapeutics. Do you happen to know what their particular bet is?
Ellen: I think there’s a couple. There’s definitely others that I’m aware of that are using Cryo-EM. And I think the idea there is very much to use Cryo-EM to understand maybe cryptic pockets, or... I don’t actually know about Gandeeva in particular, but just my understanding is with Cryo-EM you can actually get the ensembles, and with the ensembles you can maybe find new binding sites or better understand allosteric mechanisms and things like that.
[01:07:07] Ellen’s research in cryo-ET
Abhi: You’ve done a few things besides Cryo-EM as well. I’m going to talk about one thing that your future work is going to be focused on, and one thing that your past work is focused on. The future thing seems to be cryogenic electron tomography.
Ellen: Yeah.
Abhi: The basics of the technique, I think, is just exactly the same as Cryo-EM but you’re tilting the sample so you get... actually, maybe it would be best if you explain it.
Ellen: Yeah, yeah, yeah. So cryo-electron tomography or Cryo-ET is using the same microscope as in single particle Cryo-EM. But there’s two main differences. One is the tomography part is now, instead of just taking a single projection image through your ice, you’re now tilting the imaging stage, like plus or minus 60 degrees is practical, and taking a series of images at different angles. And so then you get a 3D tomogram. And so you can see things in 3D. So that’s one main change is the tilting.
The other change is that it’s usually used to image in situ samples, so like thin cellular sections. So instead of a purified solution, you’re looking directly in situ.
Abhi: Sorry, that I guess maybe answers my follow-up question of, “Can you not approximate this three-dimensional view from Cryo-EM?” But I guess the in situ part demands that each molecule is unique and you’re not going to see it repeated across the entire data.
Ellen: Yeah. So the spatial scale is different. So now you’re seeing subcellular architectures and membrane morphologies or organelles and things like that. You can still identify individual particles... and then pick all those particles. And the individual particles are super noisy and super low resolution... but you can do the same trick as in single particle Cryo-EM, combine them all together and computationally amplify the signal to get near-atomic resolution structures in situ. And so the extremely exciting part there is that now we’re determining the structures of these protein complexes instead of in a crystal, which is from X-ray crystallography, or instead of in a purified solution. Now we’re actually looking at the functional structures in the native environment.
Abhi: One instinctive question I have is...
Ellen: And you also get the spatial organization too, which is super cool.
Abhi: How large are these samples? Like, are you able to see them with your naked eye?
Ellen: No. So I think the field of view is still pretty small. So there’s many orders of magnitude between angstrom-scale things that we resolve to maybe nanometer, hundreds-of-nanometer scale sections, to whole cells. And definitely, whole-cell visual proteomics is a huge research direction for the field. Can we have those beautiful David Goodsell images of these cellular landscapes, but from experimental data?
Abhi: I was going to ask, how far are we away from being able to image an entire cell at once? Has that already happened?
Ellen: So that has already happened. It depends on how big the cell is. Okay. Yeah. And so people have done this with bacteria, which are much smaller, but for larger, more complex eukaryotic systems, it’ll require a lot of data. And then it’s an interesting question of whether you want to be hypothesis-free about it and just see all the things, or whether you want to test a specific hypothesis and have “with a drug” / “without a drug.” And I think that’s a challenge or maybe a research question for the field.
Abhi: What’s an example of a hypothesis-driven question you’d have about a Cryo-ET result? Is it like, “Does this particular protein exist in this native environment?” or is it something coarser?
Ellen: I think people are interested in personalized medicines. Right? Okay. So like, does the morphology of some organelle change? And so it is tackling this length scale that’s unique, right? Light microscopy can only get you so far, can only get you the wavelength of light, which is thousands of times larger than the wavelength of an electron.
Abhi: Yeah.
Ellen: And so with Cryo-ET, you can actually see maybe subcellular features and architectures. And then for certain complexes that are abundant, you can get molecular details and atomic details. And so really the dream would be to be able to span all these different length scales from atoms to cells.
Abhi: The idea of a Cryo-ET result being clinically relevant is interesting. I don’t think I’ve ever heard a claim like that before. Is there some particular disease you’re aware of where if we could exactly measure the physical structure of the mitochondria or something, that would help a lot? Or is it kind of a basic research thing where we’re not sure how this will help?
Ellen: Yeah, I am not sure. I feel like there’s probably scope, but it’s way above my... or I don’t know enough about the clinical side of things. It is an expensive experiment. So I think that’s one of the main challenges is the facilities... it’s expensive, it’s hard to do, it’s immature still in terms of... you know, you need a lot of... the sample preparation... everything is very much still... people are working out how to do it the most effectively and efficiently.
Abhi: So it’s like, it’s a newer method than Cryo-EM.
Ellen: Yeah. Okay.
Abhi: Is it... do you imagine... the road to getting more traction is there? It’s already shown early promising results. Maybe like Cryo-EM won the 2017 Chemistry Nobel Prize. Is this on track to win the 2035 Nobel Prize?
Ellen: I have no idea. I have no idea except that I do think because it has this unique vantage point in this length scale of cellular and molecular biology... it really spans molecular and cellular biology... it can be used to find new things and make really important discoveries for sure.
[01:13:54] Ellen’s research in NMR
Abhi: Yeah. That makes sense. Another thing you mentioned to me just when we were talking this morning is that you were also working on NMR.
Ellen: Yes.
Abhi: Which is new to me. Yeah. What are you doing there?
Ellen: Yeah, so that’s been super exciting just from a personal level, because I’ve been working on Cryo-EM for a long time and there’s so much more to do there. There’s a lot of really interesting open research questions. But we do have a new direction in the group that is small molecule structure determination.
Abhi: Mm-hmm.
Ellen: So I guess that’s the commonality is: can we analyze... can we just do useful things and help chemists? Now we’re talking to chemists. Can we help chemists elucidate the structures of novel natural products and novel molecules? That is still very much a bespoke method... done manually, requires expertise to just stare at these NMR spectra and figure out what the graph or the structure of the molecule is. So yeah, I think that’s super interesting. When I first learned about that problem, I was like, “Ooh,” I had the same feeling as the Cryo-EM problem. Like, “This is a really cool problem and I’m not aware of anybody else working on it right now. And this seems like a really useful problem to solve.”
Abhi: So the mental framework... the mental framework necessary to understand Cryo-EM was a little bit alien to me when I started reading about it. You have a bunch of these 2D images from an electron beam passing through a protein. You have an electron detector that records a signal and your job is to reconstruct all these 2D images into a 3D structure. What is the mental framework for applying ML to NMR?
Ellen: Yeah. So it’s another... I guess... it’s another inverse problem where we have imperfect experimental measurements. The spectra itself is... the NMR or the shift of the resonance of these different nuclei, either protons or C13 or carbons... isotopic carbons. And depending on the shift, that tells you about the local chemical group and the local chemical environment.
And then the puzzle that usually a human, an expert, will do when they’re looking at the spectra is like, “Oh, there’s a peak around here. So I know that this is this type of hydrogen in a benzene ring or this type of hydrogen. And there’s a peak around here and there’s some decoupling, so it must be near some other type of chemical group.” And maybe you know the chemical formula, like you’ve done mass spec or something. So you know the total composition of the molecule, but you don’t know at all how it’s connected together.
And then from the spectra you just stare at it and from all of your training, you know that these peaks are artifacts, these peaks should be there but it’s missing because this always happens. And so when I learned about this problem, I was like, “Oh, this seems perfect for machine learning.”
And I guess what was most inspiring to me in the beginning about this area of natural products is: if you look at the structures... well one, they’re oftentimes the most bioactive and therapeutically relevant molecules. They’re isolated from natural sources instead of made from rational drug discovery or these drug-like molecules. And they look crazy.
Abhi: Yeah, I’ve heard this from natural products.
Ellen: Yeah, the... if you actually read the papers and look at the structures, I’m just like, “Oh my God, these molecules are gorgeous and also totally gnarly.” They look really exotic. And I’m just like, “Who... this is what the bacteria are making when they’re conducting chemical warfare against each other or whatever they’re doing.”
And then you read the machine learning papers in chemistry and it’s a really interesting modeling problem, but most of the datasets are super small molecules that have fewer than maybe 20 heavy atoms or something like that. And I’m like, “There’s a huge gap here in terms of what we’re actually modeling from a machine learning standpoint and what are the super-cool, active molecules that we’re just extracting from nature.” So that was the main motivation.
Abhi: I imagine... whenever I see a chemical structure of a natural product... yeah, one, it does look often pretty crazy, but two, how does anyone come up with those structures today? Like, in Cryo-EM, when you arrived to the field, there were a bunch of traditional methods being used. Are there a set of traditional methods also used to determine the structure of these really crazy molecules? Or is it like they need to learn how to synthesize it and then from that they naturally get the structure?
Ellen: No, you very much don’t know how to synthesize it. So the synthesis problem is usually downstream of discovering and characterizing the molecule. Because oftentimes, all these really expensive cancer drugs, they’re still extracted naturally because we can’t figure out from a chemistry perspective how to do the total synthesis. And... yeah, I’m not a chemist, I’m just a computer scientist. But I think natural products is interesting because we just mine nature and maybe figure out how to perturb these bacteria to produce all these crazy chemicals. And then we just extract them from natural sources and then do all the biochemistry and physical chemistry and analytical chemistry experiments to figure out what it is... and figure out how it works, figure out how the enzymes that make it work... and it’s just a whole super interesting field that has been... yeah... great to get a sneak peek to in the last year or two.
Abhi: I know very little about this field, but I do remember reading these Mass Spec Foundation Model papers from Enveda Biosciences, if you know them. And they claim to be able to turn the mass spec peaks into the actual structure of the molecule. How trustable are those methods?
Ellen: I think the mass spec stuff, I don’t actually know that much about. I do think metabolomics... there’s a lot... that’s an interesting problem, especially when you have complex samples and things like that. But from our chemistry collaborators, I think when you’re trying to solve the structure of a new natural product, you typically use NMR.
Abhi: Okay.
Ellen: Which gives you more information on the actual connectivity of the atoms.
Abhi: And so the mass spec information is underdetermined for the structure problem?
Ellen: Yes. There’s just not enough.
Abhi: Okay.
Ellen: I think mass spec is usually used to identify composition or just identity of, “Is the molecule there or not?” if you already know the molecule, you know its spectral signature.
[01:21:05] How did Ellen get into the cryo-EM field?
Abhi: Right now I’ve talked about cryo-ET and NMR, both of which you’re working on right now. But for a long time... three years you were at D. E. Shaw Research before going to graduate school. And I have two several questions on that front. The first of which is, why start with molecular simulation? Why was that the thing you originally were working on when you were at University of Virginia?
Ellen: Yeah. I would say luck. You know, originally, how did I get into research and how did I get into the area that I was in, is just luck. And I’m extremely fortunate to have met the undergrad research mentor I had at UVA, and from him, that’s where I learned about protein folding and molecular dynamics. And we wrote this super cool... or I thought it was super cool... model to simulate protein folding. And that’s how I learned about Anton and D. E. Shaw Research. And then I was very lucky to work there for an internship and then full-time for three years. And so, getting into that area, I think, was just luck in the beginning. But I’m really glad I did for sure.
Abhi: Do you ever... I think there’s a common pattern of a very smart person who touches molecular simulation for a few years and then gets out of it because they think “this is not going anywhere.” Did you have a similar reaction or was it more like, “Oh, I’m curious about other things besides molecular simulation.”
Ellen: I think a little bit of both. I think working at D. E. Shaw Research, I was working on free-energy calculations and binding affinities. And then through that was working on originally the sampling problems there, which is really interesting technically. And I learned so much. And then after characterizing it on more systems was like, “There’s a force field problem here.” And then worked on some quantum stuff to better estimate the force fields. I was like, “This is a really hard problem.”
And then was like, “Okay, at the end of the day, all these predictions need to be validated experimentally.” And then I had no idea anything about biology at that time or experimental biology. And so that was just this huge area that I was like, “Huh, that’s the... you know, that’s the answer key.” And it’s a huge mystery. And so I think after a couple years, it was definitely following what’s next and wanting to understand the experimental side.
Abhi: Yeah. And were you like a pure physics person by undergraduate training?
Ellen: In undergrad I was doing chemical engineering. And so that’s how... and that’s where I was exposed to stat mech and this protein folding area. And yeah, so it’s interesting. Now I’m in a computer science department. I took the CS classes, but wasn’t that interested in some areas of computer science. And it was too hard to get into the classes ‘cause it was really oversubscribed. And I don’t know, I liked chemistry for sure.
Abhi: Yeah. Do you ever imagine returning back to molecular simulation in any capacity? Or are you kind of set on this Cryo-ET, NMR track? And maybe even those will involve some molecular simulation.
Ellen: Yeah, definitely. I am not against it at all. I think, especially now, there’s so much interest, right, in molecular simulations with neural-based approaches eating various areas of MD. And again, it’s all about what are the questions that this technique can answer. Like Cryo-ET gets this unique length scale. MD also, I feel like, accesses a unique length scale that you can’t get with experimental data. Right?
Abhi: I think, yeah, I saw this really interesting graphic that says the timescales of NMR are incredibly coarse to what you can achieve with sufficiently coupled cluster....
Ellen: Yeah. Right, right, right. And the dynamics... I mean, what originally drew me to the Cryo-EM problem was that, “Oh, we can actually get the dynamics of these proteins, but from experimental data instead of from simulating these very, very simple physics-based models.” And so that was super cool. It’s like, “Oh, this is directly from the data.”
However, in Cryo-EM, the ensembles are still just larger-scale, slow-timescale conformational changes. And so I think you’re never going to get super-fast kinetics from a Cryo-EM dataset. And so there’s still... there’s interesting areas of overlap or complementarity between MD and Cryo-EM.
Abhi: Have you ever felt... does anyone in the Ellen Zhong Lab focus on applying molecular simulation to Cryo-EM? Or it’s somewhat of a nascent field where you’re not sure where it could actually be applied?
Ellen: Some of our collaborators are definitely applying MD to Cryo-EM right now. Oh yeah. And my lab, I call EZ Lab.
Abhi: Oh, okay.
Ellen: For, yeah. Which is a fun acronym. So MD, I feel like, is also... it’s even more so this bespoke method. So... Cryo-EM is bespoke for each dataset, but MD is even more so because there’s this upfront cost of setting up a simulation and all this stuff to validate the force field terms for a particular system. So right now we’re not directly integrating MD-based modeling and Cryo-EM, but definitely using it as an interesting testbed for conformational ensembles from Cryo-EM.
[01:26:57] Why did Ellen go back to graduate school?
Abhi: During your time at D. E. Shaw Research, and the subsequent decision to go to MIT for a PhD... what was the primary motivating factor? I do know there is this track of scientific associate at D. E. Shaw Research to PhD track. Was there anything else that was on your mind of what you could have done besides?
Ellen: Oh. Yeah. I guess at that time... I wasn’t sure. Right. I wasn’t really sure exactly what I wanted to do with my career. But I thought learning more things would be great. And I think concretely, my GRE scores were going to expire, something like that. So I was like, “I might as well apply.”
And I remember going through the process and it was really useful to think about what exactly I want to... what would I want to study for a PhD? And what kinds of things am I interested in? And then after going through that process, it was pretty clear. And after starting my PhD it was so clear. I was like, “This is really cool.”
Abhi: Okay. Have you ever gone back to look at your personal statement, your statement of purpose, and seen how much you’ve deviated since then? Or was it pretty spot on, like, “Oh, I care a lot about validation, I care a lot about validation today as well.”
Ellen: I definitely remember going back to my statement of purpose to look at which groups I was interested in, and it was totally different.
Abhi: Okay. You were not even... were you aware of Cryo-EM at the time?
Ellen: No, not at all.
Abhi: Okay.
Ellen: So that was only after a year... my first year of my PhD, after doing rotations in all these different areas. And I tried experimental stuff, which was cool. Learned how to pipette, which was cool. Did some neuroscience, which was crazy. And I was very open to many different things. And then when I learned about Cryo-EM, I was like “okay”.
Abhi: The last questions I had was, if you had a hundred million dollars to spend on anything, someone gave you that money, no strings attached, what would you spend it on?
Ellen: I think I would be doing exactly what I’m doing right now.
Abhi: Well, sure, sure. Not in terms of, “Oh, I’m going to retire,” but...
Ellen: Oh, like I need to spend it or something.
Abhi: Like, would you buy a hundred Cryo-EM machines? Is each one a million dollars a piece? I’m not technically sure. But what would you be spending that on in terms of accelerating your own lab?
Ellen: Hmm. Yeah, that’s a really interesting question. I definitely wouldn’t change anything about what I’m doing right now. I think the problems are interesting. The people, my group is amazing. And the research... I don’t feel particularly limited. Okay. Maybe if I had to spend a hundred million dollars, I would branch out into more of the experimental side and have more data-generating capabilities to either test hypotheses or to design the experiments. But yeah, I feel extremely lucky right now that we get to just work on the cool problems that are hopefully useful.
Abhi: I feel like that’s a rare answer. Usually when given a hundred million dollars on the table, people are like, “Yeah, I’d buy so-and-so machine, I’d start so-and-so research group.” And especially Cryo-EM feels like the sort of thing that is so bound by the world of atoms. But yeah, I guess in practice, not.
Ellen: Yeah, I think... concretely, right? Like, then I would need to know how to do Cryo-EM experimentally or do NMR experimentally or something like that. And so that would be... that’s a whole other thing that...
Abhi: Well actually, I think that’s an interesting question. Would you trust yourself to do NMR or Cryo-EM independently today?
Ellen: Me personally, no.
Abhi: Really?
Ellen: Like I have no idea how to do that.
Abhi: Really?
Ellen: Like there are people in my group who do collect data. And that’s great, but I’ve never collected data.
Abhi: Wow. Is that the sort of thing where it’s like you genuinely need months upon months, maybe even years of training to be able to do it well?
Ellen: Yeah. I think, like I was talking about earlier, the field is so... it’s so deep. You do need a lot of expertise. And I think there was an opportunity at some point during my PhD to actually think about collecting data or something like that. But it was just never... I don’t know, I liked coding too much.
Abhi: Yeah, that makes sense.
Ellen: And I think... it’s interesting, money is not really the bottleneck. It’s time and bandwidth. And what was so amazing about the sabbatical is like, “Oh, I don’t have to teach now. Okay. And now I can code myself again and run experiments... or computational experiments.”
[01:32:17] What makes Ellen more confident about trusting an external cryo-EM paper?
Abhi: Perhaps a question that I should have asked earlier is: you were probably the person who made the particle analysis Cryo-EM machine learning field big in the first place, and now there’s entire workshops devoted to the research field. With the rise of many more talented people entering the area, I imagine there’s also some deluge of more noisy, lower-quality papers. Is there some easy way for you to tell whether to take a Cryo-EM paper seriously or not?
Ellen: The... I don’t like looking at the field with that lens. I’m like... I think everybody is doing interesting stuff as long as people have scientific integrity. And I think people are interested in different aspects of the problem, and I like that we have that flexibility, right? Of like, “Oh, they’re more interested in proof of concept.” Sure. And are writing conference papers and can get these concepts out there more quickly. And then there’s papers that are more targeted towards users. Right. And actual tools that people will use. And I think as long as there’s a healthy ecosystem where people have the flexibility to work on the style of research and, we’re making forward progress as a field... then, yeah, then I think that’s great that there exist these different types of research directions.
Abhi: I guess concretely... I know, especially in both the field that I work in right now and the field I worked in prior, there are known datasets that are... like people claim very strong things in either toy datasets or known datasets that have really big fundamental problems. I think maybe most overlapping with our work is... I think PoseBusters is known to be kind of a bizarre dataset.
Ellen: Oh yeah. I’ve heard this. Yes. I don’t know why, but I’ve heard this.
Abhi: I don’t know why either. I just... yeah... I’ve had colleagues who say this. I guess the Cryo-EM dataset world is a bit smaller than the protein world and the small molecule world. But are there any issues like that? Actually instinctively, what I thought your answer was going to be was, “Synthetic data is really hard to take seriously.” But maybe that’s not the case.
Ellen: Yeah, I think the proof is in real data and seeing whether a method works on real data. And there’s some established benchmark datasets that people are usually analyzing. So that’s definitely one of the markers of, “Does this... how well does this method work?”
And I mean, one of the challenges for especially new people coming to the field is that you need a lot of expertise... domain expertise... in interpreting the results and interpreting the structures. One of the things that one of my group members did last year, it was at the NeurIPS benchmarks track, was CryoBench, or our benchmarking effort, which was hopefully designed to both have simple diagnostic datasets that people can use for methods development that can tell them whether their method is working or not, from both qualitative and quantitative metrics... and challenging datasets that will motivate new methods development. And so we were very deliberate with the design of these datasets.
And so that’s been super cool to see where people are actually using these datasets and computing the metrics. Metrics is a huge problem for the field because... yeah, I think that continues to be an open problem for the field. And so, yeah, I think maybe the lack of metrics helps guard against blindly following a number. But yeah, I think it makes it harder for new people, like maybe who are more expert in other areas like graphics or vision, to get into the field. And so with all these things, I think there’s just nuance and it’s important to keep that in mind.
Abhi: Metrics... like what’s the story on Cryo-EM metrics? Is it just RMSD? Is there something else beyond that?
Ellen: The main metric is resolution.
Abhi: Okay.
Ellen: For a given volume, but that’s also estimated a bit heuristically. And a notion of global resolution for the structure doesn’t actually hold. And the actual computation of resolution can be hacked a bit. So that’s for sure a challenge in the field. You know, there’s enough people who have the expertise to judge the quality of a structure that you’ll get called out for it. Right. So that’s one of the main metrics is resolution, and computed via something called FSC, or Fourier Shell Correlation. For heterogeneity, for ensembles, it doesn’t exist yet.
Abhi: Okay.
Ellen: And we have some stand-ins that are FSC-based. Some people in our group are taking some of the shape-based metrics from computer vision, like Chamfer distance, just volumetric IOU, and things like that. But there’s a lot of just caveats and challenges. And so I think metrics for conformational states and measuring what is it that you care about is not solved.
Abhi: Naively, I think, Why isn’t the conformation problem just like, ‘Oh, there are these five conformations in the dataset. If your model doesn’t give me back all five or it gives me four out of five... why don’t you just report that accuracy?’ Like, what’s the nuance there?
Ellen: Yeah. I think even RMSD, right... so that doesn’t capture the specific conformational state. And I think, you know, people are using maybe local-type metrics or maybe that would be a direction to move forward. But that is very system-dependent. And so I think a general metric for conformational state just doesn’t exist right now.
Abhi: Yeah.
Ellen: Perhaps against a target dataset, you could define something like that.
Abhi: Was that done for CryoBench?
Ellen: We didn’t have dataset-specific metrics. That’s definitely something that I want to move towards. Like, especially one of the datasets is based on an MD simulation, so there’s 46,000 distinct structures in the dataset. And you’re not actually going to be able to recover 46,000 distinct structures. So how do you actually characterize the distribution? And you have distributional metrics, right, that is established. But, you know... what is the actual... yeah... what is the representation? Like there’s all these actual details in terms of the metrics and computing distances that can make a difference.
Abhi: I think that was the other last question I had.
Ellen: Yeah. Thanks for the conversation.
Abhi: Yeah, absolutely. I am struggling to think of anything else. Oh, one hour 47 minutes. Okay. That’s not bad.
Ellen: That’s pretty good.
Abhi: Yeah.
Ellen: Cool. Awesome. Okay, ship it.
