This guide is also made from me (or some of the me from a couple years back).
I haven't read the whole thing yet and it's probably clearly stated at some point (though one can deduce it with the beginning already) but the surprise for me was that this field is highly statistical. Before starting I had the (very) naive view that it was possible to read the genome as one reads a file and look at what's going on. But the sequencing technics (and accompanying algorithms) only allow to statistically read the genome. So variants/mutations found are only found with a given statistical certainty. If the sample wasn't well prepared for example it could be that this certainty is ultimately not high enough to do a proper analysis/diagnostic.
It's a fascinating field (try to watch a video on sequencing by expansion, to feel how sci-fi this field actually is) that is very hard to approach with only high-school biology level and this guide is really well done to sort of bridge this first gap.
I'm working on a project in malaria genetics this summer, and I was shocked to find out that the entire analysis toolkit is entirely based on math and statistics (and some non-trivial stuff too, e.g. hidden Markov models to predict CNV). Genotype likelihoods throw an extra wrench into the process, since even basic stuff like predicting allele frequencies requires a maximum likelihood estimator instead of simple counting. This whole area was quite eye-opening, and I'm still amazed that reading billions of base-pairs in DNA sequencing reliably works.
All one needs to do is look at the Claude Science thread here last week and note how many comments were surprised that it appeared to be a statistical/analysis tool.
Just a couple days ago I argued with an HN poster who quipped that biology is stamp collecting. A non-negligible number of "mathy" engineer types (not actual mathematicians, those usually understand the complexity of biology and even gladly contribute to the field) seem to think all biologists are quirky eccentrics dedicating 30 years to a single protein or a species of ants in the Kalahari desert. (Not that these don't exist or aren't worthy of respect, but they don't score high in the sophomoric 'hardness scale' of fields that these mathy types still subscribe to)
The rub is of course it is actually far harder to work on your rare species of ant than to do something more "noble" like human genetics. You may have to build the reference genome yourself working on your ant before you can begin with other work. Collecting your ant samples and processing them eventually into raw sequence reads. You may have to optimize this library preparation process yourself if you are really in uncharted territory here.
Meanwhile human geneticist doesn't even need to collect any data. Reference genomes are always being improved. People dump their data into public repositories (at least public for other credentialed researchers). A couple emails and filling of approval forms and you too can have access to 10,000 patient samples of some human disease already sequenced for you to an acceptable depth. Of course you will still need to pay for downstream compute needs in money and your time crafting the analysis to suit your reasoning, but still, half the battle is already won when you work on these well trodden paths. So much necessary groundwork has been performed by others for you already.
And the foundations of those statistical approaches are built on heuristics and shortcuts.
For example, sequencing instruments include base quality strings in the output. Base qualities are estimates how likely the instrument got each sequenced base right. But most people don't want to store that much noise, especially when the actual data is highly compressible. So the base qualities get quantized using more or less principled methods that seem to work well empirically.
Read aligners make similar estimates of how likely they got the correct alignment for each read. Those estimates are typically based on simplistic models and a number of assumptions. There are two main components in the estimate. One is based on comparing the primary alignment the aligner chose to the secondary alignments it also found. Another is an estimate that the aligner didn't find the correct alignment, because that part of the sequenced genome is too different from the reference. The latter is obviously handwavy. And the aligner cheats in the former. Because people don't want to wait 10x or 100x longer for better results, the aligner gives up early and estimates how good secondary alignments it might have found if it had actually done the work.
And then there is variant calling. At some point, the state-of-the-art callers were statistical. But then people got better results with neural networks. Or at least the results were empirically better.
Biology is often an intensely statistics-heavy field. A remarkably large part of statistics was developed to study issues in biology, particularly dealing with evolution and ecology.
To expand this a bit, most sequencing methods are exact, and have a low error rate (except nanopores).
But they produce short reads, and because DNA is full of repetitive fragments, it's not always clear where the read came from.
We also have two copies of genes, which also further complicates matters.
The first startup where I worked, developed synthetic long reads on top of Illumina's hardware. We could stitch together 50kbp reads, which really helped with de-novo sequencing.
One part that people from the software side tend to underestimate is how fuzzy and analog everything in biology is. Genomics look more predictable and organized at first, but even these parts are quite fuzzy and subject to all kinds of physical effects.
I'd strongly recommend in reading up on the parts of cell biology that come after this. Otherwise you'll get the wrong impression of how messy biology actually is.
And most of genomics is still stuck in 1930ties. Many people believe gender is somehow related to genes, which is objectively not true! Or that genes are somehow related to your religion!
Scientists tend to understand that part, that's more of a political/cultural thing (I'm ignoring the language part here entirely about which terms to use for which concepts here).
There's the X and Y chromosomes, those produce a binary result (unless you have a genetic anomaly). And after that comes the messy and fuzzy parts I mentioned, where those genes trigger changes in hormone levels and development. And those parts are analog, very complex and contain a lot of different parts. So the outcome is not binary anymore.
There are more combinations than merely having only X, or an XY combination. And there is more fuzziness even in the Y and X expression, as you said. It's fuzzy all the way down. The tale of Binary results has always been from compression of reality: Always has been.
As it happens, in humans there is a single gene on the Y chromosome, named SRY, that typically switches on male-linked traits.
But you're right, the full range of biological possibilities is very fuzzy . SRY itself a just a regulatory switch that other sex-linked traits are conditionally dependent on. If the switch gets broken, you develop as female. If genes that support the switch break, you might develop as female. If a sex-linked trait downstream from SRY mutates, then pretty much anything can happen. And other species do sex determination completely differently. Hell, a lot of bacterial sex basically involves throwing pseudo-viruses at each other.
If you're an engineer and want to go deeper into the core algorithms behind genomics, there's a book / course called Bioinformatics Algorithms. It was a punishing read when I was going through it a few years ago (but rewarding). It's probably much better now given the state of AI.
I've worked for a year in a lab doing cancer genomics and had to learn everything from scratch, since my background is in computer science.
It's definitely possible to learn enough to be productive within a few months, but to actually comprehend and understand the underlying biology takes much, much longer. I still don't understand much of what is presented by people from other labs outside of my specialty.
This is very very nice. when you are reading this, just keep this in the back of your mind - inside a cell- things are floating around constantly at a very high speed. those things do not have any crisp shape or boundary. so how do we tell them apart? they are phase separated. if you put an oil drop in water, you can still see the oil drop and water and tell them apart. that's a very high degree of phase separation. inside a cell the degree of phase separation is much lower. just putting this out here so that you could appreciate the complexity of the biology that you are reading. my wife educated me on this a bit.
Maybe a section on RNA degredation and DNA stability and how it would affect sequencing would be nice.
Also, down stream analyses are largely missing e.g. differential analysis, pathway enrichment. Not to mention newer single cell techniques and their up/down sides. But good start!
One area that might be worth expanding in future sections is how these concepts scale when moving from single genes to whole-genome analysis and polygenic traits.
Its like this was made for me haha ! I've been reading books about epigenomica to get an understanding. This is cool, will definitely spend my weekend going through it
Waiting for an enterprising hacker to develop mosquito gene drive in their garage. You could probably develop a thriving recurring income stream if you develop something that works
> In plants and animals, DNA is broken up into a number of large sequences called chromosomes that are tucked into the nucleus.
This is a weird description, because ... it is not really "broken up". Each chromosome could be shuffled and put into different cells in different numbers. Now, it is unlikely that the resulting cell would be viable or useful, but my contention here is the "broken up" part. Chromosomes are just a way to handle the genome set. There are reasons why bacteria do not have chromosomes and this has mostly to do with the amount of DNA. To call this "breaking up" is a very strange description. (Size is not the only reason; duplication of the DNA before cell division is another important factor; bacteria usually have just one origin of replication, eukaryotes have several on each chromosome, otherwise the S-phase in the cell cycle would simply take too long.)
> Each genome is a biochemical database that, if properly accessed, can inform how our bodies function.
This is also a very strange description, aka "biochemical database". Not everything in a genome has a role with regards to biochemistry or metabolism. Some is just regulatory RNA; some of this relates to metabolism, but you also have e. g. piwiRNA or silencers of transposons and so forth. That in itself has only very rarely a biochemical function, with some exceptions (e. g. I would classify tRNA as related to metabolism, and many viruses have tRNA or use tRNA as quick-starters, but most of those regulatory RNAs do not have any function for metabolism directly, other than e. g. repurposing energy towards their own reproduction).
To me it seems as if the article was written by an engineer. That's fine, but it also means that the thinking is quite biased. Genetics is not quite so easy to engineer; a good example are leaky promoters used in synthetic biology (just ask the people who use such promoters how to make them un-leaky) or off-target cleavage effects in CRISPR-Cas(9 or whatever is used); I am pretty certain they'll give excuses as to why 100% accurate gene therapy isn't yet ready for the masses. And they'll do that for quite some years to come, I bet, usually hiding behind "it will cost too much" - when in reality, it should cost very little, if it were to work, rather than this just becoming the new meta-milking scheme.
"Broken up" may not be the best choice of words, but you did not explain which synonym would make you happier: "divided", "segmented", "fragmented", "split", etc.
The DNA of a nucleated (a.k.a. eukaryote) cell is indeed split into multiple chromosomes and the number of the chromosomes and the number of genes on each chromosomes and the sequence of the genes on each chromosome are normally constant for a species and very similar for closely related species. The cells that have an incorrect number of chromosomes (which happens when a cell division does not work correctly) will normally die soon, because their DNA is incomplete.
Only when a cell has all the normal chromosomes, but also some extra chromosomes, it has a chance to survive, even if the extra chromosomes may interfere with some internal processes. Because superfluous chromosomes are much less harmful than missing chromosomes, there have been cases, rare at animals, but frequent at plants, when the entire genome has been doubled, when some cell division has failed, but the descendants of that cell have survived.
There are animals for which the number of the chromosomes and the sequence of the genes on them has remained unchanged for many hundreds of millions of years, though there are also animals where the DNA has been completely rearranged, because some chromosomes have fused into a single chromosome, other chromosomes have split into multiple chromosomes, and on some chromosomes the genes have been shuffled.
Nowadays, it is known that it is very likely that the common ancestor of all animals except comb jellies had 29 chromosome pairs (humans have 23 pairs). In most animal branches there were more chromosome fusions than chromosome splittings, so in the present most animals have fewer than 29 chromosome pairs.
Among the animals with the most conservative genomes are some sponges, some jellyfish, many echinoderms and some of their relatives, i.e. acorn worms, the lancelets, some nemertean worms and some bivalves.
> This Guide is written specifically by and for computer scientists and engineers. The underlying biology in cancer genomics can be exceedingly complex and requires years of study.
This looks like a great guide to read.
But I think before diving deeper and reading the rest of the guide, which granted it is from employees working in a lab inside of a hospital, I'd like to get the expert opinion of a geneticist or an expert biologist with years of experience in genomics to iron out any issues in the guide or give an additional proof-reading review.
This guide is also made from me (or some of the me from a couple years back). I haven't read the whole thing yet and it's probably clearly stated at some point (though one can deduce it with the beginning already) but the surprise for me was that this field is highly statistical. Before starting I had the (very) naive view that it was possible to read the genome as one reads a file and look at what's going on. But the sequencing technics (and accompanying algorithms) only allow to statistically read the genome. So variants/mutations found are only found with a given statistical certainty. If the sample wasn't well prepared for example it could be that this certainty is ultimately not high enough to do a proper analysis/diagnostic. It's a fascinating field (try to watch a video on sequencing by expansion, to feel how sci-fi this field actually is) that is very hard to approach with only high-school biology level and this guide is really well done to sort of bridge this first gap.
I'm working on a project in malaria genetics this summer, and I was shocked to find out that the entire analysis toolkit is entirely based on math and statistics (and some non-trivial stuff too, e.g. hidden Markov models to predict CNV). Genotype likelihoods throw an extra wrench into the process, since even basic stuff like predicting allele frequencies requires a maximum likelihood estimator instead of simple counting. This whole area was quite eye-opening, and I'm still amazed that reading billions of base-pairs in DNA sequencing reliably works.
Also gotta shout out to these incredible molecular animations by WEHI: https://www.youtube.com/watch?v=7Hk9jct2ozY
All one needs to do is look at the Claude Science thread here last week and note how many comments were surprised that it appeared to be a statistical/analysis tool.
Just a couple days ago I argued with an HN poster who quipped that biology is stamp collecting. A non-negligible number of "mathy" engineer types (not actual mathematicians, those usually understand the complexity of biology and even gladly contribute to the field) seem to think all biologists are quirky eccentrics dedicating 30 years to a single protein or a species of ants in the Kalahari desert. (Not that these don't exist or aren't worthy of respect, but they don't score high in the sophomoric 'hardness scale' of fields that these mathy types still subscribe to)
The rub is of course it is actually far harder to work on your rare species of ant than to do something more "noble" like human genetics. You may have to build the reference genome yourself working on your ant before you can begin with other work. Collecting your ant samples and processing them eventually into raw sequence reads. You may have to optimize this library preparation process yourself if you are really in uncharted territory here.
Meanwhile human geneticist doesn't even need to collect any data. Reference genomes are always being improved. People dump their data into public repositories (at least public for other credentialed researchers). A couple emails and filling of approval forms and you too can have access to 10,000 patient samples of some human disease already sequenced for you to an acceptable depth. Of course you will still need to pay for downstream compute needs in money and your time crafting the analysis to suit your reasoning, but still, half the battle is already won when you work on these well trodden paths. So much necessary groundwork has been performed by others for you already.
And the foundations of those statistical approaches are built on heuristics and shortcuts.
For example, sequencing instruments include base quality strings in the output. Base qualities are estimates how likely the instrument got each sequenced base right. But most people don't want to store that much noise, especially when the actual data is highly compressible. So the base qualities get quantized using more or less principled methods that seem to work well empirically.
Read aligners make similar estimates of how likely they got the correct alignment for each read. Those estimates are typically based on simplistic models and a number of assumptions. There are two main components in the estimate. One is based on comparing the primary alignment the aligner chose to the secondary alignments it also found. Another is an estimate that the aligner didn't find the correct alignment, because that part of the sequenced genome is too different from the reference. The latter is obviously handwavy. And the aligner cheats in the former. Because people don't want to wait 10x or 100x longer for better results, the aligner gives up early and estimates how good secondary alignments it might have found if it had actually done the work.
And then there is variant calling. At some point, the state-of-the-art callers were statistical. But then people got better results with neural networks. Or at least the results were empirically better.
Biology is often an intensely statistics-heavy field. A remarkably large part of statistics was developed to study issues in biology, particularly dealing with evolution and ecology.
To expand this a bit, most sequencing methods are exact, and have a low error rate (except nanopores).
But they produce short reads, and because DNA is full of repetitive fragments, it's not always clear where the read came from.
We also have two copies of genes, which also further complicates matters.
The first startup where I worked, developed synthetic long reads on top of Illumina's hardware. We could stitch together 50kbp reads, which really helped with de-novo sequencing.
One part that people from the software side tend to underestimate is how fuzzy and analog everything in biology is. Genomics look more predictable and organized at first, but even these parts are quite fuzzy and subject to all kinds of physical effects.
I'd strongly recommend in reading up on the parts of cell biology that come after this. Otherwise you'll get the wrong impression of how messy biology actually is.
And most of genomics is still stuck in 1930ties. Many people believe gender is somehow related to genes, which is objectively not true! Or that genes are somehow related to your religion!
Scientists tend to understand that part, that's more of a political/cultural thing (I'm ignoring the language part here entirely about which terms to use for which concepts here).
There's the X and Y chromosomes, those produce a binary result (unless you have a genetic anomaly). And after that comes the messy and fuzzy parts I mentioned, where those genes trigger changes in hormone levels and development. And those parts are analog, very complex and contain a lot of different parts. So the outcome is not binary anymore.
...well actually...
There are more combinations than merely having only X, or an XY combination. And there is more fuzziness even in the Y and X expression, as you said. It's fuzzy all the way down. The tale of Binary results has always been from compression of reality: Always has been.
As it happens, in humans there is a single gene on the Y chromosome, named SRY, that typically switches on male-linked traits.
But you're right, the full range of biological possibilities is very fuzzy . SRY itself a just a regulatory switch that other sex-linked traits are conditionally dependent on. If the switch gets broken, you develop as female. If genes that support the switch break, you might develop as female. If a sex-linked trait downstream from SRY mutates, then pretty much anything can happen. And other species do sex determination completely differently. Hell, a lot of bacterial sex basically involves throwing pseudo-viruses at each other.
Can you point to where these ideas have been confirmed as objectively not true?
If you're an engineer and want to go deeper into the core algorithms behind genomics, there's a book / course called Bioinformatics Algorithms. It was a punishing read when I was going through it a few years ago (but rewarding). It's probably much better now given the state of AI.
[1] https://cogniterra.org/course/64/info
I've worked for a year in a lab doing cancer genomics and had to learn everything from scratch, since my background is in computer science.
It's definitely possible to learn enough to be productive within a few months, but to actually comprehend and understand the underlying biology takes much, much longer. I still don't understand much of what is presented by people from other labs outside of my specialty.
This is very very nice. when you are reading this, just keep this in the back of your mind - inside a cell- things are floating around constantly at a very high speed. those things do not have any crisp shape or boundary. so how do we tell them apart? they are phase separated. if you put an oil drop in water, you can still see the oil drop and water and tell them apart. that's a very high degree of phase separation. inside a cell the degree of phase separation is much lower. just putting this out here so that you could appreciate the complexity of the biology that you are reading. my wife educated me on this a bit.
Super!
Maybe a section on RNA degredation and DNA stability and how it would affect sequencing would be nice.
Also, down stream analyses are largely missing e.g. differential analysis, pathway enrichment. Not to mention newer single cell techniques and their up/down sides. But good start!
Love the guide, out of curiosity, what is your background and what inspired you to create this?
One area that might be worth expanding in future sections is how these concepts scale when moving from single genes to whole-genome analysis and polygenic traits.
Its like this was made for me haha ! I've been reading books about epigenomica to get an understanding. This is cool, will definitely spend my weekend going through it
This is super cool, thanks
Waiting for an enterprising hacker to develop mosquito gene drive in their garage. You could probably develop a thriving recurring income stream if you develop something that works
> In plants and animals, DNA is broken up into a number of large sequences called chromosomes that are tucked into the nucleus.
This is a weird description, because ... it is not really "broken up". Each chromosome could be shuffled and put into different cells in different numbers. Now, it is unlikely that the resulting cell would be viable or useful, but my contention here is the "broken up" part. Chromosomes are just a way to handle the genome set. There are reasons why bacteria do not have chromosomes and this has mostly to do with the amount of DNA. To call this "breaking up" is a very strange description. (Size is not the only reason; duplication of the DNA before cell division is another important factor; bacteria usually have just one origin of replication, eukaryotes have several on each chromosome, otherwise the S-phase in the cell cycle would simply take too long.)
> Each genome is a biochemical database that, if properly accessed, can inform how our bodies function.
This is also a very strange description, aka "biochemical database". Not everything in a genome has a role with regards to biochemistry or metabolism. Some is just regulatory RNA; some of this relates to metabolism, but you also have e. g. piwiRNA or silencers of transposons and so forth. That in itself has only very rarely a biochemical function, with some exceptions (e. g. I would classify tRNA as related to metabolism, and many viruses have tRNA or use tRNA as quick-starters, but most of those regulatory RNAs do not have any function for metabolism directly, other than e. g. repurposing energy towards their own reproduction).
To me it seems as if the article was written by an engineer. That's fine, but it also means that the thinking is quite biased. Genetics is not quite so easy to engineer; a good example are leaky promoters used in synthetic biology (just ask the people who use such promoters how to make them un-leaky) or off-target cleavage effects in CRISPR-Cas(9 or whatever is used); I am pretty certain they'll give excuses as to why 100% accurate gene therapy isn't yet ready for the masses. And they'll do that for quite some years to come, I bet, usually hiding behind "it will cost too much" - when in reality, it should cost very little, if it were to work, rather than this just becoming the new meta-milking scheme.
"Broken up" may not be the best choice of words, but you did not explain which synonym would make you happier: "divided", "segmented", "fragmented", "split", etc.
The DNA of a nucleated (a.k.a. eukaryote) cell is indeed split into multiple chromosomes and the number of the chromosomes and the number of genes on each chromosomes and the sequence of the genes on each chromosome are normally constant for a species and very similar for closely related species. The cells that have an incorrect number of chromosomes (which happens when a cell division does not work correctly) will normally die soon, because their DNA is incomplete.
Only when a cell has all the normal chromosomes, but also some extra chromosomes, it has a chance to survive, even if the extra chromosomes may interfere with some internal processes. Because superfluous chromosomes are much less harmful than missing chromosomes, there have been cases, rare at animals, but frequent at plants, when the entire genome has been doubled, when some cell division has failed, but the descendants of that cell have survived.
There are animals for which the number of the chromosomes and the sequence of the genes on them has remained unchanged for many hundreds of millions of years, though there are also animals where the DNA has been completely rearranged, because some chromosomes have fused into a single chromosome, other chromosomes have split into multiple chromosomes, and on some chromosomes the genes have been shuffled.
Nowadays, it is known that it is very likely that the common ancestor of all animals except comb jellies had 29 chromosome pairs (humans have 23 pairs). In most animal branches there were more chromosome fusions than chromosome splittings, so in the present most animals have fewer than 29 chromosome pairs.
Among the animals with the most conservative genomes are some sponges, some jellyfish, many echinoderms and some of their relatives, i.e. acorn worms, the lancelets, some nemertean worms and some bivalves.
> This Guide is written specifically by and for computer scientists and engineers. The underlying biology in cancer genomics can be exceedingly complex and requires years of study.
This looks like a great guide to read.
But I think before diving deeper and reading the rest of the guide, which granted it is from employees working in a lab inside of a hospital, I'd like to get the expert opinion of a geneticist or an expert biologist with years of experience in genomics to iron out any issues in the guide or give an additional proof-reading review.