I explored memory models for spaced repetition in my master's thesis and later built an SRS product. This post shares my thoughts on content-aware memory models.
I believe this technical shift in how SRS models the student's memory won't just improve scheduling accuracy but, more critically, will unlock better product UX and new types of SRS.
I've been playing with something similar, but far less thought out than what you have.
I have a script for it, but am basically waiting until I can run a powerful enough LLM locally to chug through it with good results.
Basically like the knowledge tree you mention towards the end, but attempt to create a knowledge DAG by asking a LLM "does card (A) imply knowledge of card (B) or vice versa". Then, take that DAG and use it to schedule the cards in a breadth first ordering. So, when reviewing a new deck with a lot of new cards, I'll be sure to get questions like "what was the primary cause of the civil war", before I get questions like "who was the Confederate general who fought at bull run"
I've got a system for learning languages that does some of the things you mention. The goal is to be able to recommend content for a user to read which combines 1) appropriate level of difficulty 2) usefulness for learning. The idea is to have the SRS system build into the system, so you just sit and read what it gives you, and review of old words and learning new words (according to frequency) happens automatically.
Separating the recall model from the teaching model as you say opens up loads of possibilities.
Brief introduction:
1. Identify "language building blocks" for a language; this includes not just pure vocabulary, but the grammar concepts, inflected forms of words, and can even include graphemes and what-not.
2. For each building block, assign a value -- normally this is the frequency of the building block within the corpus.
3. Get a corpus of selections to study. Tag them with the language building blocks. This is similar to Math Academy's approach, but while they have hundreds of math concepts, I have tens of thousands of building blocks.
3. Use a model to estimate the current difficulty of each word. (I'm using "difficulty" here as the inverse of "retrievability", for reasons that will be clear later.)
4. Estimate the delta of difficulty of each building block after being viewed. Multiply this delta by the word value to get the study value of that word.
5. For each selection, calculate the total difficulty, average difficulty, and total study value. (This is why I use "difficulty" rather than "retrievability", so that I can calculate total cognitive load of a selection.)
Now the teaching algorithm has a lot of things it can do. It can calculate a selection score which balances study value, difficulty, as well as repetitiveness. It can take the word with the highest study value, and then look for words with that word in it. It can take a specific selection that you want to read or listen to, find the most important word in that selection, and then look for things to study which reinforce that word.
You mentioned computational complexity -- calculating all this from scratch certainly takes a lot, but the key thing is that each time you study something, only a handful of things change. This makes it possible to update things very efficiently using an incremental computation [1].
> [....] Ignoring the following factors means we are leaving useful information on the table:
> 1. The review histories of related cards. Card semantics allow us to identify related cards. This enables memory models to account for the review histories of all relevant cards when estimating a specific card’s retrievability.
> 2. [...]
I've been thinking that card semantics shouldn't be analyzed at all, and just treated as a black box. You can get so much data off of just a few users of a flashcard deck that you could build your own map of the relationships between cards, just by noticing the ones that get failed or pass together over time. Just package that map with the deck and the scheduler might get a lot smarter.
That map could give you good info on which cards were redundant, too.
edit: this may be interesting to someone, but I've also been trying to flesh out a model where agents buy questions from a market, trade questions with each other, and make bets with each other about whether the user will be able to recall the question when asked. Bankrupt agents are replaced by new agents. Every incentive in the system is parameterized by the user's learning requirements.
You mention that FSRS treats each card independently, even if they derive from the same note. I wonder whether you've tried this Anki plugin, which tries to increase the interval between reviews of 'sibling' cards: https://ankiweb.net/shared/info/759844606
I've been thinking about this for a while too as an FSRS developer [1].
In general, we can think of a spaced repetition system as being (i) Content-aware vs. Content-agnostic and (ii) Deck-aware vs. Deck-agnostic
Content-aware systems care about what you're studying (language, medecine, etc) while Content-agnostic systems don't care about what you're studying.
Deck-aware systems consider each card in the context of the rest of the cards (the "deck") while Deck-agnostic systems consider each card in pure isolation.
Currently, FSRS is both Content-agnostic as well as Deck-agnostic. This makes it extremely easy to integrate into a spaced repetition system, but this also means the model will underfit a bit.
It it interesting to note that you could in practice optimize seperate FSRS models for each deck covering different topics, which would make it Content-aware in a sense. Additionally, "fuzz" is a somewhat Deck-aware feature of the model in that it exists specifically to reduce interactions between other cards in the deck.
On the scheduling end, I'm surprised the article didn't mention https://github.com/fasiha/ebisu which uses Bayesian statistics.
When I was studying Japanese, I was thinking how it's always best to learn words in sentences and that it would be good if the sentences for a particular word were random.
Extending that, the sentences could be picked such that the other words are words scheduled for today meaning much more bang for buck per learning hour.
I'm building a SRS language learning app [1] so I've thought about this topic a bit, but I've come to a conclusion that srs algorithms might be just a nerd optimization obsession. My app has "stupid" 1,3,7,15,30 or something like that intervals, and the reality is that if I know a card, I can swipe it within 2 seconds, and if I just barely know it, I can spend 30 seconds on it.
So optimizing the algorithm such that every card comes at the exact right moment might cause all cards to feel too hard or too easy. I think having a mix of difficult and easy cards is actually a feature, not a bug.
In the language learning world there are some great tools already for adding content-awareness.
AnkiMorphs[1] will analyze the morphemes in your sentences and, taking into account the interval of each card as a sign of how well you know each one, will re-order your new cards to, ideally, present you with cards that have only one unknown word.
It doesn't do anything to affect the FSRS directly—it only changes the order of new, unlearned cards—but in my experience it's so effective at shrinking the time from new card to stable/mature that I'm not sure how much more it would help to have the FSRS intervals being adjusted in this particular domain.
Rather than relying on an embedding space, my approach is to have the cards themselves be grammars that can define the relationships between concepts explicitly. Then the problem becomes what specific sampling of all the possible outputs is optimal for a learner to see at any given time, given their knowledge state.
I’ve been working on https://phrasing.app for a while now, including many iterations of the SRS. It’s been my experience that most of these sorts of improvements are really imperceptible. While I use FSRS as a base, and I’m very happy with the results it provides, it’s really only a few percentage points off of the SM-2 algorithm from the 90s. It’s slightly less stressful, definitely more accurate, but I think only astute users would even notice the difference.
I’ve incorporated many different things into the SRS, from vector embeddings to graph based association to lemma clustering to morpheme linking, and was surprised how much of these I took out.
Most of the unlocks with the SRS have been more in application space. Doing reviews with Anki feels like a chore, and I’m always counting down the reviews left to do. Reviews with Phrasing however are much more addictive, and I routinely spent an extra 30+ minutes in that “ok just one more card” loop.
We will never be able to know with 100% certainty how well you know a card, but FSRS gets us darn close. I think the interesting stuff is less about improving that metric, and more about what can you do with that information.
Thanks to the whole FSRS team btw (I assume y’all will be reading this hn post) <3
| The main challenge in building content-aware memory models is lack of data. To my knowledge, no publicly available dataset exists that contains real-world usage data with both card textual content and review histories.
I wonder if the author has ever considered reaching out to makers of Anki decks used by premeds and medical students like the AnKing [1]. They create Anki decks for users studying the MCAT and various Med School curricula, so have a) relatively stable deck content (which is very well annotated and contains lots of key words that would make semantic grouping quite easy) b) probably contains loads of statistics on user reviews (since they have an Anki addon that sends telemetry to their team to make the decks better IIRC), and c) contains incredibly disparate information (all the way from high-school physics to neurochemistry).
To help with language learning I tried Anki, didn't like the UX and ended up writing my own SRS, from scratch.
One thing that becomes very obvious very quickly is that all cards derived from the same piece of information should be treated as a group. The last thing you'd want is to see "a cow / ?" quickly followed by "una mucca / ?". This is just pointless.
So while I appreciate the in-depth write-up by the author, I must say that its main insight - that the scheduling needs to account for the inter-card dependencies - lies right there on the surface. The fact that Anki doesn't support this doesn't make it any less obvious.
The one thing I would want from a content aware scheduler would be to not put similar together.
What ends up happening is I have two similar cards mixed up. For the first card I take a 50/50 and get it right. Then for the second card I get it correct by process of information instead of having to take another 50/50. This results in the system incorrectly thinking I knew the second card that came up.
17 comments
[ 4.4 ms ] story [ 51.6 ms ] threadI believe this technical shift in how SRS models the student's memory won't just improve scheduling accuracy but, more critically, will unlock better product UX and new types of SRS.
I have a script for it, but am basically waiting until I can run a powerful enough LLM locally to chug through it with good results.
Basically like the knowledge tree you mention towards the end, but attempt to create a knowledge DAG by asking a LLM "does card (A) imply knowledge of card (B) or vice versa". Then, take that DAG and use it to schedule the cards in a breadth first ordering. So, when reviewing a new deck with a lot of new cards, I'll be sure to get questions like "what was the primary cause of the civil war", before I get questions like "who was the Confederate general who fought at bull run"
I've got a system for learning languages that does some of the things you mention. The goal is to be able to recommend content for a user to read which combines 1) appropriate level of difficulty 2) usefulness for learning. The idea is to have the SRS system build into the system, so you just sit and read what it gives you, and review of old words and learning new words (according to frequency) happens automatically.
Separating the recall model from the teaching model as you say opens up loads of possibilities.
Brief introduction:
1. Identify "language building blocks" for a language; this includes not just pure vocabulary, but the grammar concepts, inflected forms of words, and can even include graphemes and what-not.
2. For each building block, assign a value -- normally this is the frequency of the building block within the corpus.
3. Get a corpus of selections to study. Tag them with the language building blocks. This is similar to Math Academy's approach, but while they have hundreds of math concepts, I have tens of thousands of building blocks.
3. Use a model to estimate the current difficulty of each word. (I'm using "difficulty" here as the inverse of "retrievability", for reasons that will be clear later.)
4. Estimate the delta of difficulty of each building block after being viewed. Multiply this delta by the word value to get the study value of that word.
5. For each selection, calculate the total difficulty, average difficulty, and total study value. (This is why I use "difficulty" rather than "retrievability", so that I can calculate total cognitive load of a selection.)
Now the teaching algorithm has a lot of things it can do. It can calculate a selection score which balances study value, difficulty, as well as repetitiveness. It can take the word with the highest study value, and then look for words with that word in it. It can take a specific selection that you want to read or listen to, find the most important word in that selection, and then look for things to study which reinforce that word.
You mentioned computational complexity -- calculating all this from scratch certainly takes a lot, but the key thing is that each time you study something, only a handful of things change. This makes it possible to update things very efficiently using an incremental computation [1].
But that does make the code quite complicated.
[1] https://en.wikipedia.org/wiki/Incremental_computing
> 1. The review histories of related cards. Card semantics allow us to identify related cards. This enables memory models to account for the review histories of all relevant cards when estimating a specific card’s retrievability.
> 2. [...]
I've been thinking that card semantics shouldn't be analyzed at all, and just treated as a black box. You can get so much data off of just a few users of a flashcard deck that you could build your own map of the relationships between cards, just by noticing the ones that get failed or pass together over time. Just package that map with the deck and the scheduler might get a lot smarter.
That map could give you good info on which cards were redundant, too.
edit: this may be interesting to someone, but I've also been trying to flesh out a model where agents buy questions from a market, trade questions with each other, and make bets with each other about whether the user will be able to recall the question when asked. Bankrupt agents are replaced by new agents. Every incentive in the system is parameterized by the user's learning requirements.
In general, we can think of a spaced repetition system as being (i) Content-aware vs. Content-agnostic and (ii) Deck-aware vs. Deck-agnostic
Content-aware systems care about what you're studying (language, medecine, etc) while Content-agnostic systems don't care about what you're studying.
Deck-aware systems consider each card in the context of the rest of the cards (the "deck") while Deck-agnostic systems consider each card in pure isolation.
Currently, FSRS is both Content-agnostic as well as Deck-agnostic. This makes it extremely easy to integrate into a spaced repetition system, but this also means the model will underfit a bit.
It it interesting to note that you could in practice optimize seperate FSRS models for each deck covering different topics, which would make it Content-aware in a sense. Additionally, "fuzz" is a somewhat Deck-aware feature of the model in that it exists specifically to reduce interactions between other cards in the deck.
[1] https://github.com/open-spaced-repetition/py-fsrs
I love Anki and used it before when I needed to memorize things, but would love to know what other options on the market exist.
When I was studying Japanese, I was thinking how it's always best to learn words in sentences and that it would be good if the sentences for a particular word were random.
Extending that, the sentences could be picked such that the other words are words scheduled for today meaning much more bang for buck per learning hour.
So optimizing the algorithm such that every card comes at the exact right moment might cause all cards to feel too hard or too easy. I think having a mix of difficult and easy cards is actually a feature, not a bug.
[1] https://vocabuo.com
AnkiMorphs[1] will analyze the morphemes in your sentences and, taking into account the interval of each card as a sign of how well you know each one, will re-order your new cards to, ideally, present you with cards that have only one unknown word.
It doesn't do anything to affect the FSRS directly—it only changes the order of new, unlearned cards—but in my experience it's so effective at shrinking the time from new card to stable/mature that I'm not sure how much more it would help to have the FSRS intervals being adjusted in this particular domain.
1: https://mortii.github.io/anki-morphs/intro.html
See how it's applied to Japanese learning here: https://elldev.com/feed/grsly
I’ve incorporated many different things into the SRS, from vector embeddings to graph based association to lemma clustering to morpheme linking, and was surprised how much of these I took out.
Most of the unlocks with the SRS have been more in application space. Doing reviews with Anki feels like a chore, and I’m always counting down the reviews left to do. Reviews with Phrasing however are much more addictive, and I routinely spent an extra 30+ minutes in that “ok just one more card” loop.
We will never be able to know with 100% certainty how well you know a card, but FSRS gets us darn close. I think the interesting stuff is less about improving that metric, and more about what can you do with that information.
Thanks to the whole FSRS team btw (I assume y’all will be reading this hn post) <3
And if anyone is curious I wrote up a bit about my SRS here: https://phrasing.app/blog/humane-srs
I wonder if the author has ever considered reaching out to makers of Anki decks used by premeds and medical students like the AnKing [1]. They create Anki decks for users studying the MCAT and various Med School curricula, so have a) relatively stable deck content (which is very well annotated and contains lots of key words that would make semantic grouping quite easy) b) probably contains loads of statistics on user reviews (since they have an Anki addon that sends telemetry to their team to make the decks better IIRC), and c) contains incredibly disparate information (all the way from high-school physics to neurochemistry).
---
[1]: https://www.theanking.com
One thing that becomes very obvious very quickly is that all cards derived from the same piece of information should be treated as a group. The last thing you'd want is to see "a cow / ?" quickly followed by "una mucca / ?". This is just pointless.
So while I appreciate the in-depth write-up by the author, I must say that its main insight - that the scheduling needs to account for the inter-card dependencies - lies right there on the surface. The fact that Anki doesn't support this doesn't make it any less obvious.
What ends up happening is I have two similar cards mixed up. For the first card I take a 50/50 and get it right. Then for the second card I get it correct by process of information instead of having to take another 50/50. This results in the system incorrectly thinking I knew the second card that came up.