To me, “memorization” implies an active process focused on learning a particular set of “matches” (to adopt the term you’re using here). But it seems to me that tacit knowledge (and other products of less concentrated/deliberate learning) often plays a substantial role in creativity.
That is, creativity fundamentally comes from internalized knowledge (as the article says) but internalized knowledge doesn’t necessarily come from memorization.
I think I see -- in your view "memorization" only refers to conscious learning
While internalized knowledge comes from "subconscious" (for lack of a better word) learning?
I guess I'm equating the two here and just using memorization as "committing to memory", with the belief being that you can construct the heuristic you'd normally acquire subconsciously and cut down time to mastery
I think memorization can play a role in internalizing knowledge, but it isn’t a “fundamental” as in necessary. Internalized knowledge can also come from other sources.
> But how do you know how to apply this acquired knowledge in this new context?
That question has a false precondition baked in. If you know how, it's not creative.
> It's some form of pattern matching right -- which imo is just a less obvious form of memorization
No. Sensing and matching patterns does not imply memorization. Everything you're saying is completely loaded.
Did you just discover memorization? Because the pattern I see in your words is similar to anyone who's just learned a new tool or technique - they overapply it everywhere as they learn to use it.
But you must have knowledge of the basic units of your chosen art to apply that to the new situation right?
E.g. if you're an artist, at the very least you need the knowledge of how to draw a line
From other comments here it seems the definition of "memorization" seems to be where disagreements are
Maybe this is a better explanation: once I started trying to make whatI just learned is called "tacit knowledge" more explicit and then committing it to memory, I was able to cut learning times down significantly
If you are transforming the knowledge, i dont think that is the same as memorizing it. In fact, i think most people would describe that as the opposite of memorization.
Using your definition, everything would be memorization. For example, you could describe picasso as simply pattern matching on other paintings to make something new, but i think it would be crazy to describe his work as an act of memorization (then again, the way AI generated art works... maybe its not so crazy)
This seems to resonate with my experience, although I feel myself bristling due to the baggage of the word memorization.
Although sometimes “memorization” doesn’t happen because you sit down to do it but rather that you keep using the same things over and over when solving problems that they become internalized. I find that to be a more fruitful path towards understanding that I don’t want to call memorization but it is.
And agreed -- it's this exact realization that led me to both this method and title
Imo this negative connotation has made many people refrain from calling internalization what it is
But acknowledging that it's memorization has actually made me more efficient at learning, since I can now consciously look for the heuristic, codify it, and try to commit it to memory
Maybe you should first try to separate the negative feelings you have towards the word "memorization" and the word itself "memorization". There's nothing bad about memorization. This sort of negative bias about inconsequential things is something that can easily hold us back from things that could help us further ourselves.
you guys might be interested in the latest Machine Learning Street Talk podcast [1] which right from the start is all about how LLMs are great for creativity, as in novel combinations of trained data (from memorization) -- but are not capable of the reasoning skill needed to verify if any idea is actually plausible given a set of constraints.
But LLMs to date can't really differentiate well between a creative insightful answer, and a nonsensical one. The selection process is still done by a human.
I have to say, you were naive to ever think that. Rolling a die can produce creativity — and I’m not being facetious.
Creativity, at least in fields traditionally considered to be ‘creative’ like music and visual art, is 90% randomness and 10% retroactively attempting to ascribe meaning to the randomness. And this is coming from someone who hugely values and defines oneself via an ability to be creative.
Also, LLMs haven’t hit ‘escape velocity’; that terminology has spread like a meme (because it’s an attractively dramatic yet easy to understand idea) but isn’t backed up by any science. Maybe one day we’ll have the fabled ‘AGI’, but ChatGPT isn’t it.
I think memorization gets a bad rep because you need to be acutely aware of what you're memorizing, like memorizing the sequence of an answer sheet instead of core concepts. But when done sufficiently rigorously, the foundations of memorization make room for higher-level critical thinking and reasoning.
Practice is an oft suggested solution to developing mastery, but I did like how the article framed it: creating subconscious heuristics and memory.
Absolutely, especially in real world application. If you don't have the ability to pull on fundamental ideas anywhere, anytime, then have you really mastered the learning material?
Right. I noticed this acutely in an abstract algebra course. We learned several different proof methods, then the exam was just "prove these theorems with the tools you have". I'd never been challenged with math like that before. I mean, I bombed it lol, but nobody was going to pass if they didn't remember, say, how to do a proof by induction or what it means. At some point, you need to be able to recall this information. Maybe the psychologists categorize these things differently, but I'd argue it's clear that some form of memorization is necessary for the task.
"a flash of inspiration connecting internalized concepts"
Well, okay, but rote memorization is neither necessary nor sufficient to internalize concepts.
One of the reasons people make fun of the author's approach to creativity is that systematic memorization fundamentally can't teach taste—so the systematic approach reeks of awkward, try-hard, low-brow, tasteless art.
More broadly, memorization doesn't help much with any sort of tacit knowledge, not just taste. I just figure taste is especially important in creative endeavors. That's definitely the case for programming! Memorization in programming gives us architecture astronauts and design-pattern soup rather than elegant code.
For what it's worth, I do think that it is useful and important to have a good mental model of what expertise is and how you can develop it. Memorization might be a component of this, but it's going to be a small component at most. I expect that realistic practice with fast feedback and expert mentorship matters far more. (If you're curious, I found the book Sources of Power by Gary Klein gave me a good way to think about how expertise works.)
At the same time, memorization has a real cost: it takes time and it's frightfully dull. For me, at least, trying to memorize something without context is not just ineffective but also totally kills any intrinsic motivation I have for whatever I'm learning. Sometimes a bit of memorization is unavoidable, but I've found that to be relatively rare. Otherwise, my time is generally better spent on some sort of practice in context.
One of the points I'm trying to make is that taste and elegance fundamentally stem from an internalized heuristic -- which at it's core is memorization.
I understand the connotation of "memorization" evokes an image of blindly memorizing without connecting, but isn't the tastefully developed expertise just memorization of a better heuristic?
not the parent poster but I think I agree with your perspective here. The alternative is that some individuals' taste or sense of aesthetics is somehow innate and unmoored from the statistics of the things they experience. There may be something to this, but for most practical purposes I would agree with your point.
Another alternative is that taste is something you can only learn through experience and mentorship, where memorizing simple rules and heuristics is not sufficient. Taste is an example of tacit knowledge[1].
There’s another argument though that some taste is genetically programmed, like our affinity for campfires or sweeping views. Those don’t seem to be learned as they seem to be entirely cross cultural and innate. Those aren’t examples of art of course but make the point that some sense of aesthetics may not be learned.
That's true if you broaden the definition of "memorization" to cover all learning, but "learning is necessary for creativity" would not be a particularly interesting thesis.
Expertise is the result of learning from past experience, both in developing an internal intuition for what you're doing and in having past patterns to draw upon. To the extent that experts have simple easily verbalizable heuristics, these are largely post-hoc attempts at explaining their intuition rather than an accurate reflection of how they make decisions.
And, in fact, experts can't even always do that: it is perfectly possible for experts to make good decisions without being consciously aware of why they are making them, and explaining how to make good decisions is a separate skill from being able to make them in the first place. The book I mentioned has a memorable story about a firefighter who thought he had precognition after pulling his team out of a dangerous situation without any specific indicator of the danger, but I figure a more common example is experts saying they did something because it was the "obvious" or "clean" or "better" way to do it and getting a bit flustered when pushed further.
We can see this in action pretty clearly if we look at advice for, say, writing. There is a lot of advice from good writers but just memorizing and blindly following this advice is actively counterproductive. Advice you can memorize fundamentally must lack nuance and context. We can see this clearly because so many different pieces of writing advice contradict each other and because good writers do not follow any of those suggestions with any consistency.
The same definitely applies to programming, which is why we have both "don't repeat yourself" and "you ain't going to need it", and why new programmers trying to apply either rule (or both!) to a codebase inevitably create a mess. What I've found with programming advice is that most suggestions are either actively wrong or too vague to be useful. (By the time you've learned enough about programming to be able to follow the vague advice, you don't need it very much!)
This happened about 20 years ago when they were trying to automate recognizing cancer cells. They showed photos to experienced diagnosticians and asked 'What features do you look for?' They couldn't articulate what they were seeing.
I don't think I can agree, as an extremely creative person with extremely bad memory - to a point where I pretty much never memorize anything, whether intentionally or by accident.
What I find instead, is that by just processing novel information, especially if I focus on analysing it, my brain internalizes insights and builds model of that type of thing, allowing me to either imperfectly reconstruct what I've seen, or to come up with an infinite array of permutations, extrapolations, etc which is where the real ideas come from.
Further, ideas crucially revolve not around just the information itself, but the "feel" for what role they play in the whole, how well they do it, in what way they're notable, etc.
In fact I'd straight up claim that memorization is antithetical to creativity - a perfect ML autoencoder or GAN would just regurgitate the training data. Creativity comes from generalisation while memorisation is analogous to overfitting.
A million times this.. I also am extremely creative and in fact I think the MOST creative people are really bad at intentional memoration, but are good at seeing patterns.
I feel like often the reason a creative person is hyper creative is they haven't memorized things so they are trying to rebuild information all the time in their heads from very sparse details.
This creates the transformative and relational combinations of information that a person memorizing can't see because it is created from a lack of organized specific information rather than a bounty of it.
"What I find instead, is that by just processing novel information, especially if I focus on analysing it, my brain internalizes insights and builds model of that type of thing"
I feel like every reply making a point against memorisation would benefit from having their definition of what is memorization, because every single one of those replies sound like they're still implicitly describing some sort of memorization as the better way
I feel like this is about the difference between rote/explicit memorization and organic/implicit/tacit memorization, for a lack of better words. I suspect the former could narrow/restrict your understanding because it may be constrained/limited by the vocabulary/definition itself.
I think perhaps there's a confusion of "memorization" with "rote memorization". The word "rote" connotes flashcards and dull drills, but memorization by itself, to me at least, is more like "a focused attempt at internalizing information", in whatever way that means to a person, as opposed to just ingesting it or letting it wash over you/osmosis.
But that's just my interpretation of the terms. I don't know what the "official" meanings are.
What do you call it when you remember things so you can repeat them but you can't generalize? E.g. if you learn a poem or phrase in a foreign language, but can't reuse the words in different contexts? Or being able to recite a rule, but not automatically applying it?
Is there a word for this?
Similarly, we should have a word for knowing how to reuse something in a different context, but not recall its origin or its canonical portrayal. Being able to apply a rule, without being able to recite it.
Do you think there's one word which means both of these things, which are opposites, as I've stated them?
> What do you call it when you remember things so you can repeat them but you can't generalize? E.g. if you learn a poem or phrase in a foreign language, but can't reuse the words in different contexts? Or being able to recite a rule, but not automatically applying it?
And "rote memorization" is a compound term that means what you were asking for. It's one of those things you can't get the exact meaning of by just looking at the components.
Transferring skills from one context to another is surprisingly hard to do, but not impossible. AFAIK, contexts must be similar to each other for transfer to take place.
I think I would call it internalization instead of memorization. People memorize equations not knowing what the variables are, others internalize the concepts of what is trying to be calculated.
If I cannot recall the information or even that I've come across it unprompted, is it really? Because that's my norm, and I still retain insights from that, that are then applicable across topics.
I would describe myself exactly the same way as you, and I've always been that way (noticed it at first in school where I would take forever to hand in the memorisation half of an exam but finish the analytical half in record time.)
I recommend giving spaced repetition a serious go. It doesn't cost much and you might be surprised how far it takes even someone like you. It completely changed how I view the role of memorisation in analytical work.
Strictly speaking, someone like you does not need to memorise things because you can always derive them from more fundamental principles. But being able to do that, while a blessing, is also a crutch.
Reasoning from first principles every time is slow compared to pulling out the right relationship for the problem at hand right away.
I'm this same way -- very strong semantic memory, astoundingly bad episodic memory. I've been getting into spaced repetition and I was curious if you have ideas on what to memorize via SRS?
I don't really have things in my job or intellectual pursuits that clearly lend themselves to flashcard style memorization, so I've been doing SRS on sort of quasi-useless things to help 1) get in the habit of SRS and 2) build more "scaffolding" in my memory so when I want to SRS useful things it'll be easier.
(By (2) I mean I'm memorizing some historical dates/facts because I found early in using SRS that I'd traverse known facts like a graph, so simply having more known facts would make it easier to add and remember new ones -- most things I'll learn will have a date affiliated with them, so figure anchoring a bunch of dates in my memory won't be totally useless)
I forgot to reply to this and don't want to write a long answer now in case you won't see it. If you're still interested, shoot an email to hn@xkqr.org and I'll give more details. Sorry for taking so long!
I think we have a problem of semantics here. Your notion of "the brain internalizes insights" is very close to what the author means as memorizing patterns. They even gäbe a few examples where they started with rote memorizations, which were not that useful at first, but eventually a pattern, an insight if you will, emerged.
Which is why the thesis here is boring/less useful. “All colors come from memorization” is also accurate. “All thought comes from memorization”. At that point, you’re factually accurate but saying little of use.
If you’re trying to teach creativity, what do you make people memorize? The author even points out: some cultures are great at memorizing and bad at innovation and vice versa. That’s interesting to talk about. “Try-hards use spreadsheets to be funnier” is…sad?
>One of the points I'm trying to make is that taste and elegance fundamentally stem from an internalized heuristic -- which at it's core is memorization.
seems to me there is a relatively big inductive gap there, you believe that there is an internalized heuristic and at its core is memorization, you may even have some evidence that this internalized heuristic has strongly informed your development of taste, but it is pretty difficult to make an argument that is the case for all people.
Aside from that I would say that "internalized heuristic with memorization as the core" puts everything on nurture and no input of nature - which I am pretty much in the camp of combinations of nature and nurture creating the person - of which taste must surely be a big component.
also a question - if you have better long term or short term memory how does that affect taste? How does it affect creativity, if all of these things are essentially memorization you would have to assume that people were more creative and had better taste the greater their ability to memorize things, which in the case of taste especially seems slightly absurd.
In the case of creativity it may be easier to make an argument - but surely you can find people who seem more creative with less ability to memorize.
Why are you attached to the word "memorization" here? Certainly taste comes from experience and learning. Maybe you could argue that all learning is an oblique and imperfect form of memorization—but why argue that at all?
The only reason I can see is if you think memorization could be a shortcut to good taste, which it can't. Acquiring good taste requires broad experience—more information than you can possibly remember—such that you retain a suite of sophisticated intuitions. Cutting that information down to something that can be memorized would require you to (1) already have the intuitions you're seeking to acquire, and (2) be able to express them all in plain English, which, as far as I know, cannot be done. No painter has ever expressed their aesthetic in such a way that a student could memorize that expression and then have the same creative sensibilities as the original painter.
Ultimately, there's no substitute for the process of simply consuming lots of art while paying close attention to what you like about it.
Interesting perspective. I do agree that there are people out there who develop a distinct "taste", but I can't tell if this refers to a "style", an emergent property of multiple "habits", etc? I've always wondered how one develops their "taste".
Also, would you consider a subconscious habit "memory"? What's the difference between the two?
I completely disagree with your assertion that "...rote memorization is neither necessary nor sufficient to internalize concepts."
I would recommend reading the book Moonwalking with Einstein. There is a lot of discussion there on how memory is linked directly to creativity, and to understanding concepts deeply.
---
A choice passage:
"...If the essence of creativity is linking disparate facts and ideas, then the more facility you have making associations, and the more facts and ideas you have at your disposal, the better you'll be at coming up with new ideas. As Buzan likes to point out, Mnemosyne, the goddess of memory, was the mother of the Muses.
The notion that memory and creativity are two sides of the same coin sounds counterintuitive. Remembering and creativity seem like opposite, not complementary, processes. But the idea that they are one and the same is actually quite old, and was once even taken for granted. The Latin root 'inventio' is the basis of two words in our modern English vocabulary: inventory and invention. And to a mind trained in the art of memory, those two ideas were closely linked. Invention was a product of inventorying. Where do new ideas come from if not some alchemical blending of old ideas? In order to invent, one first needed a proper inventory, a bank of existing ideas to draw on. Not just an inventory, but an indexed inventory. One needed a way of finding just the right piece of information at just the right moment.
This is what the art of memory was ultimately most useful for. It was not merely a tool for recording but also a tool of invention and composition. "The realization that composing depended on a wellfurnished and securely available memory formed the basis of rhetorical education in antiquity," writes Mary Carruthers. Brains were as organized as modern filing cabinets, with important facts, quotations, and ideas stuffed into neat mnemonic cubbyholes, where they would never go missing, and where they could be recombined and strung together on the fly. The goal of training one's memory was to develop the capacity to leap from topic to topic and make new connections between old ideas. "As an art, memory was most importantly associated in the Middles Ages with composition, not simply with retention," argues Carruthers. "Those who practiced the crafts of memory used them---as all crafts are used---to make new things: prayers, meditations, sermons, pictures, hymns, stories, and poems." ..."
Great book, motivated me to then read The Art of Memory by Frances Yates.
Although I'd say traditional mnemonic devices like memory palaces are basically linear information storage and recall devices. This can create issues in building a flexible web of information, because loci or the order of the path can become dependencies and you can run out of unique spots in a given space, leading to memory interference.
Even spaced repetition methods (e.g. Anki) tend towards fragmentation of micro-ideas. Its perfect for terms, languages, and simple one question -> one answer ideas.
I've found a hybrid method of images, nested loci and spaced repetition to be most useful, because its flexible over time, and preserves relationships of ideas.
(Context: I co-founded a SaaS in this space: www.sticky.study)
You are very correct in my experience since mnemonics backfired on me that way. It was like my brain constricted on those but recall was good in limited situation.
Thanks for sharing your alternative. I like that you’ve included your references for each component of your method. That might help as many people as your product. I’ll look into it sometime.
The internet has so much information we often can't actually find what we're looking for once we get past a certain surface level understanding. Or people don't want to pay to host it anymore. Our people disagree with it and it's taken down.
I don't disagree with Einstein, but I wonder what he would say with the modern internet at his disposal. Maybe the same?
> Memorization in programming gives us architecture astronauts and design-pattern soup rather than elegant code.
Elegance is probably orthogonal to creativity, and likely follows from some kind of minimization principle, like minimum program length. You are effectively distilling the "essence" of something from all of the noise.
Creativity seems different, more like novelty, and creativity following some kind of remix of memorized elements + some randomization seems very plausible.
You can create something novel but not elegant, and something elegant but not novel, and you can distill an elegant version of something novel that your or someone else created and that's the best of all creations.
The book Make it Stick taught me that this Don’t Cramp My Style With Your Boring Rote Learning (Man) attitude is prevalent in teaching. At least American. They argue that it is wrong for the same reason that the author does.
But saying this to a programming crowd must be the most futile thing. At least instrumentalists have to rote train their muscle memory. That lowest bar has to be passed, even if it’s just three chords.
But the article isn’t about programming creativity though. It is a general concept. But if honing in on the mythical lone-genius activity (geniuses never practice in a structured way) helps you win an argument then so be it.
> Well, okay, but rote memorization is neither necessary nor sufficient to internalize concepts.
Of course it is. It's how every human child learns initially. By rote memorization. How does a toddler learn how to say mama? By constantly hearing and repeating it. How does a kid learn their ABCs? Rote memorization is the basis of all memory.
> Memorization in programming gives us architecture astronauts and design-pattern soup rather than elegant code.
Dumbest thing I've ever read. You write programs well by doing and remembering. Same with writing. Memorization is the necessary component to programming well. In other words, you program well by remembering elegant code.
> For me, at least, trying to memorize something without context
After the basics, most memorization is contextual.
> At the same time, memorization has a real cost: it takes time and it's frightfully dull.
Oh dear. Something isn't fun all the time. What a childish worldview. It's more fun to eat candy and drink soda than eating 'dull'. It's more fun to sit and watch youtube than to workout.
> Sometimes a bit of memorization is unavoidable, but I've found that to be relatively rare.
Relatively rare? In order to be competent in anything, you have to memorize lots. You can't write a good essay without having memorized much of the material. Trying reading a book where you have to constantly look up definitions of words because you lack the vocabulary. Try having a conversation with someone who has to constantly look up words because he lacks the vocabulary. Try having code review with someone who doesn't remember anything about their code.
> Otherwise, my time is generally better spent on some sort of practice in context.
Why? Because it helps you remember?
To the idiot ( probably OP ) who downvoted, try coding without having 'memorized' the keyboard. The anti-intellectual, anti-hard work, anti-memorization agenda pushed by some 'people' online bears looking into.
> One of the reasons people make fun of the author's approach to creativity is that systematic memorization fundamentally can't teach taste—so the systematic approach reeks of awkward, try-hard, low-brow, tasteless art.
Well... can you think of an artist who didn't have a deep knowledge of their art-form before they pushed it forward? Three that jump out for me, in no particular order, are Picasso, Borges and Jack White. After all, great artists steal.
Pollock was obsessed with creating an art style that had no basis in any other style, something truly "original." He felt the abstract style he created fit that aim.
That being said, in some ways you could say that the splatter paintings he's known so well for are in fact influenced by all the art he studied and discarded along the way. They were definitely influenced by the principles of artistic design he learned, even if they looked different from what people were used to.
In my opinion, your hypothesis is supported, though maybe in a bit of a roundabout way.
Wait, why do you think Picasso didn't have deep knowledge? He studied both at the School of Fine Arts in Barcelona & the Royal Academy of San Fernando in Madrid, for ~5 years before moving to Paris.
Borges was incredibly talented, but it's worth keeping in mind his dad was a writer too.
Good art very much relies on being exposed to lots of other good art first. I don't know that rote memorization is the best way to achieve that, but you definitely need that exposure.
Sorry, I must have expressed myself badly. I'm picking examples of people I think did/do have deep knowledge of their chosen mediums.
I don't think it's possible to have "good taste" without exposure to lots of examples, because I believe taste it culturally bound. Whether you do it explicity via a system, or on a more ad hoc basis, I think most artists need it.
It might be interesting to look at film, where the process is compressed into a couple of generations. I don't know it it will support my argument or not.
I don't think film will look very different here - early film work was very much informed by theatrical tastes at the time, and then started to diverge as people figured out what else they could say in the language of film.
Fundamentally, all art exists in a cultural context. If you've ever taken an art history course, you've been hit over the head with that info a few times ;) And that means furthering/changing taste in a given field means being aware enough of the existing rules to deliberately choose which ones you're breaking, and why.
There are some (very few) artists who didn't have a formal grounding, but I'd argue that even they were steeped enough in cultural context to be informed by it. Even famous autodidacts like Grandma Moses did develop a love for art based on being exposed to a bunch of it.
(Fully recognizing that it's a somewhat tautological argument because it's kind of impossible to grow up in a society without being somewhat exposed to its predominant art forms)
There's a big difference between "artist who didn't have a deep knowledge of their art-form" and "artist who didn't follow an explicit system to memorize a bunch of rules to make their art".
I think that argument is that these artists did not memorized rules or previous pictures and then applied them. They did put a lot of effort into learning, but that is different claim. If you define "memorization" as "any learning of anything", then the word is kind of useless.
I think it is more of becoming fluent with primitives that can be composed in versatile ways. I can see how that can be poorly understood as memorization.
The main implication is that if what you are “memorizing” is not easily composable, then you won’t be able to apply them broadly or creatively.
However, I disagree with the author on what creativity is, although his definition is one experience of a creative inspiration.
I disagree with elements of this article, but enjoyed reading it. True creatives do not subvert norms consciously or with an acute awareness as part of this article suggests, I agree they need to be exposed to the norms to generate their own interpretation, but I don’t believe that true creativity is a conscious exercise.
Well said - what's funny is that many creative ideas (from an outside perspective) are often very simple connections of two existing ideas for the person that actually did something creative! It's just that other people don't have the same context/knowledge as the creator, so an idea can seem extremely original to them
It’s not memorizing, it’s actually knowing and understanding the utility of different concepts. When you learn of a problem in a new field, you go through your bucket of tools and modify one to fit the new problem.
Probably you would have had more agreement, if you would have used a different word (though I’m not able to suggest that different word that captures the meaning I think you intended)
You have to know things to be reasonably creative but there is a point where memorizing more stifles creativity. Memorization is often very passive and is fundamentally different from searching for new ways of doing things. It's hard to make sweeping statements like this because there are different modes of memory and different modes of creativity.
There is a lot of memory involved in being creative, but I think setting out to memorize things is a bad way to be creative. You have to practice being creative. In doing so, you will naturally remember a lot of stuff like what works, what doesn't, and most importantly which types of things you ought to memorize. For example if you're programming you will find it useful to remember the syntax of your languages. If you're writing you'll find it useful to remember styles and vocabulary. And so on...
I agree with the author, at least in my own creative experiences. However, it's more likely the case that 'creativity' is arrived at differently for everyone. I find memorization to be a comforting foundational activity that builds knowledge & confidence, which I can later express creatively.
This claim makes little sense because it fails to distinguish between memory and memorization. I memorize almost nothing, but I remember the broad strokes of a lot of things. This allows me to be creative.
In a way, memorization is a severe risk: if you memorized something before it changed, for example, your creativity may not mean much.
What I was trying to convey is that fundamentally learning is memorization, whether conscious "rote memorization" or more less-intentional committing to memory from doing an activity
And that recognizing this allows us to speed up the process of learning fundamentals
Which in turn enables creativity as most people see it
You state that "learning is memorization" but I don't think that is true. Of course learning involves something being persisted in one's brain, but stating that memorization and brain persistence persisting are synonyms seems like an incorrect description.
You’re right. The OP is mistaken as to the distinction between memory and memorization. I am able to be creative because, for example, I remember that a woman’s hair smelled like flowers and secrets. It would not be possible to memorize such a thing.
Conscious memorization and rote memorization are two different things. I can intentionally put things into memory without doing rote memorization - or even intentionally avoiding rote memorization. The techniques you use also give you different results in terms of whether or how you use the memorized concept or word in foreign language. (For example getting the effect where you can translate a word between foreign and your language, but can not use it foreign language sentence and do not understand it in context without translating.)
What's more is that memories are just a replaying of neuron connections activating in the brain - and when we are prompted by the world around us those connections will fire in response to the stimulus. Quite similar to how AI neural networks function - which is why I believe that AI can indeed be creative and create "new" ideas
I actually made a video diving deeper into this and comparing responses from people and ChatGPT for a creative thinking problem - https://youtu.be/l-9EUBbktqw
I think your hypothesis here (and probably the entire article as well) is strongly challenged by the 'progenitor argument.' Take humans at the dawn of humanity. Language did not even exist beyond what may have been crude sounds or gesturing and collective knowledge did not fall that far beyond 'poke him with the pointy side.' Somehow we went from that to putting a man on the Moon in what was essentially the blink of an eye.
Training an LLM on the entirety of knowledge at this dawn of humanity and, even if you give it literally infinite training time, it's never going to go anywhere. It's going to just continue making relatively simple recombinations of its training set until somebody gives it a new training set to remix. This remix-only nature is no different with modern knowledge, but simply extremely obfuscated because there's such a massive base of information, and nobody is aware of anything more than a minuscule fraction of it all.
---
As for the 'secret' of LLMs, I think it's largely that most language is extremely redundant. One thought or point naturally flows.... why do I complete the rest of this statement? You already know exactly what I'm going to say, right? And from that statement the rest of my argument will also mostly write itself. Yet we do write out the rest, which is kind of weird if you think about it. Anyhow the point is that by looking at language 'flow correlations' over huge samples, LLMs can reconstruct and remix arbitrarily long dialogue from even the shortest of initial inputs. And it usually sounds at least reasonable, except when it doesn't and we call it a hallucination, but it's quite a misnomer because the entire process is a hallucination.
Interesting point - thanks for sharing! I think one big missing piece we have with AIs today is the ability for them to learn on the fly and reconfigure the weights. We are constantly bombarded with input and our neurons adjust accordingly. Current LLMs just use a snapshot. I would be really curious to see how online-first AI models could work, focusing on a constant input stream and iterating on weights. Also I wonder how much knowledge is baked into our DNA through evolution. I have a hunch that this is somewhat analogous to model architectures.
Btw - although I see evidence of LLMs creating "new ideas" through combinations of ideas, I am a bit mystified by their apparent reasoning issues. I wonder how that is different in nature from the memory-based approach. ARC-AGI benchmark has had me thinking about this for sure.
On the other hand, episodic memory (insofar as that is distinct from "3*9=27" memorization) is built on top of creativity.
The vast majority of what we consider "memories" are the creative brain doing an on-the-fly story generation, massaged until it "seems right" and serviced plus a big dollop of emotional confidence.
You may as well say that creativity comes from writing. Because obviously all of the most creative writers write. And the writings of creative non-writers are entirely absent.
It’s just a bunch of arbitrary unprovable assertions.
Everyone here seems to have, broadly speaking; neither a) the qualifications to knowledgeably comment of the (honestly poorly understood, afaik) function of “creativity” or b) anything more meaningful than “here is my naive personal lived experience and opinion” to contribute on the topic.
It’s just armchair psychology.
If you want to wax philosophical, by all means, but I think anyone taking “thoughtful insight” away from this article or thread is fooling themselves.
Whatever the epistemic quality of the article is, it's triggered some interesting discussion here which I think is valuable. No need to denigrate talking about human experience with other humans, I think?
One might talk about it from the perspective of birdsong, which is used by mates to judge sexual fitness. First a tutee bird learns from a tutor bird, and then eventually applies variability to the original song.
It's strongly suspected that anterior forebrain pathway (AFP) may be a source of behavioral variability. We naturally age over time, including our vocal musculature, so in some sense we must constantly relearn how to use our muscles to deliver a song.
When a bird is deafened its birdsong will naturally drift, but when we precisely damage the AFP along with deafening we find that birdsong remains stable for a longer period of time, until of course inevitably it must drift due to aging vocal musculature.
You'll experience the creativity outlined in the article directly when you start doing deliberate memorization, i.e spaced repetition. No qualification needed.
I think semi-obvious would be a better criticism than unprovable.
You can't make connections unless you have things to connect to.
You can't recognize (your own discovered/inspired) novelty unless you have memorized normality.
If you are creative in the absence of knowledge of what already exists then that's considered as reinventing the wheel, and not very useful, even it it's Ramanujan reinventing much of established mathematics.
It's pseudoscience, author would benefit from reading (and memorizing) the current scientific literature on learning and cognition.
There's a bad trend, Hacker News gets this kind of blog-style self-promotion every month that gets much comment attention, but the essays are not well-researched and with made-up assertions written by programmers talking out of their lane ("Engineer's disease") and not having done homework on the subject.
> It's pseudoscience, author would benefit from reading (and memorizing) the current scientific literature on learning and cognition.
How so? Can you share what you have read and that is relevant and applicable in real life?
> There's a bad trend, Hacker News gets this kind of blog-style self-promotion every month that gets much comment attention, but the essays are not well-researched and with made-up assertions written by programmers talking out of their lane ("Engineer's disease") and not having done homework on the subject.
You are on the wrong site, this is not a scientific journal there is no need for scientific rigor in every post and comment.
P.S. The real life world is full of events and things happening if you cannot learn by yourself (create theories models on how the world works aka. pseudoscience apparently) and need academic verification for everything. Then you some kind of disorder :/ good luck
There absolutely is a need for fact checking and basic scientific literacy and you're helping foment the opposite by suddenly making this about academia and scientific journals. What's the harm of that? Snake oil promotion and creation of filter bubbles.
There's absolutely an expectation of not rejecting science just as with discussing global warming or Covid.
But explain, why do you pervert an expectation of scientific literacy into an expectation of professional scientific expertise?
Just as there are scientifically informed books and articles on a e.g. health and fitness and dieting, there is a ton of material on education psychology. Why do you allow the author to not do due diligence and at least read the basics which are very much accessible to nonexperts? Why make an exception when in every other STEM topic this would be incredibly ignorant?
If it were a Time magazine article or a newspaper article they would surely include scientific sources in an effort to be truthful. Surely any high-school student knows the steps to write a well-researched essay, so why are you making this about the irrelevant standards of professional academia here? Could it be due to a misunderstanding of the role of science in public education? Are you some kind of Covid vaccine denier or anti-science on some other topic?
In summary, you have wrongly conflated scientific literacy, being scientifically informed, public scientific awareness on a topic, etc., with professional academic scientific expertise. That is irrelevant and absurd. It also offends me that you made a bad faith construal of my statement.
As to your remark about HN, my criticism applies to ANY scientifically informed public discourse, even as HN is a STEM website.
Finally: I'm a reader giving my time and attention to a badly written essay because it got posted on a forum. It is not my job to fix their essay if it is antiscience in the way an antivaxxer makes up pseudoscience reasons why their idea is right. I'm not going to patiently explain to the nth antivaxxer why they are spouting pseudoscience, I'm just going to say that bluntly. This endless handwringing that HN people have over self help and rote memorization is just more of that, it's on them to have read a few simple books and articles on the topic before choosing to write about it for the public.
PS. In "real life", people express sarcasm and maybe your 'disorder' (completely out of line of you to say that, BTW, and against site guidelines to use personal attacks of that sort at fellow users) was failing to detect a sarcastic comment if you actually, literally, thought I meant "the author needs to memorize scientific literature" in context of their argument about memorization. That would have nothing to do with the actual accusation of pseudoscience, where being scientifically informed on a topic is just a basic standard of discourse to avoid misinformation and snake oil promotion.
Except that commenter had construed my sarcastic point and made it about academia. It is an irrelevant reading of the remark. They had a reading comprehension problem.
Without sarcasm, my point was/is not that you need to know all the theories or engage with academia or write a formal paper.
Rather, my point is that if you're going to write on a topic, at least know what the basic science has to say on it. You wouldn't write about mask wearing, or recycling plastic, etc., based purely on anecdotal evidence. You would include information from scientific sources and consensus.
Doing that raises the level of discourse and forces you, the author, to be responsible for not propagating misinformatiom and pseudoscience. Especially so in the area of self-help and learning psychology.
So again, my original comment while making a sarcastic remark about memorizing scientific literature was really about communicating your ideas in a scientifically literate way just as any lay person has a personal responsibility when discussing a topic, be it about technology or psychology or sociology, etc.
I disagree, I think writing purely based off anecdotal evidence is fine as long as you're not making scientific claims (which I'm not).
Simply sharing what works for me and why I think it does for something as personal as creativity is not the same as contesting mask wearing or recycling.
Is there some specific point in my piece that disagrees with consensus that's not just contesting the definition of "memorization" (as seems to be common in this thread)?
All very nice and handwavey, but then you see the user’s current venture is scammy deepfakes as a service, which is about as creatively bankrupt as it gets.
Shame about the national stereotypes as well. There is plenty of creativity in Asian countries. Just bizarre assertions all around.
Historically eastern Asian cultures have placed duty to a whole host of things before oneself, and in many cases the old aphorism "the nail that sticks up gets hammered down" applies as well. Plus Japan and Korea have well established cultures of adherence to tradition and mastering simple, time tested things to a ridiculous degree rather than trying to innovate.
Not true of everyone, but if you compare a culture that values conformity and tradition to a country that values the freedom for the individual and trying new things, of course it's not going to measure up by western standards of creativity.
Have you seen their art and entertainment? I assure you there is whole lot of creativity in there. And it has a whole lot MORE variety then western tend to have.
Their art and entertainment in many cases are rebellions and critiques from mainstream norms. The funny thing is that if you have a rigid conformist society, the rejects are going to double down on the "weird" much more than a well adjusted creative would.
>Their art and entertainment in many cases are rebellions and critiques from mainstream norms
No, that's just the art that Western readers notice because that's the only thing they recognize as art in the first place. Calligraphy (in China traditionally considered the most important form of visual art) for example has an astonishing tradition in East Asia, also notably related to the topic of the thread, memorization and repetition and practice and has very little to do with critiques of norms.
I found asian genres to be way more diverse and creative then western entertainment even when they dont criticize anything.
Western entertainment tend to produce the same story, in like, two genres, again and again and again and again. Most of the time you can predict the movie storyline down to minute - and people will argue that it is the only correct way.
The people arguing that is the only correct way are the capitalists putting their money on the line, and that comes from a perspective of being risk averse, which is why "corporate art" in America is in such a shambles. Independent video and music are in a great state though.
Are you seriously going to hold up calligraphy as an example of extreme Asian creativity? The art of writing letters with subtle flourish? It's literally an art of understatement, and embodies all the characteristics that I stated are reasons east Asians are culturally biased towards being less innovative.
Yes. Understatement, subtlety and an eye for detail aren't opposites of creativity, it's actually sad that this even needs to be stated. There's no indicator at all that Asian societies are, in any way, biased against being innovative. I recommend reading Jun'ichirō Tanizaki's In Praise of Shadows, it's a fantastic read on the indirect and minimal ways in which Asian societies express creativity and aesthetics.
Just because you're loud and brash and write your inventions on your forehead doesn't mean you actually are more innovative or creative than anyone else. I know we've had bad comedians in the West who have made careers out of thinking that being loud equals being funny but you seem to have made an entire worldview out of that idea.
Tell you what. You can use the word "creativity" to mean "creates things" and keep that separate from "innovation" which means to create new things. Sure, east asians are "creative" but 2000 years of evolution in calligraphy pales compared to 500 years of stylistic evolution in western art. In general, westerners like to take chances for personal glory while asians seek to elevate the things their culture already values and has done for generations.
As for your comments about my taste, it's a good thing taste is entirely subjective, you can think I'm boorish and I can think you're boring and lack vision, and we're both entitled to our opinions.
Some minds get a lot done with a lot of memorization and some minds get a lot done with seeing commonalities and creating simplifying abstraction. We need all sorts of minds.
I don’t think that’s true. People with memory problems more seem to retread the same foundational ideas repeatedly, whereas those with good memories recognize this repetition and then seek something novel.
I was the sort of person who did not believe in memorisation as a solution for anything. Then I tried getting really good at spaced repetition for a year (yes, it is a skill that needs to be trained for good results) and I've completely changed my mind.
Spaced repetition allows me to become proficient even in things I don't get the natural opportunity to practise daily, so that when the day comes and I need them, I have some level of knowledge already. This has happened to Kubernetes troubleshooting, statistics, PowerShell windows programming, and traffic engineering just in recent history.
I have yet to publish some of these, but I have examples from statistics:
I'm similar. This from your top link stood out to me:
"It’s a little like building with lego bricks or something – spaced repetition helps ensure all the tiny pieces are in the right place, so that the big castle can happen without structural integrity issues."
The book Make it Stick (by Henry L. Roediger III) had a similar idea they called 'Structure Building'. Very similar to what you described, more experienced and effective learners were creating mental schemas of how the little, but crucial parts of a subject fit together, and successfully cut through the noise.
Structure Building was associated with interleaved practice (shuffling of problem types) and spaced retrieval practice.
My argument is that it is worth memorising also the derivations, rather than re-deriving from scratch each time.
Meorising the derivation makes it easier to derive a second-order derivation, and so on. At some level of abstraction, going from first principles becomes prohibitively expensive and caching intermediary results, or so to speak, unlocks that again.
Math proof and derivations are a bit like remembering a walking route. You've seen the start and end, and the main turns taken, and there's also a general "walking" skill you need.
This is similar to a loose life-thought i've had for a while, though i lack a catchy phrase for it lol.
My thought is: You're always practicing a behavior whether you like it or not.
Your mind is always setting you up to do more of whatever it is you're doing now. Both physical and mental. To do it more efficiently. With more ease. With more frequency. etc.
It's good motivation for me to mitigate a lot of negative behaviors. Angry in traffic, self anger, etc. If i can reason at least, of course. As i'm not interested in doing a lot of things more than i am, so i should avoid doing them now - if possible.
I really enjoyed both blog posts, thank you for sharing! And I have to say, your explanation of the subexponential distribution property was remarkably clear for someone without a background in statistics :)
Would you mind sharing the flashcards you generated to build this intuition? I've been using Anki for a while and really trying to focus now on improving my prompt writing; would love to see how you managed it for this problem.
As much as I would like to, I think getting to that understanding required at least 500 flashcards on general statistical and probability concepts, ranging from fundamentals to extreme value theory. Most of those are only barely relevant at face value, but still contribute to understanding.
It's not that I set out to understand this specific thing but that I had studied statistics with flashcard support for a year and that happened to work after a few attempts.
I've long wanted to write about this but never been able to think of anything original to say, but your question forced me to face this with effort. Thanks!
When making flashcards I draw a lot from the softer type of theory-building they do in social sciences. I ask questions like
- What are the properties of this?
- What variants of this exist? I.e. how would I recognise this in the wild, or in other shapes?
- What subcomponents can this be deconstructed into?
- Into which bigger picture does this fit?
- What are the consequences of this? What are its antecedents?
- What is this a special case of? What would a generalisation of this look like?
- Which are other related things? What are their similarities and differences?
- In what context might I need to know this?
Whenever I encounter what seems like a significant thing I loosely ask some of these questions, and try to construct atomic, focused flashcards from the answers.
I say loosely because it would take forever to to through all questions for all flashcards I make, so there's some bit of intuition that attracts me to which I think are the most significant questions for any given thing.
-----
One trick to make flashcards more specific that I use (maybe even abuse) is putting part of the answer into the prompt. Instead of prompting "What is the property of subexponential distributions I found meaningful in this book?" I might prompt "What behaviour do subexponential distributions have around high barriers that others don't?" -- I'm giving away part of the answer by including "high barrier" in the prompt, but I'm okay with that.
If I'm concerned about that, I might create a second flashcard prompting something like "What can a subexponential distribution do in one step that a more well-behaved distribution needs many steps to do?" with the answer "clear a high barrier". That captures both sides of the property without making too general a prompt.
I also do this a lot with "why" questions. Instead of prompting "what is the definition of y?" I might prompt "why is the definition of y=f(x)?" That gives away essentially the entire answer but focuses on the why instead.
We recently described this to a parent of one of our students as:
Understanding -> Remembering -> Applying
If you don't understand the basics of a concept, and you're talking about memory, its probably just rote memorization. Students generally find this tedious, and since it's shallow its very hard to retain and connect to disparate but parallel ideas from other fields (roots of creativity).
But, most schools and students stop there. They hear 'memory' or 'memorizing' as only rote memory. Step 2 is critical if you want to get to higher levels of learning. As you said in the essay - "Creativity comes to those who have internalized the patterns of their art". At www.sticky.study this is what we focus on. It's fast 2D memory palaces + spaced repetition.
Only if you have understanding + remembering can you get to step 3 - applying what you learn reliably at relevant moments in your life. This is the gold standard that schools claim they desire - analysis, synthesis, application, broad transfer, and creativity. You can't reach master efficiently if you lose 80% of all you read or learn.
I hear with llm/chatGPT people don't have to blog anymore, but rather the generated plausible-sounding well-structured nonsense flows like an open sewer onto the web.
1. Creativity in a commercial context once stolen/cloned through back-channels accrues value, and manifests as several competitive campaigns
2. New disruptive ideas are usually shelved until the IP/patents expire. No one wants to go through the sometimes impossible licensing process
3. Emerging technology is usually degraded in the rush for IP assets by established firms i.e. large firms dump billions on ridiculous concepts out of fear of market fragmentation
4. Startups do not usually have cash to burn on speculative IP. Thus, real cutting-edge experimental technology is sometimes never made public for numerous reasons.
371 comments
[ 4.1 ms ] story [ 367 ms ] threadAnd again after learning how to acquire new skills quickly
But how do you know how to apply this acquired knowledge in this new context?
It's some form of pattern matching right -- which imo is just a less obvious form of memorization
i.e. you've memorized the match between inherent traits of the context with a specific application of that knowledge
That is, creativity fundamentally comes from internalized knowledge (as the article says) but internalized knowledge doesn’t necessarily come from memorization.
While internalized knowledge comes from "subconscious" (for lack of a better word) learning?
I guess I'm equating the two here and just using memorization as "committing to memory", with the belief being that you can construct the heuristic you'd normally acquire subconsciously and cut down time to mastery
That question has a false precondition baked in. If you know how, it's not creative.
> It's some form of pattern matching right -- which imo is just a less obvious form of memorization
No. Sensing and matching patterns does not imply memorization. Everything you're saying is completely loaded.
Did you just discover memorization? Because the pattern I see in your words is similar to anyone who's just learned a new tool or technique - they overapply it everywhere as they learn to use it.
E.g. if you're an artist, at the very least you need the knowledge of how to draw a line
From other comments here it seems the definition of "memorization" seems to be where disagreements are
Maybe this is a better explanation: once I started trying to make whatI just learned is called "tacit knowledge" more explicit and then committing it to memory, I was able to cut learning times down significantly
But no artist memorize how to draw a line. They learn how to draw a line, but learning isn't the same thing as memorizing.
I don't think that meets the commonly accepted definition of memorization.
E.g. https://dictionary.cambridge.org/dictionary/english/memoriza... Memorization: the act or process of learning something so that you will remember it exactly.
If you are transforming the knowledge, i dont think that is the same as memorizing it. In fact, i think most people would describe that as the opposite of memorization.
Using your definition, everything would be memorization. For example, you could describe picasso as simply pattern matching on other paintings to make something new, but i think it would be crazy to describe his work as an act of memorization (then again, the way AI generated art works... maybe its not so crazy)
Although sometimes “memorization” doesn’t happen because you sit down to do it but rather that you keep using the same things over and over when solving problems that they become internalized. I find that to be a more fruitful path towards understanding that I don’t want to call memorization but it is.
And agreed -- it's this exact realization that led me to both this method and title
Imo this negative connotation has made many people refrain from calling internalization what it is
But acknowledging that it's memorization has actually made me more efficient at learning, since I can now consciously look for the heuristic, codify it, and try to commit it to memory
After that I realized that creativity wasn't some magical quality that would be hard to reproduce mechanically.
And that also made me a little sad.
It similarly took the magic out of creativity and learning a bit, and made it all seem like work
The main way I've found around it is the joy in being creative once basic autonomy is achieved in new skills
Consciously discovering the heuristic is another fun part
[1] Do you think that ChatGPT can reason? https://www.youtube.com/watch?v=y1WnHpedi2A
Creativity, at least in fields traditionally considered to be ‘creative’ like music and visual art, is 90% randomness and 10% retroactively attempting to ascribe meaning to the randomness. And this is coming from someone who hugely values and defines oneself via an ability to be creative.
Also, LLMs haven’t hit ‘escape velocity’; that terminology has spread like a meme (because it’s an attractively dramatic yet easy to understand idea) but isn’t backed up by any science. Maybe one day we’ll have the fabled ‘AGI’, but ChatGPT isn’t it.
Practice is an oft suggested solution to developing mastery, but I did like how the article framed it: creating subconscious heuristics and memory.
But imo acknowledging this unlocks greater speeds and gets us to the "fun part" quicker
Well, okay, but rote memorization is neither necessary nor sufficient to internalize concepts.
One of the reasons people make fun of the author's approach to creativity is that systematic memorization fundamentally can't teach taste—so the systematic approach reeks of awkward, try-hard, low-brow, tasteless art.
More broadly, memorization doesn't help much with any sort of tacit knowledge, not just taste. I just figure taste is especially important in creative endeavors. That's definitely the case for programming! Memorization in programming gives us architecture astronauts and design-pattern soup rather than elegant code.
For what it's worth, I do think that it is useful and important to have a good mental model of what expertise is and how you can develop it. Memorization might be a component of this, but it's going to be a small component at most. I expect that realistic practice with fast feedback and expert mentorship matters far more. (If you're curious, I found the book Sources of Power by Gary Klein gave me a good way to think about how expertise works.)
At the same time, memorization has a real cost: it takes time and it's frightfully dull. For me, at least, trying to memorize something without context is not just ineffective but also totally kills any intrinsic motivation I have for whatever I'm learning. Sometimes a bit of memorization is unavoidable, but I've found that to be relatively rare. Otherwise, my time is generally better spent on some sort of practice in context.
One of the points I'm trying to make is that taste and elegance fundamentally stem from an internalized heuristic -- which at it's core is memorization.
I understand the connotation of "memorization" evokes an image of blindly memorizing without connecting, but isn't the tastefully developed expertise just memorization of a better heuristic?
[1]: https://en.wikipedia.org/wiki/Tacit_knowledge
This may be because I'm not good at picking up on social cues, so had to learn things more consciously
But ofc I could also be wrong and maybe there are things the subconscious can learn that the conscious cannot
Expertise is the result of learning from past experience, both in developing an internal intuition for what you're doing and in having past patterns to draw upon. To the extent that experts have simple easily verbalizable heuristics, these are largely post-hoc attempts at explaining their intuition rather than an accurate reflection of how they make decisions.
And, in fact, experts can't even always do that: it is perfectly possible for experts to make good decisions without being consciously aware of why they are making them, and explaining how to make good decisions is a separate skill from being able to make them in the first place. The book I mentioned has a memorable story about a firefighter who thought he had precognition after pulling his team out of a dangerous situation without any specific indicator of the danger, but I figure a more common example is experts saying they did something because it was the "obvious" or "clean" or "better" way to do it and getting a bit flustered when pushed further.
We can see this in action pretty clearly if we look at advice for, say, writing. There is a lot of advice from good writers but just memorizing and blindly following this advice is actively counterproductive. Advice you can memorize fundamentally must lack nuance and context. We can see this clearly because so many different pieces of writing advice contradict each other and because good writers do not follow any of those suggestions with any consistency.
The same definitely applies to programming, which is why we have both "don't repeat yourself" and "you ain't going to need it", and why new programmers trying to apply either rule (or both!) to a codebase inevitably create a mess. What I've found with programming advice is that most suggestions are either actively wrong or too vague to be useful. (By the time you've learned enough about programming to be able to follow the vague advice, you don't need it very much!)
What I find instead, is that by just processing novel information, especially if I focus on analysing it, my brain internalizes insights and builds model of that type of thing, allowing me to either imperfectly reconstruct what I've seen, or to come up with an infinite array of permutations, extrapolations, etc which is where the real ideas come from.
Further, ideas crucially revolve not around just the information itself, but the "feel" for what role they play in the whole, how well they do it, in what way they're notable, etc.
In fact I'd straight up claim that memorization is antithetical to creativity - a perfect ML autoencoder or GAN would just regurgitate the training data. Creativity comes from generalisation while memorisation is analogous to overfitting.
I feel like often the reason a creative person is hyper creative is they haven't memorized things so they are trying to rebuild information all the time in their heads from very sparse details.
This creates the transformative and relational combinations of information that a person memorizing can't see because it is created from a lack of organized specific information rather than a bounty of it.
Sorry, but this is memoization.
But that's just my interpretation of the terms. I don't know what the "official" meanings are.
Is there a word for this?
Similarly, we should have a word for knowing how to reuse something in a different context, but not recall its origin or its canonical portrayal. Being able to apply a rule, without being able to recite it.
Do you think there's one word which means both of these things, which are opposites, as I've stated them?
"Rote memorization"
Transferring skills from one context to another is surprisingly hard to do, but not impossible. AFAIK, contexts must be similar to each other for transfer to take place.
> memoization
Was this intentional and no one caught on? https://en.m.wikipedia.org/wiki/Memoization
Or just a typo?
I recommend giving spaced repetition a serious go. It doesn't cost much and you might be surprised how far it takes even someone like you. It completely changed how I view the role of memorisation in analytical work.
Strictly speaking, someone like you does not need to memorise things because you can always derive them from more fundamental principles. But being able to do that, while a blessing, is also a crutch.
Reasoning from first principles every time is slow compared to pulling out the right relationship for the problem at hand right away.
I don't really have things in my job or intellectual pursuits that clearly lend themselves to flashcard style memorization, so I've been doing SRS on sort of quasi-useless things to help 1) get in the habit of SRS and 2) build more "scaffolding" in my memory so when I want to SRS useful things it'll be easier.
(By (2) I mean I'm memorizing some historical dates/facts because I found early in using SRS that I'd traverse known facts like a graph, so simply having more known facts would make it easier to add and remember new ones -- most things I'll learn will have a date affiliated with them, so figure anchoring a bunch of dates in my memory won't be totally useless)
Memory is the basis of knowledge.
If you’re trying to teach creativity, what do you make people memorize? The author even points out: some cultures are great at memorizing and bad at innovation and vice versa. That’s interesting to talk about. “Try-hards use spreadsheets to be funnier” is…sad?
https://news.ycombinator.com/newsguidelines.html
seems to me there is a relatively big inductive gap there, you believe that there is an internalized heuristic and at its core is memorization, you may even have some evidence that this internalized heuristic has strongly informed your development of taste, but it is pretty difficult to make an argument that is the case for all people.
Aside from that I would say that "internalized heuristic with memorization as the core" puts everything on nurture and no input of nature - which I am pretty much in the camp of combinations of nature and nurture creating the person - of which taste must surely be a big component.
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes
https://arxiv.org/abs/0812.4360
In the case of creativity it may be easier to make an argument - but surely you can find people who seem more creative with less ability to memorize.
The only reason I can see is if you think memorization could be a shortcut to good taste, which it can't. Acquiring good taste requires broad experience—more information than you can possibly remember—such that you retain a suite of sophisticated intuitions. Cutting that information down to something that can be memorized would require you to (1) already have the intuitions you're seeking to acquire, and (2) be able to express them all in plain English, which, as far as I know, cannot be done. No painter has ever expressed their aesthetic in such a way that a student could memorize that expression and then have the same creative sensibilities as the original painter.
Ultimately, there's no substitute for the process of simply consuming lots of art while paying close attention to what you like about it.
Also, would you consider a subconscious habit "memory"? What's the difference between the two?
I would recommend reading the book Moonwalking with Einstein. There is a lot of discussion there on how memory is linked directly to creativity, and to understanding concepts deeply.
---
A choice passage:
"...If the essence of creativity is linking disparate facts and ideas, then the more facility you have making associations, and the more facts and ideas you have at your disposal, the better you'll be at coming up with new ideas. As Buzan likes to point out, Mnemosyne, the goddess of memory, was the mother of the Muses.
The notion that memory and creativity are two sides of the same coin sounds counterintuitive. Remembering and creativity seem like opposite, not complementary, processes. But the idea that they are one and the same is actually quite old, and was once even taken for granted. The Latin root 'inventio' is the basis of two words in our modern English vocabulary: inventory and invention. And to a mind trained in the art of memory, those two ideas were closely linked. Invention was a product of inventorying. Where do new ideas come from if not some alchemical blending of old ideas? In order to invent, one first needed a proper inventory, a bank of existing ideas to draw on. Not just an inventory, but an indexed inventory. One needed a way of finding just the right piece of information at just the right moment.
This is what the art of memory was ultimately most useful for. It was not merely a tool for recording but also a tool of invention and composition. "The realization that composing depended on a wellfurnished and securely available memory formed the basis of rhetorical education in antiquity," writes Mary Carruthers. Brains were as organized as modern filing cabinets, with important facts, quotations, and ideas stuffed into neat mnemonic cubbyholes, where they would never go missing, and where they could be recombined and strung together on the fly. The goal of training one's memory was to develop the capacity to leap from topic to topic and make new connections between old ideas. "As an art, memory was most importantly associated in the Middles Ages with composition, not simply with retention," argues Carruthers. "Those who practiced the crafts of memory used them---as all crafts are used---to make new things: prayers, meditations, sermons, pictures, hymns, stories, and poems." ..."
Have never heard of this book but adding to my list now!
Although I'd say traditional mnemonic devices like memory palaces are basically linear information storage and recall devices. This can create issues in building a flexible web of information, because loci or the order of the path can become dependencies and you can run out of unique spots in a given space, leading to memory interference.
Even spaced repetition methods (e.g. Anki) tend towards fragmentation of micro-ideas. Its perfect for terms, languages, and simple one question -> one answer ideas.
I've found a hybrid method of images, nested loci and spaced repetition to be most useful, because its flexible over time, and preserves relationships of ideas.
(Context: I co-founded a SaaS in this space: www.sticky.study)
Thanks for sharing your alternative. I like that you’ve included your references for each component of your method. That might help as many people as your product. I’ll look into it sometime.
I don't disagree with Einstein, but I wonder what he would say with the modern internet at his disposal. Maybe the same?
Elegance is probably orthogonal to creativity, and likely follows from some kind of minimization principle, like minimum program length. You are effectively distilling the "essence" of something from all of the noise.
Creativity seems different, more like novelty, and creativity following some kind of remix of memorized elements + some randomization seems very plausible.
You can create something novel but not elegant, and something elegant but not novel, and you can distill an elegant version of something novel that your or someone else created and that's the best of all creations.
But saying this to a programming crowd must be the most futile thing. At least instrumentalists have to rote train their muscle memory. That lowest bar has to be passed, even if it’s just three chords.
But the article isn’t about programming creativity though. It is a general concept. But if honing in on the mythical lone-genius activity (geniuses never practice in a structured way) helps you win an argument then so be it.
Of course it is. It's how every human child learns initially. By rote memorization. How does a toddler learn how to say mama? By constantly hearing and repeating it. How does a kid learn their ABCs? Rote memorization is the basis of all memory.
> Memorization in programming gives us architecture astronauts and design-pattern soup rather than elegant code.
Dumbest thing I've ever read. You write programs well by doing and remembering. Same with writing. Memorization is the necessary component to programming well. In other words, you program well by remembering elegant code.
> For me, at least, trying to memorize something without context
After the basics, most memorization is contextual.
> At the same time, memorization has a real cost: it takes time and it's frightfully dull.
Oh dear. Something isn't fun all the time. What a childish worldview. It's more fun to eat candy and drink soda than eating 'dull'. It's more fun to sit and watch youtube than to workout.
> Sometimes a bit of memorization is unavoidable, but I've found that to be relatively rare.
Relatively rare? In order to be competent in anything, you have to memorize lots. You can't write a good essay without having memorized much of the material. Trying reading a book where you have to constantly look up definitions of words because you lack the vocabulary. Try having a conversation with someone who has to constantly look up words because he lacks the vocabulary. Try having code review with someone who doesn't remember anything about their code.
> Otherwise, my time is generally better spent on some sort of practice in context.
Why? Because it helps you remember?
To the idiot ( probably OP ) who downvoted, try coding without having 'memorized' the keyboard. The anti-intellectual, anti-hard work, anti-memorization agenda pushed by some 'people' online bears looking into.
Well... can you think of an artist who didn't have a deep knowledge of their art-form before they pushed it forward? Three that jump out for me, in no particular order, are Picasso, Borges and Jack White. After all, great artists steal.
That being said, in some ways you could say that the splatter paintings he's known so well for are in fact influenced by all the art he studied and discarded along the way. They were definitely influenced by the principles of artistic design he learned, even if they looked different from what people were used to.
In my opinion, your hypothesis is supported, though maybe in a bit of a roundabout way.
Borges was incredibly talented, but it's worth keeping in mind his dad was a writer too.
Good art very much relies on being exposed to lots of other good art first. I don't know that rote memorization is the best way to achieve that, but you definitely need that exposure.
I don't think it's possible to have "good taste" without exposure to lots of examples, because I believe taste it culturally bound. Whether you do it explicity via a system, or on a more ad hoc basis, I think most artists need it.
It might be interesting to look at film, where the process is compressed into a couple of generations. I don't know it it will support my argument or not.
I don't think film will look very different here - early film work was very much informed by theatrical tastes at the time, and then started to diverge as people figured out what else they could say in the language of film.
Fundamentally, all art exists in a cultural context. If you've ever taken an art history course, you've been hit over the head with that info a few times ;) And that means furthering/changing taste in a given field means being aware enough of the existing rules to deliberately choose which ones you're breaking, and why.
There are some (very few) artists who didn't have a formal grounding, but I'd argue that even they were steeped enough in cultural context to be informed by it. Even famous autodidacts like Grandma Moses did develop a love for art based on being exposed to a bunch of it.
(Fully recognizing that it's a somewhat tautological argument because it's kind of impossible to grow up in a society without being somewhat exposed to its predominant art forms)
The main implication is that if what you are “memorizing” is not easily composable, then you won’t be able to apply them broadly or creatively.
However, I disagree with the author on what creativity is, although his definition is one experience of a creative inspiration.
That's the point I'm trying to make at least -- that unintentional creativity stems from learning, which at it's core is memorization
I could see that -- agreed that "creativity" can be too blunt of a word to use for all situations
Isn't learning which tools in the bucket fits the a problem best just memorizing a heuristic?
That's the point I'm attempting to make, that it's not blind memorization without context, but still memorization of a heuristic at its core
There is a lot of memory involved in being creative, but I think setting out to memorize things is a bad way to be creative. You have to practice being creative. In doing so, you will naturally remember a lot of stuff like what works, what doesn't, and most importantly which types of things you ought to memorize. For example if you're programming you will find it useful to remember the syntax of your languages. If you're writing you'll find it useful to remember styles and vocabulary. And so on...
I'm suggesting that it enables it, as it's hard to be creative when you're still trying to remember the basics of your art
But once things become autonomous -- you can focus on those higher-level explorations
But that creativity can come from many places and in many forms!
In a way, memorization is a severe risk: if you memorized something before it changed, for example, your creativity may not mean much.
What I was trying to convey is that fundamentally learning is memorization, whether conscious "rote memorization" or more less-intentional committing to memory from doing an activity
And that recognizing this allows us to speed up the process of learning fundamentals
Which in turn enables creativity as most people see it
"Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment."
Training an LLM on the entirety of knowledge at this dawn of humanity and, even if you give it literally infinite training time, it's never going to go anywhere. It's going to just continue making relatively simple recombinations of its training set until somebody gives it a new training set to remix. This remix-only nature is no different with modern knowledge, but simply extremely obfuscated because there's such a massive base of information, and nobody is aware of anything more than a minuscule fraction of it all.
---
As for the 'secret' of LLMs, I think it's largely that most language is extremely redundant. One thought or point naturally flows.... why do I complete the rest of this statement? You already know exactly what I'm going to say, right? And from that statement the rest of my argument will also mostly write itself. Yet we do write out the rest, which is kind of weird if you think about it. Anyhow the point is that by looking at language 'flow correlations' over huge samples, LLMs can reconstruct and remix arbitrarily long dialogue from even the shortest of initial inputs. And it usually sounds at least reasonable, except when it doesn't and we call it a hallucination, but it's quite a misnomer because the entire process is a hallucination.
Btw - although I see evidence of LLMs creating "new ideas" through combinations of ideas, I am a bit mystified by their apparent reasoning issues. I wonder how that is different in nature from the memory-based approach. ARC-AGI benchmark has had me thinking about this for sure.
The vast majority of what we consider "memories" are the creative brain doing an on-the-fly story generation, massaged until it "seems right" and serviced plus a big dollop of emotional confidence.
It’s just a bunch of arbitrary unprovable assertions.
Everyone here seems to have, broadly speaking; neither a) the qualifications to knowledgeably comment of the (honestly poorly understood, afaik) function of “creativity” or b) anything more meaningful than “here is my naive personal lived experience and opinion” to contribute on the topic.
It’s just armchair psychology.
If you want to wax philosophical, by all means, but I think anyone taking “thoughtful insight” away from this article or thread is fooling themselves.
It's strongly suspected that anterior forebrain pathway (AFP) may be a source of behavioral variability. We naturally age over time, including our vocal musculature, so in some sense we must constantly relearn how to use our muscles to deliver a song.
When a bird is deafened its birdsong will naturally drift, but when we precisely damage the AFP along with deafening we find that birdsong remains stable for a longer period of time, until of course inevitably it must drift due to aging vocal musculature.
You can't make connections unless you have things to connect to.
You can't recognize (your own discovered/inspired) novelty unless you have memorized normality.
If you are creative in the absence of knowledge of what already exists then that's considered as reinventing the wheel, and not very useful, even it it's Ramanujan reinventing much of established mathematics.
There's a bad trend, Hacker News gets this kind of blog-style self-promotion every month that gets much comment attention, but the essays are not well-researched and with made-up assertions written by programmers talking out of their lane ("Engineer's disease") and not having done homework on the subject.
How so? Can you share what you have read and that is relevant and applicable in real life?
> There's a bad trend, Hacker News gets this kind of blog-style self-promotion every month that gets much comment attention, but the essays are not well-researched and with made-up assertions written by programmers talking out of their lane ("Engineer's disease") and not having done homework on the subject.
You are on the wrong site, this is not a scientific journal there is no need for scientific rigor in every post and comment.
P.S. The real life world is full of events and things happening if you cannot learn by yourself (create theories models on how the world works aka. pseudoscience apparently) and need academic verification for everything. Then you some kind of disorder :/ good luck
There's absolutely an expectation of not rejecting science just as with discussing global warming or Covid.
But explain, why do you pervert an expectation of scientific literacy into an expectation of professional scientific expertise?
Just as there are scientifically informed books and articles on a e.g. health and fitness and dieting, there is a ton of material on education psychology. Why do you allow the author to not do due diligence and at least read the basics which are very much accessible to nonexperts? Why make an exception when in every other STEM topic this would be incredibly ignorant?
If it were a Time magazine article or a newspaper article they would surely include scientific sources in an effort to be truthful. Surely any high-school student knows the steps to write a well-researched essay, so why are you making this about the irrelevant standards of professional academia here? Could it be due to a misunderstanding of the role of science in public education? Are you some kind of Covid vaccine denier or anti-science on some other topic?
In summary, you have wrongly conflated scientific literacy, being scientifically informed, public scientific awareness on a topic, etc., with professional academic scientific expertise. That is irrelevant and absurd. It also offends me that you made a bad faith construal of my statement.
As to your remark about HN, my criticism applies to ANY scientifically informed public discourse, even as HN is a STEM website.
Finally: I'm a reader giving my time and attention to a badly written essay because it got posted on a forum. It is not my job to fix their essay if it is antiscience in the way an antivaxxer makes up pseudoscience reasons why their idea is right. I'm not going to patiently explain to the nth antivaxxer why they are spouting pseudoscience, I'm just going to say that bluntly. This endless handwringing that HN people have over self help and rote memorization is just more of that, it's on them to have read a few simple books and articles on the topic before choosing to write about it for the public.
PS. In "real life", people express sarcasm and maybe your 'disorder' (completely out of line of you to say that, BTW, and against site guidelines to use personal attacks of that sort at fellow users) was failing to detect a sarcastic comment if you actually, literally, thought I meant "the author needs to memorize scientific literature" in context of their argument about memorization. That would have nothing to do with the actual accusation of pseudoscience, where being scientifically informed on a topic is just a basic standard of discourse to avoid misinformation and snake oil promotion.
Not even saying I'm right or novel.
Just sharing a framework born from my observations that is actively working for me.
Honestly didn't even think this was that controversial, imo the best criticism on here is that my essay was "semi-obvious"
Without sarcasm, my point was/is not that you need to know all the theories or engage with academia or write a formal paper.
Rather, my point is that if you're going to write on a topic, at least know what the basic science has to say on it. You wouldn't write about mask wearing, or recycling plastic, etc., based purely on anecdotal evidence. You would include information from scientific sources and consensus.
Doing that raises the level of discourse and forces you, the author, to be responsible for not propagating misinformatiom and pseudoscience. Especially so in the area of self-help and learning psychology.
So again, my original comment while making a sarcastic remark about memorizing scientific literature was really about communicating your ideas in a scientifically literate way just as any lay person has a personal responsibility when discussing a topic, be it about technology or psychology or sociology, etc.
Simply sharing what works for me and why I think it does for something as personal as creativity is not the same as contesting mask wearing or recycling.
Is there some specific point in my piece that disagrees with consensus that's not just contesting the definition of "memorization" (as seems to be common in this thread)?
If you did not find anything interesting or anything that made you think, in the comments or the post itself. Then is your own failure.
Shame about the national stereotypes as well. There is plenty of creativity in Asian countries. Just bizarre assertions all around.
Not true of everyone, but if you compare a culture that values conformity and tradition to a country that values the freedom for the individual and trying new things, of course it's not going to measure up by western standards of creativity.
No, that's just the art that Western readers notice because that's the only thing they recognize as art in the first place. Calligraphy (in China traditionally considered the most important form of visual art) for example has an astonishing tradition in East Asia, also notably related to the topic of the thread, memorization and repetition and practice and has very little to do with critiques of norms.
Western entertainment tend to produce the same story, in like, two genres, again and again and again and again. Most of the time you can predict the movie storyline down to minute - and people will argue that it is the only correct way.
Just because you're loud and brash and write your inventions on your forehead doesn't mean you actually are more innovative or creative than anyone else. I know we've had bad comedians in the West who have made careers out of thinking that being loud equals being funny but you seem to have made an entire worldview out of that idea.
As for your comments about my taste, it's a good thing taste is entirely subjective, you can think I'm boorish and I can think you're boring and lack vision, and we're both entitled to our opinions.
The first two books are fantastic.
My creative writing professor, of all people, used to repeat this three times before every class. He was my favorite teacher at any level.
Bruce Lee
Not the most flattering of proverbs, but it stuck with me.
Spaced repetition allows me to become proficient even in things I don't get the natural opportunity to practise daily, so that when the day comes and I need them, I have some level of knowledge already. This has happened to Kubernetes troubleshooting, statistics, PowerShell windows programming, and traffic engineering just in recent history.
I have yet to publish some of these, but I have examples from statistics:
https://two-wrongs.com/intuition-and-spaced-repetition.html
https://two-wrongs.com/inventing-fishers-exact-test.html
The latter is certainly creative in my book, although it does imply creativity within strict bounds.
"It’s a little like building with lego bricks or something – spaced repetition helps ensure all the tiny pieces are in the right place, so that the big castle can happen without structural integrity issues."
The book Make it Stick (by Henry L. Roediger III) had a similar idea they called 'Structure Building'. Very similar to what you described, more experienced and effective learners were creating mental schemas of how the little, but crucial parts of a subject fit together, and successfully cut through the noise.
Structure Building was associated with interleaved practice (shuffling of problem types) and spaced retrieval practice.
The trick is not so much which software or settings one uses, but writing high-quality prompts.
Meorising the derivation makes it easier to derive a second-order derivation, and so on. At some level of abstraction, going from first principles becomes prohibitively expensive and caching intermediary results, or so to speak, unlocks that again.
My thought is: You're always practicing a behavior whether you like it or not.
Your mind is always setting you up to do more of whatever it is you're doing now. Both physical and mental. To do it more efficiently. With more ease. With more frequency. etc.
It's good motivation for me to mitigate a lot of negative behaviors. Angry in traffic, self anger, etc. If i can reason at least, of course. As i'm not interested in doing a lot of things more than i am, so i should avoid doing them now - if possible.
Would you mind sharing the flashcards you generated to build this intuition? I've been using Anki for a while and really trying to focus now on improving my prompt writing; would love to see how you managed it for this problem.
It's not that I set out to understand this specific thing but that I had studied statistics with flashcard support for a year and that happened to work after a few attempts.
When making flashcards I draw a lot from the softer type of theory-building they do in social sciences. I ask questions like
- What are the properties of this?
- What variants of this exist? I.e. how would I recognise this in the wild, or in other shapes?
- What subcomponents can this be deconstructed into?
- Into which bigger picture does this fit?
- What are the consequences of this? What are its antecedents?
- What is this a special case of? What would a generalisation of this look like?
- Which are other related things? What are their similarities and differences?
- In what context might I need to know this?
Whenever I encounter what seems like a significant thing I loosely ask some of these questions, and try to construct atomic, focused flashcards from the answers.
I say loosely because it would take forever to to through all questions for all flashcards I make, so there's some bit of intuition that attracts me to which I think are the most significant questions for any given thing.
-----
One trick to make flashcards more specific that I use (maybe even abuse) is putting part of the answer into the prompt. Instead of prompting "What is the property of subexponential distributions I found meaningful in this book?" I might prompt "What behaviour do subexponential distributions have around high barriers that others don't?" -- I'm giving away part of the answer by including "high barrier" in the prompt, but I'm okay with that.
If I'm concerned about that, I might create a second flashcard prompting something like "What can a subexponential distribution do in one step that a more well-behaved distribution needs many steps to do?" with the answer "clear a high barrier". That captures both sides of the property without making too general a prompt.
I also do this a lot with "why" questions. Instead of prompting "what is the definition of y?" I might prompt "why is the definition of y=f(x)?" That gives away essentially the entire answer but focuses on the why instead.
Understanding -> Remembering -> Applying
If you don't understand the basics of a concept, and you're talking about memory, its probably just rote memorization. Students generally find this tedious, and since it's shallow its very hard to retain and connect to disparate but parallel ideas from other fields (roots of creativity).
But, most schools and students stop there. They hear 'memory' or 'memorizing' as only rote memory. Step 2 is critical if you want to get to higher levels of learning. As you said in the essay - "Creativity comes to those who have internalized the patterns of their art". At www.sticky.study this is what we focus on. It's fast 2D memory palaces + spaced repetition.
Only if you have understanding + remembering can you get to step 3 - applying what you learn reliably at relevant moments in your life. This is the gold standard that schools claim they desire - analysis, synthesis, application, broad transfer, and creativity. You can't reach master efficiently if you lose 80% of all you read or learn.
1. Creativity in a commercial context once stolen/cloned through back-channels accrues value, and manifests as several competitive campaigns
2. New disruptive ideas are usually shelved until the IP/patents expire. No one wants to go through the sometimes impossible licensing process
3. Emerging technology is usually degraded in the rush for IP assets by established firms i.e. large firms dump billions on ridiculous concepts out of fear of market fragmentation
4. Startups do not usually have cash to burn on speculative IP. Thus, real cutting-edge experimental technology is sometimes never made public for numerous reasons.
Creativity:
https://www.youtube.com/watch?v=VUVix0STUqo
Best of luck, =3