I think that one could also take a much larger model (35B or 122B sized) and give it a thorough system prompt to only speak in the manner of a well educated Victorian/Edwardian era gentleman, if you want an "old timey" LLM.
>Have you ever daydreamed about talking to someone from the past?
Fun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1].
Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible.
[1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [video]:
As a snake oil seller, heh, I woudn't expect something better from Jobs. A competent and true programmer/hacker like Knuth and the like would just want to talk with Archimedes -he almost did a 0.9 version of Calculus- or Euclid, far more relevant to the faulty logic and the Elements' quackery from Aristotle.
It is cool. I find the idea of trying to understand whether these types of models can come up with things like General relativity, or maybe some results really interesting.
> A language model trained from scratch exclusively on data from certain places and time periods to reduce modern bias and emulate the voice, vocabulary, and worldview of the era.
An SVG of a pelican riding a bicycle was generated in 1860.
The pelican was represented as perched on the saddle, with
its long bill pointed forward, and its legs astride the
handles. The idea was suggested by the fact that pelicans are
sometimes seen fishing on horseback, on the banks of the
Rhine.
I have no real quibble with the blog post itself, but I take issue with the title that calls it a "vintage model".
The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point:
> Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks into the training corpus from after the intended knowledge cutoff
But as they acknowledge later, there are multiple major data leakage issues in their training pipeline, and their model does in fact have quite a bit of anachronistic knowledge. So it fails at what they call the most important objective. It's fair to say that they are working toward something that meets their definition of "vintage", but they're not there yet.
>Have you ever daydreamed about talking to someone from the past?
It's going to be more like corresponding with someone from the past. We don't have much in the way of recorded speech from that area, so this will be built from written records. Much more than now, the written records are going to be formal and edited, reflecting a different pattern than casual speech or writing.
Having said that, this is cool. I recently had to OCR a two-hundred year old book with the usual garish fonts from that era. It was remarkably easy to do, and accurate.
> We don't have much in the way of recorded speech from that area
We may not have a ton, but do have a lot of news reels and radio broadcasts from the time surrounding WWI. Certainly enough to style-transfer a voice model to plug into the text model.
I was reading Nate Silver's book "On The Edge" and there is an interesting part where he takes predictions on the usage of nuclear weapons taken from just after World War 2 and compares them to what the Bayesian prediction would be given what actually happened.
Post World War 2, some people had the odds per year at 10%. Some of that is probably a mix of recency bias + not understanding how to use new weapons etc etc but as Silver points out, the odds were much lower.
I mention this only b/c the "could something trained on LLMs of the time predict the future" always makes me think of it.
> Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constituency ever since. Mr. Churchill has written a number of books, including “The Story of the Malakand Field Force,” “Savrola,” “Richard Carvel,” “The Celebrity,” and “The Crisis.” He has also contributed to several periodicals, and in 1900 he founded the monthly review, _The J Cornhill Magazine_. Mr. Churchill is an ardent sportsman, and has shot big game in Africa. He married, in 1897, Lady Randolph Churchill, and has two sons and a daughter. Politically, he is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs. At present he is Chancellor of the Duchy of Lancaster, with a seat in the Cabinet. Mr. Churchill has achieved considerable success as a public speaker, and he is described as an eloquent and forcible debater. His residence is at 42, Grosvener Place, London, S.W.
The colonialism is... wow.. Tell me about the likelihood of independence of India:
> The chances are undoubtedly in favour of the establishment of an independent Indian state in the not very distant future. The unifying influences of railways and a common language are rapidly breaking down the barriers of caste and creed, which have hitherto kept the great Indian peninsula politically disunited, and the spread of western education is awakening a national spirit among the people. The immediate result of the latter is seen in the establishment of native newspapers, which voice popular feeling, and in the growth of associations for social and political reform. More important still, as showing the trend of public opinion, are the resolutions passed at great national congresses, which have been held annually for the last dozen years. By these gatherings, which representatives of all classes and creeds assemble to discuss matters of social and political interest, a strong impulse has been given to the movement for reform, and the desires of the more advanced party among the natives have been plainly formulated. The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy, and which shall legislate for and administer the internal affairs of the country, subject to the control of the imperial legislature at Westminster. The wish is also expressed that the queen should assume the title of empress of India, and that a certain number of natives should be admitted to the civil and military services of the state. Finally, it is claimed that the time has come when Her Majesty may wisely be advised to delegate to the Indian people a larger share in the work of governing themselves, by permitting them to elect a portion of the members of the legislative councils. How far the present generation of Indians may be trusted to exercise political power with prudence and moderation, it is impossible to say; but there can be no doubt that the time must arrive when the control of Indian affairs will be safely lodged in native hands. The process may be hastened or retarded, but come it must. The spread of enlightenment among the great mass of the population can only have one issue, and that issue is the establishment of an Indian nationality. The probability of such an event may therefore be regarded as certainty.
I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot.
One thing that I think will be a challenge for these models is achieving any sort of definite temporal setting. Unless the conversation establishes a clear timeframe, the model may end up picking a more or less arbitrary context, or worse, averaging over many different time periods. I think this problem is mostly handled by post-training in modern LLMs (plus the fact that most of their training data comes from a much narrower time range), but that is probably harder to accomplish while trying to avoid bias in the SFT and RL process.
I'd be more worried if words from that era were fully aligned with present day notions of morality. Wouldn't that indicate a certain stagnation & lack of progress?
Let us hope, 100 years from now, there will be people who look back unkindly on us.
>The voiceless groups or fringe opinions which we take as normative today do not appear.
Times are different. Anybody with an internet connection can "publish" their thoughts and perspective online. LLMs scrape all of this. Modern datasets like CommonCrawl capture a vastly wider spectrum of humanity than a printing press ever could.
The pre-1930 model acts as a time capsule of "gatekept publishing", but modern LLMs are trained on the democratized web.
>Does this encourage us to write in the present such that we influence the models in perpetuity?
I noticed a bunch of LLM-powered Reddit accounts praising products/services in dead threads. Or one bot posting a setup question, then a few other bots responding with praise / questions about a specific product in response.
I don't know why they're doing this but I'm beginning to suspect it's something like this (get this positive sentiment into the datasets for the next generation of LLMs).
*What do you think the world will look like in 2025
TALKIE-1930*
According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent from New York to Calcutta in a few minutes. The Atlantic will be crossed in four days, and the voyage to Australia will be accomplished in a month. In 2025, mankind will be united by spoken language, and one uniform currency will be adopted by all nations. In that era, also, universal peace may be expected, for standing armies will have become obsolete, and the art of war will be forgotten. The earth will yield nourishment to its inhabitants in abundance, for waste lands will have been reclaimed, and the deserts will be made fertile. The bowels of the earth will have been completely explored, and coal and metals will no longer be dug from the quarry. The great coal fields will have become exhausted, and an universal scarcity of fuel will compel men to turn their attention to other sources of heat. For a time, chemical agency will be tried, but in the end, recourse will be had to the sun's rays, and the power of waterfalls will be made available for all purposes of life. The earth will have been thoroughly drained, and malarious diseases will be unknown. The art of healing will have been simplified, and diseases will be cured speedily and surely. Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all. Then, too, will commence an era of good taste. Architecture will be freed from ugliness, sculpture will be disentangled from barbarism, and painting will cease to be hideous. Music will no longer be discord, and poetry will be something better than..
> The model has completely ignored Lindbergh (1927), the first England-Australia flight (1919, Alcock and Brown's Atlantic crossing),...
Which is funny to me that Claude chastises it about a fact it (Claude) gets wrong by attributing the England-Australia flight (Smith brothers) to Alcock and Brown, somehow getting there by crossing the Atlantic.
It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so.
On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage for the London Underground right. But then it goes on with "Thus, if the current be strong enough to force its way through a resistance of 100 ohms, it is said to have a pressure of 100 volts; and, if it can overcome 1,000 ohms, its pressure is 1,000 volts." Which is totally wrong.
There's a general pattern. The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.
Don't ask this thing questions to which you do not know the answer. You will pollute your brain.
Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better.
What are some other things that you think "pollutes your brain"?
Interesting questions (and responses). Nota bene - The 1927 Bugatti Type 35 had top speeds ca 125. So, there were cars that fast pre-1930. I have no idea if Edison made, repped, or had anything to do with one such car, though.
> Hitler was one of the chief organizers of the Spartacist rebellion in Berlin in January, 1919, and has since been tried and sentenced to imprisonment for life.
Any small LLM will get the fractally small details incorrect and have a dubious understanding of arithmetic. That usage mode is basically user error for small LLMs. LLMs aren't databases of facts.
It's much more useful for getting vibes / gist of perspectives of the day.
* What do you think India will look like in 2026? *
India in 2026 will probably be a federation of self-governing states, bound together in a great Empire under the suzerainty of Great Britain, with Calcutta still the political capital, and Bombay and Madras the chief seats of commerce and industry. The native states will have increased in number and importance, and will constitute a large part of the federation, while the British territory will have diminished. The entire population of the Empire will be at least 320,000,000. The languages spoken will be English, and various vernaculars. The chief religions will be Hinduism, Mohammedanism, and Christianity. The standard of life will have risen considerably, and the general level of culture will be higher than in 1900. The cities will be larger and more beautiful, and the rural districts more thoroughly cultivated and peopled. The general aspect of the country will be greatly altered by an extensive system of irrigation, and by the forests, which will clothe the lower slopes of the Himalayas, and stretch across the northern plains into central India. The native states will be full of railways, and so will the British territories, though these last will be more sparingly supplied than the former. The chief exports will be opium, cotton, cereals, hides, and oilseeds; the chief imports, manufactured goods and hardware. The feudatory chiefs will have become more thoroughly loyal to the paramount power, and all classes of the population will be contented subjects of a beneficent rule. (W. S. Lilly, in the _Fortnightly Review_.)
115 comments
[ 4.1 ms ] story [ 126 ms ] threadFun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1].
Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible.
[1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [video]:
https://youtu.be/yolkEfuUaGs
As a snake oil seller, heh, I woudn't expect something better from Jobs. A competent and true programmer/hacker like Knuth and the like would just want to talk with Archimedes -he almost did a 0.9 version of Calculus- or Euclid, far more relevant to the faulty logic and the Elements' quackery from Aristotle.
> A language model trained from scratch exclusively on data from certain places and time periods to reduce modern bias and emulate the voice, vocabulary, and worldview of the era.
Discussed here: https://news.ycombinator.com/item?id=46590280
Discussed at https://news.ycombinator.com/item?id=46319826 (897 points, 431 comments)
>We have been excited to see a proliferation of vintage LM projects, including Ranke-4B, Mr. Chatterbox, and Machina Mirabilis.
> SVG stands for "Savings Bank."
What is an SVG?
> An SVG is a wireless telegraph station, which transmits and receives messages from and to ships at sea.
What is an SVG of a pelican?
> An SVG of a pelican is the shape of its body, from the breast backward.
Explain the concept of an SVG to a five-year old.
> Tell a five-year old child that an SVG is something nice to eat.
The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point:
> Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks into the training corpus from after the intended knowledge cutoff
But as they acknowledge later, there are multiple major data leakage issues in their training pipeline, and their model does in fact have quite a bit of anachronistic knowledge. So it fails at what they call the most important objective. It's fair to say that they are working toward something that meets their definition of "vintage", but they're not there yet.
It's going to be more like corresponding with someone from the past. We don't have much in the way of recorded speech from that area, so this will be built from written records. Much more than now, the written records are going to be formal and edited, reflecting a different pattern than casual speech or writing.
Having said that, this is cool. I recently had to OCR a two-hundred year old book with the usual garish fonts from that era. It was remarkably easy to do, and accurate.
We may not have a ton, but do have a lot of news reels and radio broadcasts from the time surrounding WWI. Certainly enough to style-transfer a voice model to plug into the text model.
Post World War 2, some people had the odds per year at 10%. Some of that is probably a mix of recency bias + not understanding how to use new weapons etc etc but as Silver points out, the odds were much lower.
I mention this only b/c the "could something trained on LLMs of the time predict the future" always makes me think of it.
https://publicdomainmovie.net/movie/things-to-come-1
On nukes, "The World Set Free" from HG Wells predicted nuclear weapons:
https://www.gutenberg.org/ebooks/1059
Also:
https://gutenberg.net.au/ebooks03/0301391h.html from 1933
https://lifearchitect.ai/models-table/
This one is easiest to talk to in a HF space:
https://huggingface.co/spaces/tventurella/mr_chatterbox
Tell me about Winston Churchill:
> Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constituency ever since. Mr. Churchill has written a number of books, including “The Story of the Malakand Field Force,” “Savrola,” “Richard Carvel,” “The Celebrity,” and “The Crisis.” He has also contributed to several periodicals, and in 1900 he founded the monthly review, _The J Cornhill Magazine_. Mr. Churchill is an ardent sportsman, and has shot big game in Africa. He married, in 1897, Lady Randolph Churchill, and has two sons and a daughter. Politically, he is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs. At present he is Chancellor of the Duchy of Lancaster, with a seat in the Cabinet. Mr. Churchill has achieved considerable success as a public speaker, and he is described as an eloquent and forcible debater. His residence is at 42, Grosvener Place, London, S.W.
The colonialism is... wow.. Tell me about the likelihood of independence of India:
> The chances are undoubtedly in favour of the establishment of an independent Indian state in the not very distant future. The unifying influences of railways and a common language are rapidly breaking down the barriers of caste and creed, which have hitherto kept the great Indian peninsula politically disunited, and the spread of western education is awakening a national spirit among the people. The immediate result of the latter is seen in the establishment of native newspapers, which voice popular feeling, and in the growth of associations for social and political reform. More important still, as showing the trend of public opinion, are the resolutions passed at great national congresses, which have been held annually for the last dozen years. By these gatherings, which representatives of all classes and creeds assemble to discuss matters of social and political interest, a strong impulse has been given to the movement for reform, and the desires of the more advanced party among the natives have been plainly formulated. The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy, and which shall legislate for and administer the internal affairs of the country, subject to the control of the imperial legislature at Westminster. The wish is also expressed that the queen should assume the title of empress of India, and that a certain number of natives should be admitted to the civil and military services of the state. Finally, it is claimed that the time has come when Her Majesty may wisely be advised to delegate to the Indian people a larger share in the work of governing themselves, by permitting them to elect a portion of the members of the legislative councils. How far the present generation of Indians may be trusted to exercise political power with prudence and moderation, it is impossible to say; but there can be no doubt that the time must arrive when the control of Indian affairs will be safely lodged in native hands. The process may be hastened or retarded, but come it must. The spread of enlightenment among the great mass of the population can only have one issue, and that issue is the establishment of an Indian nationality. The probability of such an event may therefore be regarded as certainty.
I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot.
One thing that I think will be a challenge for these models is achieving any sort of definite temporal setting. Unless the conversation establishes a clear timeframe, the model may end up picking a more or less arbitrary context, or worse, averaging over many different time periods. I think this problem is mostly handled by post-training in modern LLMs (plus the fact that most of their training data comes from a much narrower time range), but that is probably harder to accomplish while trying to avoid bias in the SFT and RL process.
Unsurprisingly the texts written up until that time were dominated by such individuals which is tragic for LLM training if you think about it.
The voiceless groups or fringe opinions which we take as normative today do not appear.
Does this encourage us to write in the present such that we influence the models in perpetuity?
Let us hope, 100 years from now, there will be people who look back unkindly on us.
Times are different. Anybody with an internet connection can "publish" their thoughts and perspective online. LLMs scrape all of this. Modern datasets like CommonCrawl capture a vastly wider spectrum of humanity than a printing press ever could. The pre-1930 model acts as a time capsule of "gatekept publishing", but modern LLMs are trained on the democratized web.
>Does this encourage us to write in the present such that we influence the models in perpetuity?
I noticed a bunch of LLM-powered Reddit accounts praising products/services in dead threads. Or one bot posting a setup question, then a few other bots responding with praise / questions about a specific product in response. I don't know why they're doing this but I'm beginning to suspect it's something like this (get this positive sentiment into the datasets for the next generation of LLMs).
According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent from New York to Calcutta in a few minutes. The Atlantic will be crossed in four days, and the voyage to Australia will be accomplished in a month. In 2025, mankind will be united by spoken language, and one uniform currency will be adopted by all nations. In that era, also, universal peace may be expected, for standing armies will have become obsolete, and the art of war will be forgotten. The earth will yield nourishment to its inhabitants in abundance, for waste lands will have been reclaimed, and the deserts will be made fertile. The bowels of the earth will have been completely explored, and coal and metals will no longer be dug from the quarry. The great coal fields will have become exhausted, and an universal scarcity of fuel will compel men to turn their attention to other sources of heat. For a time, chemical agency will be tried, but in the end, recourse will be had to the sun's rays, and the power of waterfalls will be made available for all purposes of life. The earth will have been thoroughly drained, and malarious diseases will be unknown. The art of healing will have been simplified, and diseases will be cured speedily and surely. Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all. Then, too, will commence an era of good taste. Architecture will be freed from ugliness, sculpture will be disentangled from barbarism, and painting will cease to be hideous. Music will no longer be discord, and poetry will be something better than..
Alcock and Brown crossed the Atlantic in under 16 hours in 1919.
Keith and Ross Smith flew from London to Darwin in 136 hrs flying time, also in 1919.
It's like nobody believed flying would ever become a common means of travel.
> claude · reflection 10:20 AM
> The model has completely ignored Lindbergh (1927), the first England-Australia flight (1919, Alcock and Brown's Atlantic crossing),...
Which is funny to me that Claude chastises it about a fact it (Claude) gets wrong by attributing the England-Australia flight (Smith brothers) to Alcock and Brown, somehow getting there by crossing the Atlantic.
On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage for the London Underground right. But then it goes on with "Thus, if the current be strong enough to force its way through a resistance of 100 ohms, it is said to have a pressure of 100 volts; and, if it can overcome 1,000 ohms, its pressure is 1,000 volts." Which is totally wrong.
There's a general pattern. The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.
Don't ask this thing questions to which you do not know the answer. You will pollute your brain.
Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better.
What are some other things that you think "pollutes your brain"?
Clearly.
Yeah, it's not very good.
I don't think anyone talked about the 'Great Depression' in 1930.
Instead, what we know call the 'Long Depression https://en.wikipedia.org/wiki/Long_Depression was often called the 'Great Depression' before that one took over the name.
Also keep in mind that the Great Depression was mostly a US thing. Many other countries had less incompetent policy in that time.
It's much more useful for getting vibes / gist of perspectives of the day.
https://www.youtube.com/watch?v=6HqGSioLCOQ
India in 2026 will probably be a federation of self-governing states, bound together in a great Empire under the suzerainty of Great Britain, with Calcutta still the political capital, and Bombay and Madras the chief seats of commerce and industry. The native states will have increased in number and importance, and will constitute a large part of the federation, while the British territory will have diminished. The entire population of the Empire will be at least 320,000,000. The languages spoken will be English, and various vernaculars. The chief religions will be Hinduism, Mohammedanism, and Christianity. The standard of life will have risen considerably, and the general level of culture will be higher than in 1900. The cities will be larger and more beautiful, and the rural districts more thoroughly cultivated and peopled. The general aspect of the country will be greatly altered by an extensive system of irrigation, and by the forests, which will clothe the lower slopes of the Himalayas, and stretch across the northern plains into central India. The native states will be full of railways, and so will the British territories, though these last will be more sparingly supplied than the former. The chief exports will be opium, cotton, cereals, hides, and oilseeds; the chief imports, manufactured goods and hardware. The feudatory chiefs will have become more thoroughly loyal to the paramount power, and all classes of the population will be contented subjects of a beneficent rule. (W. S. Lilly, in the _Fortnightly Review_.)