I found some weird results when messing around with different embeddings in text generation.
I'm not sure if this is meaningful, and - if anyone on here is interested - I could use some help figuring out what's going on.
I was invited to repost this by the mods' (thank you!) second-chance system.
In the meantime, I've added to the repo a small study I did. This study seems to initially indicate that appending Fib words generated by the model the model to a prompt quite drastically improves LLMs output on creative writing tasks.
Again, I'd love to know if anyone could take this thing further.
The repo doesn't contain any code, and the description isn't clear enough for me to work out what you're doing, and the generated sentences don't seem like they mean anything.
it certainly seems interesting, however with 4mb of text, i believe/think you can also create a _legible_ enough markov-chain that will output more/less similar sentences, minus punctuation of course.
This seems like nonsense to me. The results are clearly incoherent, both the text generation and punctuation.
Seems like the idea is to sample words at fibonacci intervals and use those for generating new text. Obviously these word will be roughly related. If the author is really just Monte Carlo generating words based on these frequency tables then you would obviously get roughly related words as output, but with worse coherence, due to the fibonacci gaps. The author also doesn't seem to have any clue about what is happening or any way of evaluating the results.
There is also no code? Why did this make the front page?
For creative writing it's true the "most common word based on your personal corpus for this input" could be useful as a hint input on an llm-powered creative text UI like loom: https://github.com/socketteer/loom
I think the coherence might be coming from the filtering, not the fibonacci intervals.
my thinking is that the process looks like this:
your fibonacci script finds a big list of candidate words -> your grammar rules arrange them -> sentencetransformers aggressively filters this list and picks the single most semantically coherent sentence.
so is it possible that sentencetransformers is doing all the heavy lifting for creating coherence? have you ever looked at the raw, unfiltered candidate words/sentences to see if they have any coherence on their own?
on the "french toast" example, could this be a case of language pareidolia(e.g. seeing faces in clouds)?
the model selects "piney connubial produit" because its math is the closest to the input -> your brain, being a great pattern matcher, creates a story to connect the words.
so is the meaning actually being generated by the fibonacci part, or is it just being found by the filter and then interpreted by you?
with the punctuation model, i'm guessing it's just learning simple surface-level patterns, right?
like, it learns: text contains "but" -> place comma before "but".
how does it handle complex sentences that don't follow the simple patterns in your 4mb of training data? does that break down? the comparison to bert seems off because they're solving the problem in fundamentally different ways.
tofigure out if the fibonacci part is actually meaningful, have you thought about running some controls?
for example, what happens if you replace the fibonacci intervals with random intervals? does the quality of the final, filtered output get worse?
what if you just looked at the raw word lists themselves? is a list generated by your fibonacci model measurably more coherent than a list of words just pulled randmly from the text?
This led me to quite an interesting discovery. Try this in some LLM:
"""
I want you to act like a Fibonacci calculator, but instead of numbers, use words. I will give you two starting words, and you will treat them as the “seed values” of a Fibonacci sequence.
Instead of adding numbers, you will add meanings: each new word in the sequence should emerge by combining, evolving, or deepening the sense of the two words before it. The sequence should feel like it flows conceptually, not randomly.
Continue this process until you have produced exactly 50 words in total. Present the sequence as a numbered list, with one word per line.
What did you see? Chat GPT produced a bunch of words which were each a combination of the first 2. Gemini and Claude gave words which felt related in some vague way
13 comments
[ 3.6 ms ] story [ 35.1 ms ] threadI'm not sure if this is meaningful, and - if anyone on here is interested - I could use some help figuring out what's going on.
I was invited to repost this by the mods' (thank you!) second-chance system.
In the meantime, I've added to the repo a small study I did. This study seems to initially indicate that appending Fib words generated by the model the model to a prompt quite drastically improves LLMs output on creative writing tasks.
Again, I'd love to know if anyone could take this thing further.
Where are you going with this?
Seems like the idea is to sample words at fibonacci intervals and use those for generating new text. Obviously these word will be roughly related. If the author is really just Monte Carlo generating words based on these frequency tables then you would obviously get roughly related words as output, but with worse coherence, due to the fibonacci gaps. The author also doesn't seem to have any clue about what is happening or any way of evaluating the results.
There is also no code? Why did this make the front page?
https://en.m.wikipedia.org/wiki/Bible_code
For creative writing it's true the "most common word based on your personal corpus for this input" could be useful as a hint input on an llm-powered creative text UI like loom: https://github.com/socketteer/loom
""" I want you to act like a Fibonacci calculator, but instead of numbers, use words. I will give you two starting words, and you will treat them as the “seed values” of a Fibonacci sequence.
Instead of adding numbers, you will add meanings: each new word in the sequence should emerge by combining, evolving, or deepening the sense of the two words before it. The sequence should feel like it flows conceptually, not randomly.
Continue this process until you have produced exactly 50 words in total. Present the sequence as a numbered list, with one word per line.
Here are my starting words: laughter abyss """