The system that they describe, Codefusion is interesting because it's a diffusion model for generating code rather than an autoregressive model like most LLM code generators.
ding ding ding ding !! prizes for the spectator who correctly names the Tip of the Iceberg for AI-assisted-communications at every level. You have heard of evolution? creativity ? Free-Will ? now, with AI assisted communication technology, every road leads back to MSFT ! So you wondered about Proust, or that line in Hamlet? surprise - it is only the entry topic to get back to a clever rhyming ad-jingle for your next software subscription update. btw- our agent is authorized to check your bank account balances while you renew. according to that user agreement you acknowledged.
I know you're being tongue in cheek but this is why the evolution of open source models capable of cheaply running on your own hardware (i.e. the r/locallamma folks) is so important.
I just want the 100k context window Anthropic gave Claude, but for GPT. Claude likes to hallucinate when I ask it to build chapter notes for a class I’m taking, and I don’t want to have to break up the text into tiny bits for GPT…
The large context is the reason its hallucinating and gpt would likely have the same problem. You can see this effect even on gpt-3.5-turbo-16k. Performance and accuracy are directly correlated with context size
The lost-in-the-middle phenomenon makes me _not_ want larger context windows because the model cannot effectively use it. It seems like we need another arch to process large ctx size.
That 1 trillion number was quickly confirmed to be false.
It's believed that GPT4 is a Mixture of Experts model -- it's 8 smaller models (estimated around 200B each) and a router picks the best model to redirect to based on input.
This was expected because open source models of the same size already beat GPT-3.5 in many ways. And Mistral 7B makes you think if huge parameters are even needed for something like GPT-3.5 level.
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
This seems like a valid exception from this rule. The interesting tidbit here is what the title suggests, even though it is not the title of the paper. It should be possible to point out to facts like these hidden in papers, reports, etc without having to create a separate article about it. You can of course add a comment as you suggested, but that does not reflect how people use HN.
25 comments
[ 3.3 ms ] story [ 38.0 ms ] threadThe system that they describe, Codefusion is interesting because it's a diffusion model for generating code rather than an autoregressive model like most LLM code generators.
> We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules.
Microsoft Excel conditional formatting, really? The future we want is one in which that abomination has died out, not one in which it is thriving.
/fiction
Relevant paper: Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172
It's believed that GPT4 is a Mixture of Experts model -- it's 8 smaller models (estimated around 200B each) and a router picks the best model to redirect to based on input.
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...