The whole paradigm changes, though, when you can do daily cancer scans. You don't get a biopsy when the scan shows a lump. You get a biopsy after a couple weeks of daily scans showing the lump growing. Plus, having all…
GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5…
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as…
The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to…
Yes, with 4 bit quantization.
We could not do that inadvertently. To block even 1% of light using Starlink sized satellites (~30 m^2 with solar panels deployed) would require tens of billions of satellites. We could do it on purpose with huge…
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see…
It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
There will always be overhead, but that doesn't mean it will always be a huge amount of overhead. I believe the state of the art is 97% efficiency (https://www.osti.gov/biblio/1495980) which is better than a lot of…
I did my own calculations based on plotting loss on benchmarks compared to models with known parameters and training data, as well as using a quote from Sam Altman that said that GPT-4 would not use very many more…
I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more…
They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation: "Snapshot of gpt-4…
It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work: 1. Someone develops a procedure for training models with distributed computing…
I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to…
If you bought an 8xA100 machine for $140k you would have to run it continuously for over 10,000 hours (about 14 months) to train the 7B model. By that time the value of the A100s you bought would have depreciated…
When I checked yesterday I believe the signature said OpenAI CEO Sam Altman, so it was definitely a joke signature, not a case of two people having the same name.
The Reflexion paper (https://arxiv.org/abs/2303.11366) that came out recently shows how this kind of mistake might be overcome. Asking the model to think about the answer after it's generated a first draft greatly…
Theoretically it should be way less energy intensive as well, since there won't be an animal expending energy to live for months before slaughter. Nor will there be a need to grow feathers, bones, or blood that end up…
When you are speaking to a person, they have inner thoughts and outer actions/words. If a person sees a chess board they will either consciously or unconsciously evaluate all the legal moves available to them and then…
Alternatively: 1. Quickly reduce costs by increasing model and computation efficiency. 2. Massively reduce prices while still maintaining some gross margin. 3. Massively increase market size and take the vast majority…
It could be even smaller than a Chinchilla optimal model. The Chinchilla paper was about training the most capable models with the least training compute. If you are optimizing for capability and inference compute you…
We have, but it's not a single process. We can convert light to electricity quite cheaply and efficiently with solar PV panels and then use that electricity to electrolyze hydrogen from water and capture CO2 from air(or…
Vinyl chloride, when burned, can create poisonous byproducts such as phosgene and carbon monoxide. Vinyl chloride that leaks into the environment is a carcinogen that can cause damage decades into the future. It's a…
I think the biggest reason for Tesla's gross margins is that millions of people want EVs for various reasons (gas prices, environmental concerns, fun, status) and Tesla is one of the only companies making them in large…
Surprisingly it appears not to be too far off standard solar panel efficiencies. According to this source[0], five nines silicon (5N) is called Upgraded Mettalurgical-grade (UMG) silicon. According to this paper[1],…
The whole paradigm changes, though, when you can do daily cancer scans. You don't get a biopsy when the scan shows a lump. You get a biopsy after a couple weeks of daily scans showing the lump growing. Plus, having all…
GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5…
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as…
The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to…
Yes, with 4 bit quantization.
We could not do that inadvertently. To block even 1% of light using Starlink sized satellites (~30 m^2 with solar panels deployed) would require tens of billions of satellites. We could do it on purpose with huge…
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see…
It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
There will always be overhead, but that doesn't mean it will always be a huge amount of overhead. I believe the state of the art is 97% efficiency (https://www.osti.gov/biblio/1495980) which is better than a lot of…
I did my own calculations based on plotting loss on benchmarks compared to models with known parameters and training data, as well as using a quote from Sam Altman that said that GPT-4 would not use very many more…
I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more…
They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation: "Snapshot of gpt-4…
It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work: 1. Someone develops a procedure for training models with distributed computing…
I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to…
If you bought an 8xA100 machine for $140k you would have to run it continuously for over 10,000 hours (about 14 months) to train the 7B model. By that time the value of the A100s you bought would have depreciated…
When I checked yesterday I believe the signature said OpenAI CEO Sam Altman, so it was definitely a joke signature, not a case of two people having the same name.
The Reflexion paper (https://arxiv.org/abs/2303.11366) that came out recently shows how this kind of mistake might be overcome. Asking the model to think about the answer after it's generated a first draft greatly…
Theoretically it should be way less energy intensive as well, since there won't be an animal expending energy to live for months before slaughter. Nor will there be a need to grow feathers, bones, or blood that end up…
When you are speaking to a person, they have inner thoughts and outer actions/words. If a person sees a chess board they will either consciously or unconsciously evaluate all the legal moves available to them and then…
Alternatively: 1. Quickly reduce costs by increasing model and computation efficiency. 2. Massively reduce prices while still maintaining some gross margin. 3. Massively increase market size and take the vast majority…
It could be even smaller than a Chinchilla optimal model. The Chinchilla paper was about training the most capable models with the least training compute. If you are optimizing for capability and inference compute you…
We have, but it's not a single process. We can convert light to electricity quite cheaply and efficiently with solar PV panels and then use that electricity to electrolyze hydrogen from water and capture CO2 from air(or…
Vinyl chloride, when burned, can create poisonous byproducts such as phosgene and carbon monoxide. Vinyl chloride that leaks into the environment is a carcinogen that can cause damage decades into the future. It's a…
I think the biggest reason for Tesla's gross margins is that millions of people want EVs for various reasons (gas prices, environmental concerns, fun, status) and Tesla is one of the only companies making them in large…
Surprisingly it appears not to be too far off standard solar panel efficiencies. According to this source[0], five nines silicon (5N) is called Upgraded Mettalurgical-grade (UMG) silicon. According to this paper[1],…