GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5…
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as…
The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to…
Yes, with 4 bit quantization.
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see…
It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
I did my own calculations based on plotting loss on benchmarks compared to models with known parameters and training data, as well as using a quote from Sam Altman that said that GPT-4 would not use very many more…
I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more…
They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation: "Snapshot of gpt-4…
It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work: 1. Someone develops a procedure for training models with distributed computing…
I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to…
When I checked yesterday I believe the signature said OpenAI CEO Sam Altman, so it was definitely a joke signature, not a case of two people having the same name.
The Reflexion paper (https://arxiv.org/abs/2303.11366) that came out recently shows how this kind of mistake might be overcome. Asking the model to think about the answer after it's generated a first draft greatly…
Theoretically it should be way less energy intensive as well, since there won't be an animal expending energy to live for months before slaughter. Nor will there be a need to grow feathers, bones, or blood that end up…
Alternatively: 1. Quickly reduce costs by increasing model and computation efficiency. 2. Massively reduce prices while still maintaining some gross margin. 3. Massively increase market size and take the vast majority…
It could be even smaller than a Chinchilla optimal model. The Chinchilla paper was about training the most capable models with the least training compute. If you are optimizing for capability and inference compute you…
We have, but it's not a single process. We can convert light to electricity quite cheaply and efficiently with solar PV panels and then use that electricity to electrolyze hydrogen from water and capture CO2 from air(or…
Vinyl chloride, when burned, can create poisonous byproducts such as phosgene and carbon monoxide. Vinyl chloride that leaks into the environment is a carcinogen that can cause damage decades into the future. It's a…
I think the biggest reason for Tesla's gross margins is that millions of people want EVs for various reasons (gas prices, environmental concerns, fun, status) and Tesla is one of the only companies making them in large…
Surprisingly it appears not to be too far off standard solar panel efficiencies. According to this source[0], five nines silicon (5N) is called Upgraded Mettalurgical-grade (UMG) silicon. According to this paper[1],…
Unfortunately I think you're off by an order of magnitude. I think it would be 810 Kilojoules, which is approximately equivalent to a 1kg lithium-ion battery. Of course, you could move thousands of rocks up and down a…
Density in tilted installations is quite bad. If you want to capture morning and evening sun at an optimal angle you have to space the panels out a lot, like 5-10 panel heights. You can have them closer, but then you…
Based on my calculations, at my latitude (40 degrees North), you would need about 16% more panels to generate an equivalent amount of energy per year. This isn't taking into account potential issues with snow buildup…
There is a small loss of efficiency (<2%) at extreme angles, but it is not too significant[1] [1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6611928/#:~:tex....
Another thing to keep in mind with solar is that so far we have been optimizing primarily for cost, rather than land use. This means installing panels at a high angle, far apart from each other to minimize shading and…
GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5…
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as…
The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to…
Yes, with 4 bit quantization.
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see…
It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
I did my own calculations based on plotting loss on benchmarks compared to models with known parameters and training data, as well as using a quote from Sam Altman that said that GPT-4 would not use very many more…
I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more…
They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation: "Snapshot of gpt-4…
It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work: 1. Someone develops a procedure for training models with distributed computing…
I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to…
When I checked yesterday I believe the signature said OpenAI CEO Sam Altman, so it was definitely a joke signature, not a case of two people having the same name.
The Reflexion paper (https://arxiv.org/abs/2303.11366) that came out recently shows how this kind of mistake might be overcome. Asking the model to think about the answer after it's generated a first draft greatly…
Theoretically it should be way less energy intensive as well, since there won't be an animal expending energy to live for months before slaughter. Nor will there be a need to grow feathers, bones, or blood that end up…
Alternatively: 1. Quickly reduce costs by increasing model and computation efficiency. 2. Massively reduce prices while still maintaining some gross margin. 3. Massively increase market size and take the vast majority…
It could be even smaller than a Chinchilla optimal model. The Chinchilla paper was about training the most capable models with the least training compute. If you are optimizing for capability and inference compute you…
We have, but it's not a single process. We can convert light to electricity quite cheaply and efficiently with solar PV panels and then use that electricity to electrolyze hydrogen from water and capture CO2 from air(or…
Vinyl chloride, when burned, can create poisonous byproducts such as phosgene and carbon monoxide. Vinyl chloride that leaks into the environment is a carcinogen that can cause damage decades into the future. It's a…
I think the biggest reason for Tesla's gross margins is that millions of people want EVs for various reasons (gas prices, environmental concerns, fun, status) and Tesla is one of the only companies making them in large…
Surprisingly it appears not to be too far off standard solar panel efficiencies. According to this source[0], five nines silicon (5N) is called Upgraded Mettalurgical-grade (UMG) silicon. According to this paper[1],…
Unfortunately I think you're off by an order of magnitude. I think it would be 810 Kilojoules, which is approximately equivalent to a 1kg lithium-ion battery. Of course, you could move thousands of rocks up and down a…
Density in tilted installations is quite bad. If you want to capture morning and evening sun at an optimal angle you have to space the panels out a lot, like 5-10 panel heights. You can have them closer, but then you…
Based on my calculations, at my latitude (40 degrees North), you would need about 16% more panels to generate an equivalent amount of energy per year. This isn't taking into account potential issues with snow buildup…
There is a small loss of efficiency (<2%) at extreme angles, but it is not too significant[1] [1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6611928/#:~:tex....
Another thing to keep in mind with solar is that so far we have been optimizing primarily for cost, rather than land use. This means installing panels at a high angle, far apart from each other to minimize shading and…