They did denied selling older generation, so I would expect TPU 3 is not going to be available as well. The only way to use those is to use Google Cloud.
No, they're going to be used by Google and nobody else, so they can establish dominance in the cloud AI/AI SaaS space using their immense resources. This will be used by internal Google projects to rapidly develop their large scale models and, if you're lucky, available on Google Cloud one day (TPUv2 already is, at least).
Google doesn't need "developer buy in" for these to make sense -- they need better hardware for training and deploying deep learning models for their products, which is the overwhelming motivation. And, if they offer these to you, it's only because you're willing to pay for faster TensorFlow iteration.
Models are the new bitcoins and no one is willing to sell the hardware directly. Weirdly enough exact same thing happened in mining (specialized hardware passed nvidia gpus in performance at some point).
Nvidia is on their 5th generation of datacenter ASIC. Google is on their third. Nvidia has a faster interconnect than anyone else (nvswitch), which allows close to the same access bandwidth to HBM on a remote GPU than a local GPU. If Google has that, they haven't announced it, and all I can see is they operate over PCIe v3 since they are slaves to the CPU manufacturers. If the future was cloud providers using their own silicon, Google would have made a CPU to compete with Intel. Instead, they use Intel exclusively and are evaluating POWER. They made a single ASIC that is very simple relatively speaking, that can perform a single task and cannot be accessed generically. They are not 2 generations ahead. They chose a path that's much easier for many reasons, but mostly because it allows their own internal training to be accelerated.
All comes down to the cost. We can see that the TPUs were about 1/2 as much as using Nvidia before the TPU 3.0. This is based on doing the same task with the TPUs versus Nvidia with AWS. But it might be a bigger difference and Google taking margins. Could have been the other way but think WaveNet proves not that direction.
Would hope we get an even bigger price difference with this generation.
BTW, you are mixing up graphic generations and ML generations. Google never did or will graphics.
For ML Google is 2 generations ahead. Hard to imagine Nvidia ever catching up as so much of the AI breakthroughs are coming from Google.
A perfect example is WaveNet. Google is using a NN for audio in real time at 16k cycles a second. That is competing with using the old way of doing this. The Google approach gets a better result but people are only willing to pay too much more for a better result.
So Google had to price WaveNet competitively to the old approach which they have. I suspect that would have been impossible using Nvidia. The cost in running would have been just too high. The computation required for WaveNet is huge and the old way it is minimal.
Also we are going to need to really separate training with inference. They have different requirements and the costs in running are different.
Has Nvidia done an inference only solution? Or do they still always mix them?
"Google would have made a CPU to compete with Intel. "
I do a lot of surfing and have to say one of the more silly things to read. There is little gains today with CPUs. Why on earth would you invest in doing your own?
Google has ported all their stuff to Power so they have a backup and not reliant on Intel. But CPUs are NOT the future and would be crazy to invest into them at this point.
Processing is moving to TPU type processors and we will see more and more traditional things. A fantastic paper on this from Jeff Dean at Google I suggest you read.
One thing I love about Google is they do NOT reinvent the wheel. If there is something that can work like the Linux kernel they use. Versus having an ego of having to do yourself.
They did all their own network silicon by hiring the Lanai team several years ago because there was nothing that could work for their needs. They then created a determinate network stack so they could do Spanner. There was nothing off the shelve. Heck they laid their own fiber under the ocean to make possible.
Spanner is the first and only horizontally scalable RDBMS solution to exist. Google beat the speed of light by using their custom silicon, custom stack that is determinate, their own fiber. Then using atomic clocks with GPS to remove the latency from speed of light out of the equation. Highly recommend the papers including the Spanner paper.
The network uses custom silicon inside and then they put the intelligence on out edge. Then they control exactly the traffic so they get a determinate result. This also allows them to use far cheaper hardware as do not need to over provision as they never drop packets on the ground or need much memory for buffers.
Google would have used Nvidia in a second if they could do what they need.
But what is being done in AI (ML) today requires unique silicon. Just not possible without as the power footprint would be prohibitive.
I do expect Google more and more to leverage on client ML where needed with their PVC. So it is NOT just the TPUs but for somethings you are going to need on client.
Take the new voices as an example. They are doing 6 voices as each has a different model. The cost in switching models is prohibitive in the cloud.
So you will see them eventually move to some things being done on the client. ...
This is great news. It’s extremely important to the research community that large companies enter into the DL silicon space to contend with NVIDIA’s monopoly.
NVIDIA is now exerting pricing power to the point where they’ve decided to train their sales people to disregard the metric that is most important to customers: cost of training. Talk with one of their enterprise sales people and you’ll find they’ll say things like “FLOPS / $ doesn’t matter” to justify a 10x increase in price for their TESLA line. As history has shown, a monopolist sows the seeds of their own destruction and, by disregarding the metrics that matter, they alienate their customers.
Here are open source projects that you can contribute to to break the monopoly:
Well until that actually happens I will be buying more NVIDIA stock.
I have done deep learning also and I don't see NVIDIA being an incumbent that is easy to be knocked out. I think we would need some distruption such as a new technique that is more performant or different design goals. Intel is still leading the way and I don't see NVIDIA being unseated with its head start anytime soon.
I'm not as pessimistic. 3dfx and Matrox didn't look like easy to knock out opponents in the 90s and yet here we are. Markets are powerful incentive mechanisms.
"Slowly grinds the mill of the gods, but it grinds fine."
Costs to compete in the silicon business have inflated massively since that era, its perhaps 100 or even 1000 times more expensive to build stuff like fabs (although NVIDIA dosent own fabs), and design chips.
>3dfx and Matrox didn't look like easy to knock out opponents in the 90s and yet here we are.
Matrox didn't give much of a damn to 3D gaming, 3Dfx were bust doing all the wrong things. It was certainly not "easy", but don't seems hard or impossible either at the time.
We're working to provide more choice with PlaidML as well. You can use it to run Keras, ONNX, and TF models on AMD, Intel, and mobile/embedded GPUs, on most operating systems and OpenCL, LLVM, OpenGL, and soon CUDA and Metal, etc. We would welcome contributions for TensorFlow integration, ROCm support, and anything else that widens support to more people.
A couple people internally have looked at Vulkan, it seems like it should work really well but we're not actively working on it. You can get an idea of how Vulkan support would look by looking at the existing HALs:
Choice in the marketplace is still choice. And, Google might eventually decide to sell this to outside companies. Eventually the desire to build competitive advantages for Google Cloud might be surpassed by the desire to compete with NVIDIA. Commoditizing the complement is a powerful strategy.
TPUs seem like great hardware. Unfortunately google's insistence on keeping them in a walled garden is a major deterrent to building something that relies on their platform.
You seem to be missing the point again and again every time this topic comes up. There is a reason why TensorFlow is generic enough to run on many types of hardware. That allows people to port their code around. But there exists a world outside of tensorflow that you might not be aware of, and for that world this chip is not a competitor. That world is significantly larger than the TF world.
Written in Nim, which looks and feels a lot like Python, but runs a lot like C++ or Rust, especially in the speed and FFI departments, and deploys like Go as a small single fast executable. Already supports CPU, CUDA and OpenCL. Still early, but with a lot of promise. Use is modeled on some mixture of PyTorch and Keras.
Self-plug: Gorgonia[0] can support OpenCL as well (but time and effort is required to write support libs). Would appreciate contributions. I recently gave a talk on how to interface with foreign libs[1]. Writeup soon to come.
Water cooling on their TPU is probably because they cost way more to buy than the extra cost of special cooling. It is also probably to increase “spec” on their new version without the best improvements in hardware/silicon.
What will be interesting to see is if they are going the hardware specialization route, like Nvidia with their support for efficient 4x4 fused FP16 matmuls with FP32 matrix output (they call them 'tensor cores' - hah).
I suspect with the liquid cooling, they are just dialing up the matmul speed, which is probably the right way to go, IMO.
We are doing data-center experiments - using oil cooling. It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.
Yes, it's not my research - a colleagues. He has some big vats of oil. It's a sureal experience to see servers just being oil-boarded (can i say that? :) ) and coming out ok!
I'd love to discuss your research. We've been working low-profile during the last years doing a lot of R&D and experimenting with different fluids and components.
Mineral oil is ok for experimentation, but for long term material compatibility and fire risk I wouldn't recommend it.
FWIW I co-founded https://submer.com where we've developed an all-embedded computing immersion cooling solution that is virtually compatible with any kind of hardware (even fiber optics) and it's orders of magnitude more efficient than traditional data center cooling technologies.
> Google CEO Sundar Pichai said the new TPU is eight times more powerful than last year
Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself.
And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs.
So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive...
Google started playing this game with the tpuv2, wherein they define a "TPU" as whatever they want to make it sound suitably impressive (I.e. Cloud TPU is four chips). This in turn led to nvidia calling the DGX-2 "the world's largest GPU
What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic.
How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7.
So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
The thing is ML for training is much more bumpy. You need further capacity at certain times.
But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware.
The other is buying the hardware is much harder from a cash flow standpoint.
So you completely ignored my questions, and simply repeated your previous comment word for word?
You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense?
Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
I do not know your personal use case. But if buying something makes sense then I would do it.
But the future is the cloud. I have fought people on this for well over a decade and thought everyone by now got it. Apparently not.
On TPU being a miracle chip not clear to me what a "miracle" chip would even be? Plus things keep improving and how could you have a static "miracle" chip?
What we do know is Google is doing WaveNet in production in real-time. We can hear the results. They are offering at a competitive price to traditional TTS.
We can also see the TPUs are about 1/2 the cost of using Nvidia.
Otherwise that is it. But I do not know your faith but personally I would not consider that a "miracle". But maybe just me.
Maybe Lewandowski would as I have heard he prays to a AI god or something like that.
The big difference is Google is working top down and Nvidia has to work bottom up.
So Google wants to roll out the best TTS there is to the world and at scale.
So they are going to from the top down to be able to achieve.
Clearly Google has a goal of creating the singularity and the Silicon is just a piece of the equation.
Versus I am not really sure what Nvidia goal is. Silicon is not for silicon sake. But rather a means to an end.
Plus Google having in production has the data to optimize in future versions where Nvidia just is not going to be able to.
I am long Nvidia and owned it for a bit. Was disappointed on how the stock traded after ER. Do think it will do fine as Google does all these incredible AI things. It is an alternative that will be good for them.
But for the very long term I would not hold. Google is a hold forever. Nvidia would keep on it.
BTW, would say Nvidia is now two generations behind.
Clearly Google has a goal of creating the singularity
Oh wow. Didn't realize you're one of those. I used to be a Kurzweil fan too, when I was 18. That phase usually passes by senior year in college. Well, I guess some people need longer to see reality. I recommend cutting back on pop tech news consumption.
No. You don't pay google to run a task. You pay google to run a TPU instance for some amount of time. The time you spend leaving it on, setting it up, tearing it down, etc, are all still time you pay for. When they have TPU lambda jobs, that's different, but they don't. From their page:
Virtual machine pricing
In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
No, you can't. Under the best case circumstance where you have a V100 and TPU both using tensorflow for something that's optimized for both, the TPU is about 37% better:
The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
We can see the cost and they are signficantly less for TPU 2.0. But now Google has the 3.0.
But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia.
Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia.
It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem.
It will be how it is done for a very long time and just the NN will improve.
Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise.
I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know.
Hope Google will share a paper now on the TPU 2.0 and their secrets.
Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.
Oh man... You think Google is performing some kind of secret that Nvidia has no idea how to do? I really don't understand your fanboyism. The only reason the TPU can perform more operations per watt is because it is a severely limited processor. There is nothing special about that. They chose to dedicate more silicon to a smaller number of features, while Nvidia made the processor more general. If anything, Google uses the generic tsmc Fab, well Nvidia has their own subprocess at tsmc. If Nvidia really, really wanted to make a chip that was dedicated just for deep learning and nothing else, they could. But that's only useful for Google.
The big difference is Google works top down. So Google comes up with WaveNet as an example. That then causes a need to optimize the entire stack to be possible to offer at scale.
So WaveNet part of the Google Assistant and things like Duplex. But Nvidia silicon is just not going to make it possible. So Google does the TPUs as it just would not be possible to do at a reasonable price with Nvidia.
They are doing 16k cycles through a NN in real-time and competing against a far less compute intensive technique.
Now do not get me wrong I am long Nvidia and been for a while. Bit disapointed on the hit after an incredible earnings report.
I think they will do well from a investor marketing standpoint as really the only alternative to Google hardware.
But they have a fundamental disadvantage. They just do not have the applications like Google. They do NOT have the data to iterate like Google. It is why they appears to be 2 generations behind Google.
What is the goal of Nvidia? It is hardware for hardware sake? IMO, it should be driven top down and just do not see that happening at Nvidia. Is their goal the singularity?
Versus Google clearly wants to create the singularity and the silicon is in support. That is a very different calculus compared to Nvidia.
We can see it so strongly this week. The Google Keynote was all about AI applications. Then the TPUs are too support. Take a look at the duplex video for example. That is the focus and the silicon is what makes it possible.
But the more success for Google and presentations like Duplex this week and the buzz across the Internet helps Nvidia if the goal is investing. But from an actual technical solution it just points out the problem for Nvidia that much stronger.
I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod".
So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS.
If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not.
If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
While DGX-2 includes 16 V100, you can't replicate DGX-2 performance with 16 V100. Much of its performance is due to NVSwitch/NVLink. I think it's same with TPU.
This is about scaling. Since you usually can't achieve 8x speed with 8 chips (there will be issues with interconnect, algorithm, etc.), 8x speed with 8 chips is noteworthy. Although I wouldn't describe that scaling work as "8 times more powerful" myself.
~8x speed with 8 chips of the same generation would obviously be very impressive. But 8x speed with 8 chips compared to a last year's single chip is definitely not the same thing.
Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
Google is the leader on AI algorithms. Where GANs and Capsules and so many others came from for example. So things change and Google can implement in silicon far earlier than anyone else as needed.
But right now they have what they need.
BTW, I would be curious on the TPU 3 design considerations to better support dynamic routing with Capsule networks. I suspect creates different memory access patterns and where the big power savings are at today.
You kind of summed it up right there...'it's just an arithmetic logic unit for tensors'...whether its a matrix multiply engine or implmenting sha256 still a custom rapidly iterated chip for narrow very specific use case. Google accomplishment here clearly the software, but doubt they only ones going to crack asic systolic arrays. At some point china inc figures this out in mass...maybe bitmain themselves.
Well... I'll concede that 2-instructions that aren't reading or writing memory is a pretty narrow set of operations. But it's also much more generally useful than computing one specific hash function.
Completely agree with you there. Guess the analogy I was making was more about the evolutionary speed we saw there. Google seems to be moving awful fast here.
This very typical google. V100 comes out, they don't deploy it in their cloud immediately. Then they launch their tpu cloud. Spend 2 months touting cost/speed in benchmarks. Then a week before io they make v100 available in gce, nearly six months after aws. Then at io they announce tpu3. This is supposed to be nvidia's most captive customer, and now looks like going to be their biggest problem. Would love to see google spend with them and how its changed since they ramped tpu2.
For all the google hate, nvidia is too blame as well. Their ceo running around saying 'we saving you money every time you spend 10k on a gpu' is getting old. The should have been far more aggressive price wise with this biz considering the resources of their target customers. Almost every 'choose not to use a gpu' use case being touted by msft/goog etc is about cost. Meanwhile nvda busy tweeting stuff like msft seeing ai app for blind powerd by nvda gpu. Soon micron will be tweeting they powering all ai as well, and then arista...all the way down to the power companies. Watching goog today its not looking good for the competition, by the time all these ml/dl accelerators are commoditized google will have one hell of a moat.
But does it not make sense for Google to remove their dependency on Nvidia?
Plus could they do the new text to speech at a competitive price using Nvidia Silicon?
Google approach gets you a better result but the computational requirements would be huge compared to the traditional approach. You have to get the compute cost down as far as you can.
Disclosure: I work on Google Cloud (and even on the GCE parts).
No conspiracy here, we were simply late to market on the V100 Beta. As many have noted, we also hold a really high reliability bar before going to Beta (“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases) and we’ve be in Alpha for months. If you’d like to be included in Alpha offerings drop us a line.
As I’ve said in previous threads: Compute Engine intends to be the best place for computing. That includes the kind of workloads TPUs are tuned for, but it also includes all the stuff that GPUs excel at. That’s not going away, and GCE isn’t favoring one over the other. We sell infrastructure, and we don’t try to (overtly) pick favorites, we let our customers do that.
Apologies, stand corrected. Conspiracy theories are always more fun:) Guess was just coincidence that after grabbing all this press on GCE v100 finally being available, tpu3 is announced few days later.
You do not want to be dependent on Nvidia for your own stuff is why Google has to do the TPUs.
It is like how they ported to Power for their stuff if needed.
Look at the shortage of late with Nvidia. Google does not have to deal with it.
But also I suspect they could not do WaveNet without their own silicon as the cost would be prohibitive. Using a NN for speech and competing against traditional approaches and have a competitive price is going to require really efficient silicon.
But you are missing my point. Google is not dependent on Nvidia.
Really does not look like Nvidia could do what they need anyway. Doing their new text to speech on Nvidia I would not expect to have a price point to make viable.
So let me get this straight. You are saying Nvidia is having a massive shortage of parts, while ignoring that both AWS and GCP offer GPU instances that you can get any time. Not only that, the article you reference is about Geforce GPUs, which is not a data center GPU. You are comparing apples and oranges.
Please show me an article where there's a shortage of Tesla cards.
It is about being dependent on someone else for something that is critical to you.
But really now it is just Nvidia does not have the ability to meet Google's needs.
I am not aware of anything from Nvidia that could do 16k cycles through a NN in real-time at a cost that you could roll out at scale.
Google priced WaveNet competitively to the old way of doing. You can only do that with a very low cost for computation and the power required which Nvidia could not match with the TPU 1.0 let alone now they have the 3.0.
Just to play devil's advocate for a moment: I'm excited that there will finally be some viable competition for GPUs, but am disappointed that this isn't a general-purpose multiprocessor.
We're long overdue for a general-purpose CPU with say 1024 cores, that avoids a central main memory, where each core can be independently programmed just like any other CPU. Google's may count as a somewhat general-purpose DSP, which is definitely a step forward. But no matter how mature or mainstream a framework like TensorFlow gets, it can never replace full programmability.
Without seeing the internals, I'm going to have to give this a nay vote for now. There are many other rather exciting problems that need to be opened up to a new generation of tinkerers. Off that top of my head, it's things like: content-addressable memory to provide high data locality (evolving various interconnects instead of hardwiring them), exploring other types of general vector processing like the kind MATLAB/Octave uses, and exploring other hill-climbing algorithms than backpropagation/neural nets.
I picture something more like network topology-agnostic Docker containers programmed in Elixer/Erlang/Go that can act as semi-autonomous agents and switch into various modes in order to solve the problem at hand. I just find that a much simpler metaphor to work with than OpenCL/CUDA/TensorFlow. Yes it would take more silicon and would probably violate YAGNI, but only full programmability gives us the freedom to explore the problem space at the level that's going to be required to implement artificial general intelligence.
117 comments
[ 7.3 ms ] story [ 198 ms ] threadGoogle doesn't need "developer buy in" for these to make sense -- they need better hardware for training and deploying deep learning models for their products, which is the overwhelming motivation. And, if they offer these to you, it's only because you're willing to pay for faster TensorFlow iteration.
Google has set the bar where others will have to match. Also looks like Google is now 2 generations ahead of Nvidia.
Would hope we get an even bigger price difference with this generation.
BTW, you are mixing up graphic generations and ML generations. Google never did or will graphics.
For ML Google is 2 generations ahead. Hard to imagine Nvidia ever catching up as so much of the AI breakthroughs are coming from Google.
A perfect example is WaveNet. Google is using a NN for audio in real time at 16k cycles a second. That is competing with using the old way of doing this. The Google approach gets a better result but people are only willing to pay too much more for a better result.
So Google had to price WaveNet competitively to the old approach which they have. I suspect that would have been impossible using Nvidia. The cost in running would have been just too high. The computation required for WaveNet is huge and the old way it is minimal.
Also we are going to need to really separate training with inference. They have different requirements and the costs in running are different.
Has Nvidia done an inference only solution? Or do they still always mix them?
"Google would have made a CPU to compete with Intel. "
I do a lot of surfing and have to say one of the more silly things to read. There is little gains today with CPUs. Why on earth would you invest in doing your own?
Google has ported all their stuff to Power so they have a backup and not reliant on Intel. But CPUs are NOT the future and would be crazy to invest into them at this point.
Processing is moving to TPU type processors and we will see more and more traditional things. A fantastic paper on this from Jeff Dean at Google I suggest you read.
https://arxiv.org/abs/1712.01208
One thing I love about Google is they do NOT reinvent the wheel. If there is something that can work like the Linux kernel they use. Versus having an ego of having to do yourself.
They did all their own network silicon by hiring the Lanai team several years ago because there was nothing that could work for their needs. They then created a determinate network stack so they could do Spanner. There was nothing off the shelve. Heck they laid their own fiber under the ocean to make possible.
Spanner is the first and only horizontally scalable RDBMS solution to exist. Google beat the speed of light by using their custom silicon, custom stack that is determinate, their own fiber. Then using atomic clocks with GPS to remove the latency from speed of light out of the equation. Highly recommend the papers including the Spanner paper.
https://ai.google/research/pubs/pub39966
The network uses custom silicon inside and then they put the intelligence on out edge. Then they control exactly the traffic so they get a determinate result. This also allows them to use far cheaper hardware as do not need to over provision as they never drop packets on the ground or need much memory for buffers.
Google would have used Nvidia in a second if they could do what they need.
But what is being done in AI (ML) today requires unique silicon. Just not possible without as the power footprint would be prohibitive.
I do expect Google more and more to leverage on client ML where needed with their PVC. So it is NOT just the TPUs but for somethings you are going to need on client.
Take the new voices as an example. They are doing 6 voices as each has a different model. The cost in switching models is prohibitive in the cloud.
So you will see them eventually move to some things being done on the client. ...
NVIDIA is now exerting pricing power to the point where they’ve decided to train their sales people to disregard the metric that is most important to customers: cost of training. Talk with one of their enterprise sales people and you’ll find they’ll say things like “FLOPS / $ doesn’t matter” to justify a 10x increase in price for their TESLA line. As history has shown, a monopolist sows the seeds of their own destruction and, by disregarding the metrics that matter, they alienate their customers.
Here are open source projects that you can contribute to to break the monopoly:
ROCm: https://rocm.github.io/
MIOpen: https://github.com/ROCmSoftwarePlatform/MIOpen
TensorFlow: https://github.com/tensorflow/tensorflow
I have done deep learning also and I don't see NVIDIA being an incumbent that is easy to be knocked out. I think we would need some distruption such as a new technique that is more performant or different design goals. Intel is still leading the way and I don't see NVIDIA being unseated with its head start anytime soon.
"Slowly grinds the mill of the gods, but it grinds fine."
Matrox didn't give much of a damn to 3D gaming, 3Dfx were bust doing all the wrong things. It was certainly not "easy", but don't seems hard or impossible either at the time.
https://github.com/plaidml/plaidml
https://github.com/plaidml/plaidml/tree/master/tile/hal
It's kind of cool to look anyway because it's a lot of value in less code than you might expect (complexity is in the layers above HAL).
If someone wanted to add support for Vulkan or something more ambitious like Qualcomm Hexagon or the RPi with QPU that would be pretty cool though.
How fantastic.
TF removes the silicon from the equation. So if Nvidia cheaper use but if TPUs cheaper use. If something new that is cheaper then move. TF is the key.
ArrayMancer: https://mratsim.github.io/Arraymancer/
Written in Nim, which looks and feels a lot like Python, but runs a lot like C++ or Rust, especially in the speed and FFI departments, and deploys like Go as a small single fast executable. Already supports CPU, CUDA and OpenCL. Still early, but with a lot of promise. Use is modeled on some mixture of PyTorch and Keras.
[0]: https://github.com/gorgonia
[1]: https://www.youtube.com/watch?v=7R39CMW4MJQ
We are doing data-center experiments - using oil cooling. It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.
If you're interested, Alibaba has been doing some work along the lines of production immersion cooling.
What kind of oil are you using?
Mineral oil is ok for experimentation, but for long term material compatibility and fire risk I wouldn't recommend it.
FWIW I co-founded https://submer.com where we've developed an all-embedded computing immersion cooling solution that is virtually compatible with any kind of hardware (even fiber optics) and it's orders of magnitude more efficient than traditional data center cooling technologies.
Also - some oils while won't hurt your circuits, they might start dissolving the various plastic connectors on cables and board.
That will make a datacenter fire into something else entirely.
100F and below is considered flammable.
Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself.
And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs. So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive...
Is there any actual spec released?
How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
Even more so with ML training. It is much more bumpy.
But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded.
Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease.
But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example.
Just one year and we have the 3.0.
So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware.
The other is buying the hardware is much harder from a cash flow standpoint.
You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense?
Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
But the future is the cloud. I have fought people on this for well over a decade and thought everyone by now got it. Apparently not.
On TPU being a miracle chip not clear to me what a "miracle" chip would even be? Plus things keep improving and how could you have a static "miracle" chip?
What we do know is Google is doing WaveNet in production in real-time. We can hear the results. They are offering at a competitive price to traditional TTS.
We can also see the TPUs are about 1/2 the cost of using Nvidia.
Otherwise that is it. But I do not know your faith but personally I would not consider that a "miracle". But maybe just me.
Maybe Lewandowski would as I have heard he prays to a AI god or something like that.
The big difference is Google is working top down and Nvidia has to work bottom up.
So Google wants to roll out the best TTS there is to the world and at scale.
So they are going to from the top down to be able to achieve.
Clearly Google has a goal of creating the singularity and the Silicon is just a piece of the equation.
Versus I am not really sure what Nvidia goal is. Silicon is not for silicon sake. But rather a means to an end.
Plus Google having in production has the data to optimize in future versions where Nvidia just is not going to be able to.
I am long Nvidia and owned it for a bit. Was disappointed on how the stock traded after ER. Do think it will do fine as Google does all these incredible AI things. It is an alternative that will be good for them.
But for the very long term I would not hold. Google is a hold forever. Nvidia would keep on it.
BTW, would say Nvidia is now two generations behind.
Oh wow. Didn't realize you're one of those. I used to be a Kurzweil fan too, when I was 18. That phase usually passes by senior year in college. Well, I guess some people need longer to see reality. I recommend cutting back on pop tech news consumption.
Take it easy, and have a nice day!
BTW, I am old as in 50s. Have to explain to me in a little more detail what you mean by your post?
I am very curious but just not following?
Virtual machine pricing In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
How much it cost you to complete that task is what matters. How it is done is here or there as long as get the precision.
We can see right now Google with their TPUs is about 1/2 the cost of using Nvidia with AWS.
https://blog.riseml.com/comparing-google-tpuv2-against-nvidi...
The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia.
Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia.
It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem.
It will be how it is done for a very long time and just the NN will improve.
Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise.
I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know.
Hope Google will share a paper now on the TPU 2.0 and their secrets.
Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.
So WaveNet part of the Google Assistant and things like Duplex. But Nvidia silicon is just not going to make it possible. So Google does the TPUs as it just would not be possible to do at a reasonable price with Nvidia.
They are doing 16k cycles through a NN in real-time and competing against a far less compute intensive technique.
Now do not get me wrong I am long Nvidia and been for a while. Bit disapointed on the hit after an incredible earnings report.
I think they will do well from a investor marketing standpoint as really the only alternative to Google hardware.
But they have a fundamental disadvantage. They just do not have the applications like Google. They do NOT have the data to iterate like Google. It is why they appears to be 2 generations behind Google.
What is the goal of Nvidia? It is hardware for hardware sake? IMO, it should be driven top down and just do not see that happening at Nvidia. Is their goal the singularity?
Versus Google clearly wants to create the singularity and the silicon is in support. That is a very different calculus compared to Nvidia.
We can see it so strongly this week. The Google Keynote was all about AI applications. Then the TPUs are too support. Take a look at the duplex video for example. That is the focus and the silicon is what makes it possible.
But the more success for Google and presentations like Duplex this week and the buzz across the Internet helps Nvidia if the goal is investing. But from an actual technical solution it just points out the problem for Nvidia that much stronger.
So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS.
If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.
Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
- 64 TPUs per pod (11.5 petaflops)
- 4 chips per TPU (180 teraflops)
https://twitter.com/soumithchintala/status/99391656047246131...
But right now they have what they need.
BTW, I would be curious on the TPU 3 design considerations to better support dynamic routing with Capsule networks. I suspect creates different memory access patterns and where the big power savings are at today.
They shipped Deep Learning Accelerating Card SC1 at $589. You can't buy it right now because it is sold out.
But what I wanted to know is performance in terms of joules compared to TPU 2s.
My hope is we get a paper on the TPU 2 now the 3 is released. We got the TPU 1 paper as they were releasing the TPU 2 I suspect.
Plus could they do the new text to speech at a competitive price using Nvidia Silicon?
Google approach gets you a better result but the computational requirements would be huge compared to the traditional approach. You have to get the compute cost down as far as you can.
No conspiracy here, we were simply late to market on the V100 Beta. As many have noted, we also hold a really high reliability bar before going to Beta (“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases) and we’ve be in Alpha for months. If you’d like to be included in Alpha offerings drop us a line.
As I’ve said in previous threads: Compute Engine intends to be the best place for computing. That includes the kind of workloads TPUs are tuned for, but it also includes all the stuff that GPUs excel at. That’s not going away, and GCE isn’t favoring one over the other. We sell infrastructure, and we don’t try to (overtly) pick favorites, we let our customers do that.
Wow, that's so ... I don't even have a word for it.
It is like how they ported to Power for their stuff if needed.
Look at the shortage of late with Nvidia. Google does not have to deal with it.
But also I suspect they could not do WaveNet without their own silicon as the cost would be prohibitive. Using a NN for speech and competing against traditional approaches and have a competitive price is going to require really efficient silicon.
GPU shortage equals four-month wait time for buyers"
https://www.theregister.co.uk/2018/03/30/nvidia_geforce_chip...
But you are missing my point. Google is not dependent on Nvidia.
Really does not look like Nvidia could do what they need anyway. Doing their new text to speech on Nvidia I would not expect to have a price point to make viable.
Please show me an article where there's a shortage of Tesla cards.
But really now it is just Nvidia does not have the ability to meet Google's needs.
I am not aware of anything from Nvidia that could do 16k cycles through a NN in real-time at a cost that you could roll out at scale.
Google priced WaveNet competitively to the old way of doing. You can only do that with a very low cost for computation and the power required which Nvidia could not match with the TPU 1.0 let alone now they have the 3.0.
We're long overdue for a general-purpose CPU with say 1024 cores, that avoids a central main memory, where each core can be independently programmed just like any other CPU. Google's may count as a somewhat general-purpose DSP, which is definitely a step forward. But no matter how mature or mainstream a framework like TensorFlow gets, it can never replace full programmability.
Without seeing the internals, I'm going to have to give this a nay vote for now. There are many other rather exciting problems that need to be opened up to a new generation of tinkerers. Off that top of my head, it's things like: content-addressable memory to provide high data locality (evolving various interconnects instead of hardwiring them), exploring other types of general vector processing like the kind MATLAB/Octave uses, and exploring other hill-climbing algorithms than backpropagation/neural nets.
I picture something more like network topology-agnostic Docker containers programmed in Elixer/Erlang/Go that can act as semi-autonomous agents and switch into various modes in order to solve the problem at hand. I just find that a much simpler metaphor to work with than OpenCL/CUDA/TensorFlow. Yes it would take more silicon and would probably violate YAGNI, but only full programmability gives us the freedom to explore the problem space at the level that's going to be required to implement artificial general intelligence.