This is huge. If they really do offer such a perf/watt advantage, they're serious trouble for NVIDIA. Google is one of only a handful of companies with the upfront cash to make a move like this.
I hope we can at least see some white papers soon about the architecture--I wonder how programmable it is.
Yup, I assume they're gonna keep them in house as a competitive advantage for a time. I doubt they'll do it forever; the most valuable part of NVIDIA's CUDA is the ecosystem, and I think Google knows that.
Sure, but that's the deal. I'll buy the latest nVidia 1080 card as soon as I can but renting these custom chips per minute would be a way better option for me.
GPUs also have this nice side effect of being great at playing games on. Purely as a guess I'd think that the gaming market is bigger than the AI researcher market.
In a future where AI is everywhere, Nvidia hopes it can sell GPUs by the hundreds and thousands to large data centers. You can make a lot more money a lot faster selling your hardware this way, and Nvidia is very interested in it judging from how much they talked about it at their recent conference.
I would be surprised if they weren't working on their own specialised chips then, though Google have the advantage of already having the software specs to build for.
> Purely as a guess I'd think that the gaming market is bigger than the AI researcher market.
Machine learning isn't just targeting the AI researcher market though -- it's widely used by a huge number of companies, and of course, by many of Google's most important products. I would argue that those markets combined are larger than gaming.
There's no way Google lets this leave their datacenters. Chip fabrication is a race to the bottom at this point. [1]
Google is doubling down on hosting as a source of future revenue, and they're doing that by building an ecosystem around Tensorflow.
What I think is interesting is how weak Apple looks. Amazon has the talent and money to be able to compete with Google on this playing field. Microsoft is late, but they can, too.
Where's Apple? In the corner dreaming about mythical self-driving luxury cars?
Apple's strength is in consumer (and to a lesser extent, developer) ecosystems; the cozy comfortable bubble you get when you're surrounded by everything Apple. Getting access to your stuff across multiple devices is virtually effortless and continually seamless, with almost no configuration required.
Whether that's good or not may be arguable, but it's certainly a selling point for many and I don't see Google or any other company's offerings approaching the same experience, and I suspect that's by design; they have to be more open and support all devices but that kinda dilutes everything. Apple will only get stronger in that aspect IMO.
I'm not sure I agree with you, long term (about the chips). I think that the value here is in the ecosystem. If Google can compete with CUDA, they'll be doing really well.
> There's no way Google lets this leave their datacenters. Chip fabrication is a race to the bottom at this point. [1]
I’d hope someone somewhere steals the blueprints and posts all of them publicly online.
The whole point of patents was that companies would publish everything, but get 20 years of protection.
But by now, especially companies like Google don’t do so anymore – and everyone loses out.
EDIT: I’ll add the standard disclaimer: If you downvote, please comment why – so an actual discussion can appear, which usually is a lot more useful to everyone.
Same misuse happens with copyright. Both were invented to foster publishing and not creating life-long monopolies. The life times of copyright and patents must be way shorter as well. Everyone builds on something that came before. It's impossible to build a better bike if you have to test drive on a street with patent mines.
Re EDIT: Downvotes must be comment-mandatory or not allowed otherwise.
There's little need for anyone to steal the blueprints. It's unlikely there's anything particularly "special" there other than identifying operations in Tensorflow that take long enough and are carried out often enough and are simple enough to be worth turning into an ASIC. If there's a market for it, there will be other people designing chips for it too.
Quantum computers, OpenPower, RISC-V, and now this - I'm really liking Google's recent focus on designing new types of chips and bringing some real competition into the chip market.
"General purpose" isn't that general, if you look at the actual operations they support and their threading model. It's already fairly optimized for these sorts of operations, and this amount of claimed headroom makes me suspicious.
The fact that I can compile arbitrary programs for the GPGPU means it is general purpose. NVIDIA isn't writing softmax or backprop into silicon as a CPU instruction.
Look at how much faster ASICs for bitcoin mining are than the GPU... orders of magnitude.
"Backprop" isn't even close to something that would be a "CPU instruction", it's an entire class of algorithm. It's like saying "calculus" should be a CPU instruction. Matrix multiplication & other operations, on the other hand, do neatly decompose into such instructions, which have been implemented by NVidia et al., since that's the core set of functionality they've been pushing for like a decade now.
Additional die space on additional functionality might hurt the power envelope (which is where the focus on performance / watt rather than performance kicks in) but it doesn't make your chips slower per se.
That was my impression too. ML under the hood was a lot of linear algebra, not very different than most shaders. But maybe Google decided to hardcode a few important ML primitives because the ROI was that good in terms of grabbing customers. Also they might have very large scale applications not found elsewhere that motivates this.
Ok I was obviously oversimplifying things but my point is since we can only speculate, it's clear that when you know specific algorithms/math operations/memory layouts/applications you want to optimize for you can create dedicated chips that optimize and do that quickly. That bitcoin miners are all dedicated chips and run circles around GPUs demonstrates exactly this fact.
Furthermore the fact that ML can be error tolerant means you also get to optimize certain floating point operations for speed or energy efficiency at the cost of accuracy. NVIDIA doesn't get to do this in their linear algebra support.
Bitcoin mining is an extremely well-defined task compared to machine learning. It remains to be seen how general these TPUs are in practice - whether they will support the neural network architectures common two years from now.
Google has a lot of potential options that NVidia doesn't have. They can size their cache heirarchy to the task at hand. They can partition their memory space. They can drop scatter/gather. They can gang ALUs into dataflows that they know are the majority of machine learning workloads. They can partition their register file at the ISA level or maybe even drop it entirely. They can drop the parts of the IEEE754 floating point spec they don't need and they can size their numbers to the precision they need.
If they balance compute to memory better than GPUs, you could definitely see a 10x. GPUs have large off chip memory and small caches (like 256kb). Cost to going to off chip memory can be 1-2 orders of magnitude more than on chip memory. You can certainly fit 4+MB on modern processors, but they likely bought designs from a company like Samsung because designing high performance, low power memory cells is tricky. I'm surprised they were able to keep things a secret.
it's ASIC tuned for specific calculations, I'm sure it's better power consumption than general purpose GPUs. Same as crypto mining ASIC's crush GPU's in terms of power efficiency.
There isn't much data yet but I'm also guessing they probably have access to much more RAM than NVidia cards and can process much bigger data sets
That's what I'm thinking. I was anticipating the release of GPU instances, but now I'm thinking that they will simply leapfrog over GPU instances straight to this.
2. It works really well (or else they wouldn't be using it so broadly)
3. Considering how long this was said to be in development, it also likely means they are working on the next big improvement before these guys have even gotten the current one working.
They are already using it in production and have been for over a year? That is a huge difference compared to what people are working on and building. They beat Lee Sedol using this hardware. Over 100 teams at Google are using machine learning. This hardware already accelerates their projects.
IBM's TrueNorth chip is taking a much more neuromorphic design approach by trying to approximate networks of biological neurons. They are investigating a new form of computer architecture away from the classic Von Neumann model.
TPUs are custom ASICs that speed up math on tensors i.e. high-dimensional matrices. Tensors feature prominently in artificial neural networks, especially the deep learning architectures. While GPUs help accelerate these operations, they are optimized first and foremost for video rendering/gaming applications -- compute-specific features are mostly tacked on. TPUs are optimized solely for doing ML-related computations.
3 generations ahead of moore law??? I really wonder how they are accomplishing this beyond implementing the kernels in hardware. I suspect they are using specialized memory and an extremely wide architecture.
Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LOVE for them to market these for general use.
I doubt FP hardware size is the limiting factor in their implementation (it's not in GPUs and definitely not in high perf CPUs). More likely they came up with higher level architectural tricks that let them specialize for machine learning (i.e. taking better advantage of locality in the application, etc).
Generally speaking GPUs are already very good at running with float32s, usually much better than they are at using float64s in fact. The big advantages of using an ASIC are mostly on the storage side but they also allow you to get away with non IEEE floating point numbers that don't necessarily implement subnormals, NaN, etc.
The rule of thumb I was taught was that going from a DSP/GPU to a custom ASIC would give you a 10X advantage in performance/power which is pretty close to this. And look at how much bitcoin mining ASICs out compete GPUs.
From the article: "TPU is tailored to machine learning applications, allowing the chip to be more tolerant of reduced computational precision, which means it requires fewer transistors per operation."
In addition, there's a lot of literature on optimizing hardware implementations of fundamental arithmetic operations like addition and multiplication. I recall seeing a paper a while ago which talked about reducing the number of gates by allowing some bounded imprecision in the results - unfortunately, I don't remember the title right now, but it sounds like that's what they may be doing.
these are ASICs, Application Specific Integrated Circuits, emphasis on the SPECIFIC. It's a chip built specifically for Tensor Flow. Anytime you build a chip to handle a specific application you are going to see a significant performance improvement. You can move into a new apartment using a Honda Civic, but you are going to see considerable performance improvement using a vehicle designed specifically for moving.
The article confirms that "AlphaGo was powered by TPUs in the matches against Go world champion, Lee Sedol, enabling it to "think" much faster and look farther ahead between moves."
It seems entirely reasonable to me, and there's good historical precedent for it. Let's look at SHA256 hashing as an example. The maximum number of hashes that the best GPU around can do is around 1 GHash/s. However, for the same cost, of around $600, specialized hardware can do around 5 THash/s. That's about five thousand times the performance/price. There's no reason that hardware that is super-specific to neural network computation can't similarly have large gains.
No - tensorflow is open source and you can run it on many platforms. TPUs are about efficiency. You might not be able to do image recognition as efficiently without one, but you can still perform exactly the same tasks.
I would be shocked if tensorflow optimizations where useful 1:1 for stock Intel chips or GPU's. So, there is still plenty of lock-in even if your process runs. GPU vendors love to play this game by helping optimize games.
It's possible, but I think that the majority of ML optimization as seen by a programmer using tensorflow is more about optimizing the balance of accuracy, training & inference speed, and memory use, and a lot of the solutions in this space are pretty hardware independent. There's an entire other type of optimization about, e.g., making conv2d insanely fast, but that's not something that a typical data scientist-type user deals with.
(To elaborate -- it's questions like "how deep should I make this convolution? Should I use tf.relu or tf.sigmoid? How many fully-connected layers should I put here, and how big should I make them?". These are really knotty deep learning design questions, but they're often h/w independent. Not always - we certainly have some ops on TF that we only support in CPUs and not on GPUs, for example - but often.)
1. Best price/performance is tensorflow right now. So, the best software choice is platform X.
2. Then in 2 years.. Well we are using Platform X so tensorflow is clearly the best option.
In other words once you pick conv2d, you tend to also stick with whatever conv2d is optimized for. Which also means HW vendors love to help optimize popular platforms.
All of the comparable tools are practically locked into nVidia gpus/CUDA. TensorFlow is rapidly reaching performance parity on that hardware [1] and can now be run on this so it's actually sort of the least locked in framework.
I think the difference you observe relates directly to the difference between what Google does outside of Cloud Platform and what Amazon does outside of AWS.
no it's just that google has so many shiny things that come and go, if you build a business with shiny new things, you will propably fail if you do that over and over again, just because its shiny and new.
If I recall correctly, it took Google a while to actually offer the boring stuff. For a while, you could get a Google Compute Engine but you couldn't just get a dang VM image, because Google knows better than you and you should do things their way. They've fixed it now, but lost a lot of potential market share for that conceit.
If I recall correctly myself (getting old!), Google has had the ability to build your own custom image since GCE went GA 2.5 years ago. Now, admittadly, it took a while to get IAM and VPC going, but we done did it now!
I'd love to hear what other boring stuff has been a showstopper for you, in case we missed something dumb :)
If I recall correctly myself (getting old!), Google has had the ability to build your own custom image since GCE went GA 2.5 years ago. Now, admittadly, it took a while to get IAM and VPC going, but we done did it now!
I'd love to hear what other boring stuff has been a showstopper for you, in case we missed something dumb :)
If you're evaluating something today, how does it change your decision that we were late to market with Compute Engine (and in this specific case "bring-your-own-kernel")?
If it's about future boring stuff, I think the list of boring stuff isn't too long ;).
All given, the fact that Google itself doesn't extensively use GC is kind of a red flag(I know quite a few Googlers from search infrastructure and none of them said their teams used GCE internally).
A solid guarantee with AWS is if AWS goes down, then a multitude of Amazon's services also will go down(ex Amazonian myself), so it gives me a belief that AWS's uptime is more important to Amazon itself that it is for external customers.
Search Infra (and Ads for that matter) is an extreme case. Google Search might be one of the worlds most highly tuned infrastructure projects: a marriage of code and hardware design to maximize performance, scoring, relevance and ultimately ROI.
Before we had custom machine types (November 2015 GA), we wouldn't have been remotely close to what they needed. I'm not even sure we've had anyone evaluate the amount of overhead KVM adds in either latency or throughput.
tl;dr: Don't let Search be your "not until they do it". We've got folks in Chrome, Android, VR, and more building on top of Cloud (as well as much of our internal tooling being on App Engine specifically).
"So" Google lost potential business for a while from people who wanted to spin up VMs rather than wanting to ship code to a proprietary execution framework.
I think you're agreeing: We certainly missed a huge segment of the market at the time, but now that we've got GCE new business can certainly come our way.
Not implying that AWS hasn't had them. It's just that adopting GCE this early makes you a bit of a guinea pig because GCE isn't used internally at Google.
What is the capabilities that a piece of hardware like this needs to have to be suitable for machine learning (and not just one specific machine learning problem)?
It is interesting how malleable are neural networks.
- you can drop half the connections and it still works, in fact it works even better, during training
- you can represent the weights on as little as one bit, but still use real numbers for computing activations
- you can insert layers and extend the network
- you can "distill" a network into a smaller, almost as efficient network or an ensemble of heavy networks into a single one with higher accuracy
- you can add a fixed weights random layer and sometimes it works even better
- you can enforce sparsity of activations and then precompute a hash function to only activate those neurons that will respond to the input signal, thus making the network much faster
It seems the neural network is a malleable entity with great potential for making it faster on the algorithmic side. They got 10x speedup mainly on exploiting a few of these ideas, instead of making the hardware 10x faster. Otherwise, they wouldn't have made it the size of a HDD - because they would need much more ventilation in order to dissipate the heat. It's just a specialized hardware taking advantage of the latest algorithmic optimizations.
Sacrifice generality, accuracy and ability to randomly access a lot of memory so that you can implement fast and power-efficient matrix operations with a single, low accuracy datatype thus requiring less memory, bandwidth and transistors.
Now this is really interesting. I've been asking myself why this hadn't happened before. Its been all software, software, software for the last decade or so. But now I get it. We are at a point in time where it makes sense to adjust the hardware to the software. Funny how things work. It used to be the other way around.
This is known as the Wheel of Reincarnation. Functionality moves to special-purpose hardware then back to software, and the cycle repeats. (The computer term is from 1968 so this has been happening for a long time.)
At some point in the past, Rob Pike mentioned that when we was working on Voyager (that spaceship that almost 40 years after launch, has left the solar system and continues to send back valuable science data), he had a relatively good understanding of the system from the quantum level (transistors are based on quantum theory) to the solar system. He wasn't kidding, either.
Very interesting fact. But the average programmer is not Rob Pike. How do you see this panning out for the average programmer? Will people need to learn a bit about chips to build more efficient CRUD apps?
I think there will always be an API. In the case of Deep Learning you already see Caffe, TensorFlow, etc. readily available for developers.
I don't think the average developer will need to understand chip design but I do think _many_ developers will need to know how to use deep learning frameworks.
It had definitely started as an API side detail. But it might become an industry thing. It allows very easy vendor lock-in for all the *AAS products (PAAS, SAAS, etc). You could be required an add-on for your server or machine to be able to use their product.
i'm an average programmer but reading "High Performance Computing" http://shop.oreilly.com/product/9781565923126.do made a huge difference for me even when writing more efficient CRUD apps. A big issue is understanding the cost of the operations in the stack you are using and the overheads caused by your stack.
I dunno... I'm not Rob Pike, but I could basically describe computing from the quantum level to the solar system dynamics and electromagnetic influences of a robotic interplanetary craft.
Then again, I've been coding for ~25 years, am an avid amateur astronomer, and have a degree in physics, so maybe the moral here is that some things just just take more time to master, beyond those first 6 weeks in a coding camp learning how to put together your first jQuery.
I've been thinking for a while that with the end of silicon shrinkages, we may start seeing the final cycles of the wheel, with a final stop at mostly specialized hardware for greater power efficiency.
Not only power efficiency but software efficiency. Custom chips combined with DSLs are a powerful combination. At the expense of segmentation, of course.
Interesting, I hadn't heard of this term before but it sounds about right!
Maybe the next iteration will be assembly instructions specifically for Neural Nets built into CPUs...actually something like a convolution assembly instruction wouldn't even surprise me at this point.
I'm guessing that the performance / watt claims are heavily predicated on relatively low throughput, kind of similar to ARM vs Intel CPUs - particularly because they're only powering it & supplying bandwidth via what looks like a 1X PCIE slot.
IOW, taking their claims at face value, a Nvidia card or Xeon Phi would be expected to smoke one of these, although you might be able to run N of these in the same power envelope.
But those bandwidth & throughput / card limitations would make certain classes of algorithms not really worthwhile to run on these.
> I'm guessing that the performance / watt claims are heavily predicated on relatively low throughput, kind of similar to ARM vs Intel CPUs - particularly because they're only powering it & supplying bandwidth via what looks like a 1X PCIE slot.
Agreed. Also tells you that they don't need to communicate with the CPU much, given that it only has a PCIE. Reminds me of Knights Ferry, in this respect.
> a Nvidia card or Xeon Phi would be expected to smoke one of these
Will be very interesting to see some head-to-head benchmarks between these guys (on tensorflow and other libraries) in the next few months. Especially as Knights Landing starts to appear, and the new Nvidia card.
> IOW, taking their claims at face value, a Nvidia card or Xeon Phi would be expected to smoke one of these, although you might be able to run N of these in the same power envelope.
That seems unlikely, right? GPGPU software beating an ASIC? I guess it depends on just how abstract and adaptable the TPU IC is. Sounds like they use a custom number representation -- if they can squeeze it into less than half precision (IEEE FP16) then it would be super hard for a Phi or GPU to beat it.
But ultimately it all comes down to specific applications' bottlenecks. If they don't have to go off-card ever and their workset fits into the on-die memory, then they'd have no real advantage by using a PCIe GPU with GDDRx on it.
So now open sourcing of "crown jewels" AI software makes sense.
Competitive advantage is protected by custom hardware (and huge proprietary datasets).
Everything else can be shared. In fact it is now advantageous to share as much as you can, the bottleneck is a number of people who know how to use new tech.
Agree. To attract devs to using your spec, then eventually go to your platform. But since the spec are open, AMZN/MS can make their own too. So it is still a good thing.
Indeed. I kind of realized this more recently when I saw some new chips promote that Tensorflow works on them. This is really what it was all about - getting chips everywhere to support Google's version of AI, which means that in the long term they can get those chips themselves in volume - chips that are optimized for their AI out of the box.
Cool how he foreshadows the end of Sun (takeover by Oracle in 2010) in that article from 2002:
"Sun's two strategies are (a) make software a commodity by promoting and developing free software (Star Office, Linux, Apache, Gnome, etc), and (b) make hardware a commodity by promoting Java, with its bytecode architecture and WORA. OK, Sun, pop quiz: when the music stops, where are you going to sit down? Without proprietary advantages in hardware or software, you're going to have to take the commodity price, which barely covers the cost of cheap factories in Guadalajara, not your cushy offices in Silicon Valley."
What? He also predicted how and why it would fail. Sun was a big enough player then that it could have survived plenty of other ways. Apple of today looks nothing like the company in 2000, but Sun got caught out more or less exactly as described and never adapted.
Joel Spoelsky : "commoditize your complements." Steve Ballmer : "developers! developers! developers!"
Developers are a complement to hardware, software, and data based businesses. Google probably spends more on employees than on hardware or data. They really want to commoditize developers that can work on their stuff.
Google's business is based on having more and better data than their competitors. In many cases they have been happy to commoditize hardware, making open their datacenter designs for instance. That helps to cheapen their costs for building datcenters.
In other cases they open source software, like tensorflow and go language. These choices are made to commoditize developers. Google wants there to be a big pool of people who know how to use the technologies that Google uses. More developers means less costs for Google to hire and train employees. Which is their biggest expense: win!
With the TPU, as long as no one else is doing that kind of hardware for machine learning, it is a proprietary advantage to keep it secret. But at some point the logic flips: when others start to do similar things Google would rather commoditize their version of the tech. Because the lesson of the last 40 years is any widely used hardware WILL become commoditized. The inertia is with software codebases and developer knowledge.
Maybe they play one move every time someone gets to go there to fix something? or could it be just a way of numbering the racks or something eccentric like that?
I wouldn't be surprised if Google is looking to build (or done so already) a highly dense and parallel analog computer with limited precision ADC/DACs. I mean that's simplifying things quite a bit, but it would probably map pretty well to the Tensorflow application.
If you have an application that can tolerate error (like classification), then analog computing can give enormous gains in terms of speed _and_ power efficiency. Essentially, the savings come from using physics to perform the math (see Kirchhoff's current law) vs. using discrete time steps vs. fully-unrolling the logic. Google may not be using analog processing for this version, but I read an analog neural network researcher's page who said he moved to Google last year. (Sorry, I can't find the page again, but I think he was from the UK.)
For the curious, Optalysys has built a general purpose optics-based correlation/pattern matching machine. From some of their predecessor-company marketing material: The correlator performs pattern matching on large data sets such as high-resolution images, providing a measure of similarity and relative position between objects within the input scene. This allows large images [and general data converted to images] to be analysed far faster than electronic equivalents.
Going back to the topic of NN-based computing, I found this talk to be intriguing: https://www.youtube.com/watch?v=dkIuIIp6bl0. The main argument is that because Moore's law may no longer be in effect, it will become increasingly important to explore alternate computing solutions. (Google's TPU could be supporting evidence for this argument.) The speaker also co-authored a paper which I liked "General-Purpose Code Acceleration with Limited-Precision Analog Computation".
This video was very cool. Are there any IC's that can perform analog computing for neural networks on the market now? I'm picturing something like an FPGA but with a bunch of op amps that you can connect into summers or amplifiers.
If not, how would one practically implement an analog computer for neural network programming (without several tables full of op-amps?)
You can implement an analog neural network yourself using a Field Programmable Analog Array. (I've never done it, but you'll see academics online writing papers about it.)
Another thing that is sort of related is Lyric Semiconductor; they built these cool application-specific probabilistic processors; they were purchased by Analog Devices a while back.
True. ML applications are well-suited to analog computation not only because they can tolerate errors, they also have an ability to adapt to errors, provided training algorithms are ran in hardware.
Analog circuits run at full-speed (no clock), use little power, and take up little space. An analog computer directly implements the mathematical function it represents as circuits instead of emulating it on a von Neumann-like machine. Long story short, they have issues that made people go digital. Yet, if you can use analog, you can get significant advantages. Example for math acceleration:
Brain is a bunch of components that are spread out 3D that operate like a mathematical function at slow speed. Mostly sounds analog. Results in us. So, a huge spread of analog components could get some results directly simulating something like that. Here's one of my favorites which is a wafer-scale, analog computer for neural networks.
Computing with opamp primitives has somewhat fallen out of favor since you've been away..
The primary cost factor in anything computing is just power. You can always buy more of the things, but power is the ongoing cost and every watt in computing costs you extra in cooling power.
I wonder if opening this up as a cloud offering is a way to get a whole bunch of excess capacity (if it needs it for something big?) but have it paid for.
I'm happy to hear that this is finally public so I can actually talk about the work I did when I was at Google :-).
I'm a bit surprised they announced this, though. When I was there, there was this pervasive attitude that if "we" had some kind of advantage over the outside world, we shouldn't talk about it lest other people get the same idea. To be clear, I think that's pretty bad for the world and I really wished that they'd change, but it was the prevailing attitude. Currently, if you look at what's being hyped up at a couple of large companies that could conceivably build a competing chip, it's all FPGAs all the time, so announcing that we built an ASIC could change what other companies do, which is exactly what Google was trying to avoid back when I was there.
If this signals that Google is going to be less secretive about infrastructure, that's great news.
When I joined Microsoft, I tried to gently bring up the possibility of doing either GPUs or ASICs and was told, very confidentially by multiple people, that it's impossible to deploy GPUs at scale, let alone ASICs. Since I couldn't point to actual work I'd done elsewhere, it seemed impossible to convince folks, and my job was in another area, I gave up on it, but I imagine someone is having that discussion again right now.
Just as an aside, I'm being fast and loose with language when I use the word impossible. It's more than my feeling is that you have a limited number of influence points and I was spending mine on things like convincing my team to use version control instead of mailing zip files around.
Google has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had.
My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo was based on this hardware, they indicate to researchers around the world that if you want to build the most advanced ML models you need to be at Google. Same reason they talked about MapReduce/GFS back in the day.
Sure... I think you are missing the point. In Nov., the Nature paper which contains the AlphaGo algorithm, the hardware detailed was exclusively CPU+GPUs.
Between that time and the Lee Sedol match, the hardware running AlphaGo was switched, to these TPU.
From the paper:
"The final version of AlphaGo used 40 search threads, 48 CPUs, and 8 GPUs. We also implemented a distributed version of AlphaGo that exploited multiple machines, 40 search threads, 1,202 CPUs and 176 GPUs."
Or they consider the TPU to be something similar to a math co-processor, it just offloads work from the CPU. They also didn't discuss the custom built power modules, or the custom built network switches used for interconnects.
Note that the version of AlphaGo that beat Fan Hui and was presented in Nature is significantly different from the Version that played Lee Sedol. Unless you believe that they didn't work on it for half a year.
This explains a lot. Going into the match, Sedol thought he could beat DeepMind but thought it might only be a couple of years until the technology outpaced him. We knew Fan Hui and other Go professionals were helping the team, but a massive speedup is always nice too.
It's a bit underhanded, however. IMHO, the player should be able to study recent games before the match. But this is pretty typical, there were similar late-stage improvements with Chinook (checkers) and Deep Blue (chess).
It really boosts Google Cloud Platform with image and prestige, they're coming out with what is pretty much tensorflow as a service with their machine learning product.
Coming out saying they can give you a service no one else can right down to a custom chip may sway a few buyers in the market.
For me this is true. GPUs on Amazon are not cheap, and if google passes the savings on from performance power consumption I'd certainly offload some work to Google cloud.
This is true - I've heard rumors that GCE has been trying hard to poach cloud consumers from other providers, this could be another stroke in that effort.
Well this also shows their commitment to Tensorflow if they willing to pay to fab custom chips for it. Then again, in the Google scheme of things that's probably not a huge cost.
They've probably carefully modeled the energy savings over GPU/FPGA and found it to be substantial enough, even taking into account costs of design changes.
My vague impression is that there had started to be more and more research on use of ASICs for ML (though I can't say I follow it that closely). Enough of a changing tide that the competitive advantage was going?
I'm suspecting that the design contains a lot of repetitive units, each of which can easily be tested in software. So perhaps testing on an FPGA isn't even needed.
I think zymhan meant prototyping on an FPGA to see if you could actually see an improvement by mapping parts of your algorithm to custom hardware. This would be a sensible first step before committing to a full ASIC run. I'm sure Google did do this initially.
As for testing for correctness, ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.
> ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.
It's actually very common to test ASIC RTL on FPGA. You are right that if you are targeting an FPGA you may write things differently, but this doesn't mean RTL written for an ASIC won't work it's just it might not clock as fast as it could and may not make the most efficient use of resources.
There is a lot of engineering work to take an ASIC targeted IP and run it in an FPGA, but thorough verification of an ASIC design is extremely important so it's usually worth the time to bring up an FPGA that can run it (sometimes as it's too large you have to split it across multiple FPGAs).
Chip designs are usually simultaneously tested in software simulation, FPGA and hardware emulators (basically special purpose super computers that run Verilog or HDL, not quite FPGA speeds but far better debug capabilities, approaching that of software RTL simulation, downside is they cost a few million a piece).
You'll also break a design down into units or blocks. These can be tested separately. E.g. you might see an L1 cache as a single unit and have a testbench (piece of HDL to stimulate and check a design) to test only that. This is useful as it's far quicker than a full design test however extensive full design testing is still required as many bugs you see as the consequence of the interactions between units.
Once you've done a lot of testing you'll tape out a chip. This may well have some issues so you'll do a new 'spin'.
I'm not even a hardware or AI person, and even I could have told you ASICs would make way more sense than GPUs or FPGAs for machine learning. It's all about data locality. Fetching memory is the most costly thing a GPU does, and for ML (DNNs) there no big need for global access memory 99% of the time.
Anyone the casually follows AI knows that people have been talking about making DNN ASICS for some time. It was all a matter of time and $$$$$$
There is no doubt FB is working on them too. Which is why Google is finally publicaly saying that "we did it first ;)"
> Google is finally publicaly saying that "we did it first ;)"
Makes sense. In that respect yeah, they probably wanted to keep it under wraps to avoid Facebook/others from getting a timeline estimate out of it and jump ahead.
That's kind of what Movidius says, too, about its Myriad 2 VPU, which is kind of a GPU (SIMD-VLIW) with larger amounts of local memory combined with hardware accelerators.
Well, it's a first announcement on a blog. They say it accelerates TensorFlow by 10x. They say it fits in an HDD slot. And the whole announcement must stay within a page or two.
It's a "more details to follow" type of thing. Pretty standard actually.
The cool thing here is that it's already widely deployed in production, so we know it actually works and that it's not just vaporware. The "more details to follow" is thus a lot more convincing than for a lot of similar-seeming announcements of things that don't actually exist yet (and often don't ever).
You can make some assumptions though. If the power consumption was equal, the performance is 10x.
The speed at which an ASIC will run is constrained by temperature (power dissipation) and and logic timing, which itself has a dependency on temperature.
So we could call that vertical scaling, to some power ceiling which may not take us all the way to 10x, but it's not impossible.
Then there is horizontal, which I assume is applicable to these problems... running more in parallel.
In both cases, I think it's safe to assume they are getting a performance increase in the instantaneous sense.
> You can make some assumptions though. If the power consumption was equal, the performance is 10x.
While I agree some performance per unit increase is likely, how does a direct 10x increased based on power savings follow? Less power usage does not mean that the chip can run through more flops in the same amount of time, right?
Also, it's not known that the TPU have a way to allow to increase the clockspeed arbitrarily, nor is it known whether their architecture is capable of ensuring correctness at arbitrary clock frequencies. Some architectures make assumptions like "The time for this gate to reach saturation is very small compared to the clock frequency, so we'll pretend that it's instantaneous."
Well, that's the kind of metric you'd expect from a cloud provider. That's what's important to them.
If you're a tinkerer dabbling in TPU acceleration on your gaming/coding PC alongside with GPU acceleration, then the metric that would be interesting for you is speed increase per unit.
In March at the GCP NEXT keynote [1], Jeff Dean demos Cloud ML on the GCP. He casually mentions passing in the argument "replicas=20" to get "20 way parallelism in optimizing this particular model". GCE does not currently offer GPU instances. I've never heard the term replicas in the GPU ML discourse. These devices may enable a type of parallelism that we have not seen before. Furthermore, his experiments are apparently using the Criteo dataset, which is a 10GB dataset. Now, I haven't looked into the complexity of the model or to what extent they train it to, but right now that sounds really impressive to me.
Since nobody has pointed this out, although he may be able to tell people "I was working on that thing", he likely can't say much more. Many large companies don't allow employees to disclose details that haven't been released publicly.
The article doesn't actually imply (though people might wish so) that the described TPU is intended to ever be a product available to others.
It's a post saying "that's how we do it" that serves a bunch of political and PR goals, but it's quite likely that this will stay an internal technology; maybe available indirectly as a cloud computing offering - in which case they'll give the specs and price of the whole solution, not of a particular model of TPU chip.
Not just lower precision but probably less error correction in the lower bits. -Or- ... they convert the floating point values to analog values and do all the math in the analog domain and convert it back, just like old analog computers. But by going fully analog, I think the gains would be over 10x better, so probably not.
They likely use half precision (i.e. 16 bit) floats in some of the computing stages. They've also discussed using even fewer bits of precision in some of their research output. When training and following an error gradient you kind of need lots of precision, but when running a learned model you often don't need much at all, 8 or even 4 bit numbers are sufficient in some calcs where rounding error propagation isn't an issue.
I've only dabbled in this area, so it's entirely possible my question doesn't make sense. Machine learning algos in general seems to have a bunch of iterative components, but within each iteration there is massive parallelism. Did you consider using a whole bunch of DSPs? They can also do parallel processing but might be cheaper in bulk than GPUs.
ASIC stands for "application specific integrated circuit" and the name conveys a special purpose design of an integrated circuit, like for example bit coin mining.
But that's just terminology/convention. One could argue a GPU is an ASIC, and a CPU is an ASIC. The only thing to argue is how specific does an application have to be to call it an ASIC instead of some other made up name.
The prime differentiators are the process, which is the term used for the many steps of fabrication of an IC (processes are often referred to as nodes, distinguished by the smallest feature size of a transistor they create), and the ability of the design engineers to create an optimal design, in terms of boolean logic and semiconductor physics.
Given the right team of engineers, and a top notch foundry, and a great deal of experience in the problem domain (machine learning in this case), a custom IC could very likely trounce a GPU.
GPUs were designed for the domain of graphics processing, which happens to have some commonality with the processing in machine learning. But, at least until recently, GPUs weren't focused on machine learning. Just graphics.
Now the GPU vendors are trying to leverage their knowledge of graphics processing and building of graphics processors to create machine learning processors, but the thing they are leveraging could also be what handcuffs them. Which gives opportunity for a company like Google to do a fresh take on the problem domain without the baggage of the knowledge of graphics processing.
What was your design flow? Did you use the standard cadence, Synopsys toolset to build this? How did you achieve basic design and JIT(or whatever it is you call the machine code part of the Python VM ) together?
It makes a lot more sense now why they open sourced Tensor Flow -- they have an ASIC design that supports it, and if they either want to sell the chips, or sell access to them via Google Cloud Platform, they'll need customers. And it's not like HP is about to roll out a server line sporting these any time soon, so Google will continue to have their advantage over the outside world.
For the curious, that's a plaque on the side if the rack showing the Go board at the end of AlphaGo vs Lee Sedol Game 3, at the moment Lee Sedol resigned and AlphaGo won the tournament (of five games).
There's a big difference between just cranking down the GHz and going for ASIC specialization. GPUs, for example, already represent a point on this spectrum -- it's true that they run at reduced GHz compared to high-end CPUs, but they're arithmetic monsters. The blog post notes, in fact, that the use of TPUs in AlphaGo let them do more searching. So why would you assume automatically that they're slow?
Because there's an absolute sippy straw of bandwidth to the thing if that's a 1x pci-e connection.
For if it were delivering performance on par with a $1000 Maxwell class GPU, why wouldn't you guys crow about it? That would be a really big deal wouldn't it? TitanX for 20W? That'd be awesome.
And having suffered through multiple pitches for us to buy various FPGA and boutique processors, I have yet to see someone who produced perf per watt numbers first, subsequently produce an impressive performance number. In fact, it took nearly yelling at one vendor for them to finally admit perf was abysmal.
Finally, training does not equal inference. Training requires strong scaling, but inference need only weak scale. So I suspect that Urs had to bite his tongue and buy a bunch of gpus for training networks.
That wasn't really my point - I'm simply noting that Urs is one of the last people I'd think to hop on the wimpy crazy train. His published articles suggest that he's got a very good grasp of the tradeoffs involved in "real" TCO -- i.e., taking a fairly global view of both the human, capital, and operating expenses involved in a technology decision such as using wimpies (no) or fabricating a custom ASIC for machine learning (yes).
That doesn't mean a TPU is faster or slower than anything in particular, it just means that quite likely that it's good for some machine learning tasks that Google cares enough about to spend the whatever dollars it cost to make the thing.
Apples and oranges. And as it turned out, it's an 8-bit fixed point processor. Useless for training without heroic handholding, limited use for inference. Google's PR machine does it again.
I'd be interested to know more technical details. I wonder if they're using 8-bit multipliers, how many MACs running in parallel, power consumption, etc.
Another point is that they will be able to provide much higher computing capabilities at a much lower price point that any competitors. I really like the direction that the company is taking.
Now these heatsinks can be deceiving for boards that are meant to be in a server rack unit with massive fans throwing a hurricane over them, but even then that is not very much power we're looking at there.
282 comments
[ 728 ms ] story [ 590 ms ] threadUsing it for over a year? Wow
I hope we can at least see some white papers soon about the architecture--I wonder how programmable it is.
So... just use Google's machine learning cloud thingy.
The software can build the community, where the supercharging is only available when you run it on Google cloud.
(although GPU performance isn't bad either, so you don't have to, thus community)
Machine learning isn't just targeting the AI researcher market though -- it's widely used by a huge number of companies, and of course, by many of Google's most important products. I would argue that those markets combined are larger than gaming.
Google is doubling down on hosting as a source of future revenue, and they're doing that by building an ecosystem around Tensorflow.
What I think is interesting is how weak Apple looks. Amazon has the talent and money to be able to compete with Google on this playing field. Microsoft is late, but they can, too.
Where's Apple? In the corner dreaming about mythical self-driving luxury cars?
[1]: http://spectrum.ieee.org/semiconductors/design/the-death-of-...
Where Apple really looks weak is in datacenters, networking, and cloud services.
I get the feeling from today's announcements that Google sees the 2021 version of Google Now as the selling point for their 2021 Nexus line.
I don't think Apple is preparing to compete on that.
Whether that's good or not may be arguable, but it's certainly a selling point for many and I don't see Google or any other company's offerings approaching the same experience, and I suspect that's by design; they have to be more open and support all devices but that kinda dilutes everything. Apple will only get stronger in that aspect IMO.
I’d hope someone somewhere steals the blueprints and posts all of them publicly online.
The whole point of patents was that companies would publish everything, but get 20 years of protection.
But by now, especially companies like Google don’t do so anymore – and everyone loses out.
EDIT: I’ll add the standard disclaimer: If you downvote, please comment why – so an actual discussion can appear, which usually is a lot more useful to everyone.
Re EDIT: Downvotes must be comment-mandatory or not allowed otherwise.
Look at how much faster ASICs for bitcoin mining are than the GPU... orders of magnitude.
Additional die space on additional functionality might hurt the power envelope (which is where the focus on performance / watt rather than performance kicks in) but it doesn't make your chips slower per se.
Furthermore the fact that ML can be error tolerant means you also get to optimize certain floating point operations for speed or energy efficiency at the cost of accuracy. NVIDIA doesn't get to do this in their linear algebra support.
There isn't much data yet but I'm also guessing they probably have access to much more RAM than NVidia cards and can process much bigger data sets
1. It works already (IE it's already in use)
2. It works really well (or else they wouldn't be using it so broadly)
3. Considering how long this was said to be in development, it also likely means they are working on the next big improvement before these guys have even gotten the current one working.
TPUs are custom ASICs that speed up math on tensors i.e. high-dimensional matrices. Tensors feature prominently in artificial neural networks, especially the deep learning architectures. While GPUs help accelerate these operations, they are optimized first and foremost for video rendering/gaming applications -- compute-specific features are mostly tacked on. TPUs are optimized solely for doing ML-related computations.
Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LOVE for them to market these for general use.
[0] If google is smart, they'd ditch +/- infinity and if they were ballsy, they'd ditch zero in their FP implementation.
For example -> https://petewarden.com/2016/05/03/how-to-quantize-neural-net...
In addition, there's a lot of literature on optimizing hardware implementations of fundamental arithmetic operations like addition and multiplication. I recall seeing a paper a while ago which talked about reducing the number of gates by allowing some bounded imprecision in the results - unfortunately, I don't remember the title right now, but it sounds like that's what they may be doing.
Link to the specialized hardware for SHA256 hashing: http://www.amazon.com/Antminer-~4-73TH-25W-Bitcoin-Miner/dp/...
AWSs offerings seem fairly vanilla and boring. Google are offering more and more really useful stuff:
- cloud machine learning
- custom hardware
- live migration of hosts without downtime
- Cold storage with access in seconds
- bigquery
- dataflow
- Cloud Shell
to name a couple more.
(I work on TF this year.)
(To elaborate -- it's questions like "how deep should I make this convolution? Should I use tf.relu or tf.sigmoid? How many fully-connected layers should I put here, and how big should I make them?". These are really knotty deep learning design questions, but they're often h/w independent. Not always - we certainly have some ops on TF that we only support in CPUs and not on GPUs, for example - but often.)
[1] It has been climbing the charts at https://github.com/soumith/convnet-benchmarks for example.
I read "Vanilla" and "Boring" as "Horray, I don't have to spend time rewriting all this complicated code I already have!"
If I'm just dipping my toes into (say) Caffe or Theano, I don't have to rewrite it from scratch.
That is a huge advantage---not a disadvantage!---of AWS over google.
Google does boring stuff very well too.. and one can argue much better than AWS as well.. take a look at Quizlet's story: https://quizlet.com/blog/whats-the-best-cloud-probably-gcp
(shamelessly biased Googler)
I'd love to hear what other boring stuff has been a showstopper for you, in case we missed something dumb :)
I'd love to hear what other boring stuff has been a showstopper for you, in case we missed something dumb :)
If you're evaluating something today, how does it change your decision that we were late to market with Compute Engine (and in this specific case "bring-your-own-kernel")?
If it's about future boring stuff, I think the list of boring stuff isn't too long ;).
Disclosure: I work on Compute Engine.
A solid guarantee with AWS is if AWS goes down, then a multitude of Amazon's services also will go down(ex Amazonian myself), so it gives me a belief that AWS's uptime is more important to Amazon itself that it is for external customers.
(Firebase Engineer here)
Before we had custom machine types (November 2015 GA), we wouldn't have been remotely close to what they needed. I'm not even sure we've had anyone evaluate the amount of overhead KVM adds in either latency or throughput.
tl;dr: Don't let Search be your "not until they do it". We've got folks in Chrome, Android, VR, and more building on top of Cloud (as well as much of our internal tooling being on App Engine specifically).
Not implying that AWS hasn't had them. It's just that adopting GCE this early makes you a bit of a guinea pig because GCE isn't used internally at Google.
- you can drop half the connections and it still works, in fact it works even better, during training
- you can represent the weights on as little as one bit, but still use real numbers for computing activations
- you can insert layers and extend the network
- you can "distill" a network into a smaller, almost as efficient network or an ensemble of heavy networks into a single one with higher accuracy
- you can add a fixed weights random layer and sometimes it works even better
- you can enforce sparsity of activations and then precompute a hash function to only activate those neurons that will respond to the input signal, thus making the network much faster
It seems the neural network is a malleable entity with great potential for making it faster on the algorithmic side. They got 10x speedup mainly on exploiting a few of these ideas, instead of making the hardware 10x faster. Otherwise, they wouldn't have made it the size of a HDD - because they would need much more ventilation in order to dissipate the heat. It's just a specialized hardware taking advantage of the latest algorithmic optimizations.
http://www.movidius.com/solutions/machine-vision-algorithms/...
TensorFlow on a chip....
http://www.catb.org/jargon/html/W/wheel-of-reincarnation.htm...
OT: Cool blog! :)
I don't think the average developer will need to understand chip design but I do think _many_ developers will need to know how to use deep learning frameworks.
Oh, and imagine a Facebook chip. :)
Then again, I've been coding for ~25 years, am an avid amateur astronomer, and have a degree in physics, so maybe the moral here is that some things just just take more time to master, beyond those first 6 weeks in a coding camp learning how to put together your first jQuery.
:-)
Maybe the next iteration will be assembly instructions specifically for Neural Nets built into CPUs...actually something like a convolution assembly instruction wouldn't even surprise me at this point.
http://uranium.vaxpower.org/~isildur/vax/week.html
IOW, taking their claims at face value, a Nvidia card or Xeon Phi would be expected to smoke one of these, although you might be able to run N of these in the same power envelope.
But those bandwidth & throughput / card limitations would make certain classes of algorithms not really worthwhile to run on these.
Agreed. Also tells you that they don't need to communicate with the CPU much, given that it only has a PCIE. Reminds me of Knights Ferry, in this respect.
> a Nvidia card or Xeon Phi would be expected to smoke one of these
Will be very interesting to see some head-to-head benchmarks between these guys (on tensorflow and other libraries) in the next few months. Especially as Knights Landing starts to appear, and the new Nvidia card.
That seems unlikely, right? GPGPU software beating an ASIC? I guess it depends on just how abstract and adaptable the TPU IC is. Sounds like they use a custom number representation -- if they can squeeze it into less than half precision (IEEE FP16) then it would be super hard for a Phi or GPU to beat it.
But ultimately it all comes down to specific applications' bottlenecks. If they don't have to go off-card ever and their workset fits into the on-die memory, then they'd have no real advantage by using a PCIe GPU with GDDRx on it.
Competitive advantage is protected by custom hardware (and huge proprietary datasets).
Everything else can be shared. In fact it is now advantageous to share as much as you can, the bottleneck is a number of people who know how to use new tech.
"Sun's two strategies are (a) make software a commodity by promoting and developing free software (Star Office, Linux, Apache, Gnome, etc), and (b) make hardware a commodity by promoting Java, with its bytecode architecture and WORA. OK, Sun, pop quiz: when the music stops, where are you going to sit down? Without proprietary advantages in hardware or software, you're going to have to take the commodity price, which barely covers the cost of cheap factories in Guadalajara, not your cushy offices in Silicon Valley."
Developers are a complement to hardware, software, and data based businesses. Google probably spends more on employees than on hardware or data. They really want to commoditize developers that can work on their stuff.
Google's business is based on having more and better data than their competitors. In many cases they have been happy to commoditize hardware, making open their datacenter designs for instance. That helps to cheapen their costs for building datcenters.
In other cases they open source software, like tensorflow and go language. These choices are made to commoditize developers. Google wants there to be a big pool of people who know how to use the technologies that Google uses. More developers means less costs for Google to hire and train employees. Which is their biggest expense: win!
With the TPU, as long as no one else is doing that kind of hardware for machine learning, it is a proprietary advantage to keep it secret. But at some point the logic flips: when others start to do similar things Google would rather commoditize their version of the tech. Because the lesson of the last 40 years is any widely used hardware WILL become commoditized. The inertia is with software codebases and developer knowledge.
Developers developers developers developer developers!
Maybe they play one move every time someone gets to go there to fix something? or could it be just a way of numbering the racks or something eccentric like that?
For the curious, Optalysys has built a general purpose optics-based correlation/pattern matching machine. From some of their predecessor-company marketing material: The correlator performs pattern matching on large data sets such as high-resolution images, providing a measure of similarity and relative position between objects within the input scene. This allows large images [and general data converted to images] to be analysed far faster than electronic equivalents.
Going back to the topic of NN-based computing, I found this talk to be intriguing: https://www.youtube.com/watch?v=dkIuIIp6bl0. The main argument is that because Moore's law may no longer be in effect, it will become increasingly important to explore alternate computing solutions. (Google's TPU could be supporting evidence for this argument.) The speaker also co-authored a paper which I liked "General-Purpose Code Acceleration with Limited-Precision Analog Computation".
If not, how would one practically implement an analog computer for neural network programming (without several tables full of op-amps?)
You can implement an analog neural network yourself using a Field Programmable Analog Array. (I've never done it, but you'll see academics online writing papers about it.)
Another thing that is sort of related is Lyric Semiconductor; they built these cool application-specific probabilistic processors; they were purchased by Analog Devices a while back.
http://www.cisl.columbia.edu/grads/gcowan/vlsianalog.pdf
Brain is a bunch of components that are spread out 3D that operate like a mathematical function at slow speed. Mostly sounds analog. Results in us. So, a huge spread of analog components could get some results directly simulating something like that. Here's one of my favorites which is a wafer-scale, analog computer for neural networks.
www.kip.uni-heidelberg.de/Veroeffentlichungen/download.cgi/4713/ps/1856.pdf
The primary cost factor in anything computing is just power. You can always buy more of the things, but power is the ongoing cost and every watt in computing costs you extra in cooling power.
I'm a bit surprised they announced this, though. When I was there, there was this pervasive attitude that if "we" had some kind of advantage over the outside world, we shouldn't talk about it lest other people get the same idea. To be clear, I think that's pretty bad for the world and I really wished that they'd change, but it was the prevailing attitude. Currently, if you look at what's being hyped up at a couple of large companies that could conceivably build a competing chip, it's all FPGAs all the time, so announcing that we built an ASIC could change what other companies do, which is exactly what Google was trying to avoid back when I was there.
If this signals that Google is going to be less secretive about infrastructure, that's great news.
When I joined Microsoft, I tried to gently bring up the possibility of doing either GPUs or ASICs and was told, very confidentially by multiple people, that it's impossible to deploy GPUs at scale, let alone ASICs. Since I couldn't point to actual work I'd done elsewhere, it seemed impossible to convince folks, and my job was in another area, I gave up on it, but I imagine someone is having that discussion again right now.
Just as an aside, I'm being fast and loose with language when I use the word impossible. It's more than my feeling is that you have a limited number of influence points and I was spending mine on things like convincing my team to use version control instead of mailing zip files around.
My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo was based on this hardware, they indicate to researchers around the world that if you want to build the most advanced ML models you need to be at Google. Same reason they talked about MapReduce/GFS back in the day.
Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.
Between that time and the Lee Sedol match, the hardware running AlphaGo was switched, to these TPU.
From the paper: "The final version of AlphaGo used 40 search threads, 48 CPUs, and 8 GPUs. We also implemented a distributed version of AlphaGo that exploited multiple machines, 40 search threads, 1,202 CPUs and 176 GPUs."
It's a bit underhanded, however. IMHO, the player should be able to study recent games before the match. But this is pretty typical, there were similar late-stage improvements with Chinook (checkers) and Deep Blue (chess).
Coming out saying they can give you a service no one else can right down to a custom chip may sway a few buyers in the market.
https://cloud.google.com/ml/
They've probably carefully modeled the energy savings over GPU/FPGA and found it to be substantial enough, even taking into account costs of design changes.
As for testing for correctness, ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.
It's actually very common to test ASIC RTL on FPGA. You are right that if you are targeting an FPGA you may write things differently, but this doesn't mean RTL written for an ASIC won't work it's just it might not clock as fast as it could and may not make the most efficient use of resources.
There is a lot of engineering work to take an ASIC targeted IP and run it in an FPGA, but thorough verification of an ASIC design is extremely important so it's usually worth the time to bring up an FPGA that can run it (sometimes as it's too large you have to split it across multiple FPGAs).
Especially if you try to make it go faster
You'll also break a design down into units or blocks. These can be tested separately. E.g. you might see an L1 cache as a single unit and have a testbench (piece of HDL to stimulate and check a design) to test only that. This is useful as it's far quicker than a full design test however extensive full design testing is still required as many bugs you see as the consequence of the interactions between units.
Once you've done a lot of testing you'll tape out a chip. This may well have some issues so you'll do a new 'spin'.
Obviously details and plans how it was done is where all the good stuff is, so that being hidden is understandable.
Anyone the casually follows AI knows that people have been talking about making DNN ASICS for some time. It was all a matter of time and $$$$$$
There is no doubt FB is working on them too. Which is why Google is finally publicaly saying that "we did it first ;)"
Makes sense. In that respect yeah, they probably wanted to keep it under wraps to avoid Facebook/others from getting a timeline estimate out of it and jump ahead.
That's kind of what Movidius says, too, about its Myriad 2 VPU, which is kind of a GPU (SIMD-VLIW) with larger amounts of local memory combined with hardware accelerators.
It's a "more details to follow" type of thing. Pretty standard actually.
They say 10x performance / watt, nothing about performance per unit time.
The speed at which an ASIC will run is constrained by temperature (power dissipation) and and logic timing, which itself has a dependency on temperature.
So we could call that vertical scaling, to some power ceiling which may not take us all the way to 10x, but it's not impossible.
Then there is horizontal, which I assume is applicable to these problems... running more in parallel.
In both cases, I think it's safe to assume they are getting a performance increase in the instantaneous sense.
While I agree some performance per unit increase is likely, how does a direct 10x increased based on power savings follow? Less power usage does not mean that the chip can run through more flops in the same amount of time, right?
http://electronics.stackexchange.com/questions/122050/what-l...
(see graph in the first answer)
Also, it's not known that the TPU have a way to allow to increase the clockspeed arbitrarily, nor is it known whether their architecture is capable of ensuring correctness at arbitrary clock frequencies. Some architectures make assumptions like "The time for this gate to reach saturation is very small compared to the clock frequency, so we'll pretend that it's instantaneous."
Well, that's the kind of metric you'd expect from a cloud provider. That's what's important to them.
If you're a tinkerer dabbling in TPU acceleration on your gaming/coding PC alongside with GPU acceleration, then the metric that would be interesting for you is speed increase per unit.
1: https://youtu.be/HgWHeT_OwHc?t=2h13m6s
It's a post saying "that's how we do it" that serves a bunch of political and PR goals, but it's quite likely that this will stay an internal technology; maybe available indirectly as a cloud computing offering - in which case they'll give the specs and price of the whole solution, not of a particular model of TPU chip.
See 'low-power (inexact|approximate) computing'
A bit old but cf. DE Shaw and Anton https://en.m.wikipedia.org/wiki/Anton_(computer)
and Anton wasn't really "scale" in this sense. It was a vertically scaled single (or several) machines.
But that's just terminology/convention. One could argue a GPU is an ASIC, and a CPU is an ASIC. The only thing to argue is how specific does an application have to be to call it an ASIC instead of some other made up name.
The prime differentiators are the process, which is the term used for the many steps of fabrication of an IC (processes are often referred to as nodes, distinguished by the smallest feature size of a transistor they create), and the ability of the design engineers to create an optimal design, in terms of boolean logic and semiconductor physics.
Given the right team of engineers, and a top notch foundry, and a great deal of experience in the problem domain (machine learning in this case), a custom IC could very likely trounce a GPU.
GPUs were designed for the domain of graphics processing, which happens to have some commonality with the processing in machine learning. But, at least until recently, GPUs weren't focused on machine learning. Just graphics.
Now the GPU vendors are trying to leverage their knowledge of graphics processing and building of graphics processors to create machine learning processors, but the thing they are leveraging could also be what handcuffs them. Which gives opportunity for a company like Google to do a fresh take on the problem domain without the baggage of the knowledge of graphics processing.
Rrally interested to know how this was managed...
Let's see this sucker train AlexNet...
Google's always been cautious about the balance of speed and efficiency, out of concerns about programmer productivity, parallelization, and generality. See, for example, Urs's article in response to my and a few other people's crazy-academic research on using "Wimpy" nodes: http://static.googleusercontent.com/media/research.google.co... - vs http://www.cs.cmu.edu/~fawnproj/
There's a big difference between just cranking down the GHz and going for ASIC specialization. GPUs, for example, already represent a point on this spectrum -- it's true that they run at reduced GHz compared to high-end CPUs, but they're arithmetic monsters. The blog post notes, in fact, that the use of TPUs in AlphaGo let them do more searching. So why would you assume automatically that they're slow?
For if it were delivering performance on par with a $1000 Maxwell class GPU, why wouldn't you guys crow about it? That would be a really big deal wouldn't it? TitanX for 20W? That'd be awesome.
And having suffered through multiple pitches for us to buy various FPGA and boutique processors, I have yet to see someone who produced perf per watt numbers first, subsequently produce an impressive performance number. In fact, it took nearly yelling at one vendor for them to finally admit perf was abysmal.
Finally, training does not equal inference. Training requires strong scaling, but inference need only weak scale. So I suspect that Urs had to bite his tongue and buy a bunch of gpus for training networks.
Am I missing something?
That doesn't mean a TPU is faster or slower than anything in particular, it just means that quite likely that it's good for some machine learning tasks that Google cares enough about to spend the whatever dollars it cost to make the thing.
The WSJ article has a few more quotes from Norm Jouppi, btw.: http://www.wsj.com/articles/google-isnt-playing-games-with-n... (Sorry if that gets paywalled. Googling "wall street journal google tensor processing unit" got me there.)
Now these heatsinks can be deceiving for boards that are meant to be in a server rack unit with massive fans throwing a hurricane over them, but even then that is not very much power we're looking at there.
No wondering where you left the Torx drivers with this one.