The mean transduction in the sense of sequence-to-sequence models (transducing one sequence to another).
Attention based models don't necessarily need to be sequence to sequence. They can be classifiers, decoder only, etc. Attention is just one tool in the ML architecture toolkit.
Patent lingo is necessarily more generic than the lingo used within the machine learning space. For example, within patents, they will talk about using computers to do things, which is obvious outside of the patent world. But when writing a patent, you have to actually mention that you're going to use a computer to implement a neural network.
Should probably put a (2019) in the title. Furthermore considering how fast the ML space moves, the fact that google hasn't used this to create a model significantly better than competitors seems to show that the patented architecture did not perform better than others.
Or, it shows that Google can’t deliver products in this space to (literally) save its life, no matter how good its technology is (which they’ve also shown lots of other ways.)
Do you think Google has 8 billion products in the AI space?
Because it seems pretty clear that the criticism is that Google is an advertising company that uses search as leadgen, and is struggling to ship AI products.
The patent was granted in 2019. The patented architecture is the Transformer architecture that everyone and their mom is now using for LLMs. Patents in your portfolio help defend against trolls and other hostile entities.
Consider Amazon One-click purchase. That patent was one of very few (out of tens of thousands) that they defended heavily. Every other company on the planet had to have at least two steps to purchase an item or Amazon would come knocking with lawyers. They did this until the day that patent finally expired.
One has to ask: was that to the benefit of society?
Google may make good use of this tech. But it would be better for all of us if everyone did and didn't have to pay them a fee for the privilege.
100% it’s probably worth it in the end. You quite literally may not have had Amazon without it.
In this case, google might not have funded the development of transformers which could dramatically reduce costs of everything for humanity.
Same goes for drug development.
That said, I think there’s a question around software patents and how long they should last. Perhaps it should just be to recoup costs, plus some multiple. I’m not sure.
We may not have had Amazon if other online stores had been allowed a one-click purchase flow? The lack of a second click was a unique innovation critical to their competitive advantage?
That doesn't seem likely to me. And if it were true that Amazon was so vulnerable that checkout flow click parity would wipe them out, I hardly think the law should intercede to save them - that seems like a scenario where they had a bad business model.
I went to a talk done by Amazon's longest serving IP lawyer and he explicitly called out that Amazon has only gone to court (or asserted its patent, can't remember the phrase) over a kindle related patent.
I think this patent is invalid based on the disclosure date.
The attention is all you need paper was on Arxiv on 12 Jun 2017 and you have to patent within one year. In this case June 28th 2018 is too late..
Apparently, they thought of that, Line 3 of the patent application US10452978B2:
Provisional Application No. 62/510,256, filed on May 23, 2017
There are however other patents and applications:
Neural machine translation with latent tree attention
US20180300317A1 James BRADBURY
I'm not an expert so I can't read the spec and claims as to relevance with authority. However, it still wouldn't count directly as prior art as it was published in Oct 2018. Patents are now first to file and not first to invent (to match the rest of the world).
edit: They do also site non-patent prior are regarding attention. Whether this covers self attention I'm not clear and of course their claims have been reviewed by the examiner in light of the art so they're presumed valid until re-exam.
Luong et al. "Effective approaches to attention based neural machine translation," arXiv 1508.04025v2, Sep. 20, 2015, 11 pages.Luong et al. "Effective approaches to attention based neural machine translation," arXiv 1508.04025v2, Sep. 20, 2015, 11 pages.
> However, it still wouldn't count directly as prior art as it was published in Oct 2018. Patents are now first to file and not first to invent (to match the rest of the world).
There is an investor's grace period, as long as your own public prior art disclosure is the earliest, you get 1 year to file even under first to file. First to file refers to people filing for undisclosed inventions, disclosure acts as prior art against anyone else filing, along with the grace period for delaying your filing.
Often "wow the company detailing internals at this conference is so generous!" is actually them getting an extra year on the of patent expiration date by taking advantage of the grace period.
- has models years ahead of everyone else (e.g video ones)
They sure are likely to be a filure, they're just late to the party. Keep in mind Apple is always late to the party.
While I would agree with you the CEO of Google doesn't know what he's doing, Google has the best of everything to succeed in the race. It is incredible they are doing this all openly and sharing their research with everyone while doing it.
The complaint with OpenAI is they are going too fast (the 6 months letter..) with Google they were going at a responsible speed - which is where in a game theory fight with someone else going faster, would leave them in the position they are today in terms of perception, but not the loser.
Can’t wait to see the patent system break under AI. It’s time for patents to go away or at least lower the timeframes of protection so we don’t stifle the innovation waves that are inevitably coming.
Whole areas will be off limits and dominated by a few companies — the patent system is a system for the olden times.
The patent system unfairly enforced incumbent advantage and needs to be reformed.
I'm very curious to see if patents/copyright/IP just become "washable" with AI making them near useless.
Given the recent determinations, it almost sounds like since it won't be human made, it might not be protectable? Not to mention the exceptions generally made for modifications to the IP to improve it.
People are going to work around this. Midjourney generates an image, but then I change the opacity 0.01% and so it becomes copyrightable (or whatever the smallest amount of human modification is required).
> but then I change the opacity 0.01% and so it becomes copyrightable
AFAIK Courts generally have determined it must be "transformative" for fair use, and for patents - but like a substantive change. You can't just add a pixel in a corner and call it good.
Do you mean like, "Here is an algorithm for attention-based sequence transduction neural networks. Come up with an algorithm that is as similar as possible and produces the same outputs, but is unencumbered by patents"?
Alice would like to have a word. That case (Alice Corp vs CLS Bank) makes it hard to have a software patent, particularly if that software has no physical effect on the world.
Code is a specific expression covered by copyright, not an incredibly broad claim that covers everything from toasters to spacecraft.
If the code's algorithm is patented, you cannot get around the patent by using different function and variable names and perturbing the code organization.
Worth noting that you can't just get around a piece of code's copyright by using different function and variable names and perturbing the organisation either. At least not if you're a human.
I bet you're correct and we wont be able to use ML models to strip code of its patents the way we use them to strip code of its copyright protection. But I wouldn't have expected the copyright stripping trick to work either, yet here we are.
You can't legally get around copyright with technicalities, because any bit string can be derived from any other bit string by a sequence of one bit edits, and according to the legal doctrine, if we start with a copyrighted work, each such edit is creates a derived work.
You can disguise borrowed code such that, in the first place, nobody will suspect that it's derived from anything, and even then, they won't have evidence that it's derived.
Changing a few names and superficial organization details, on a large and significant work, will likely not fool anyone.
If it's a small module or just one function, quite probably easily so. You don't even have to understand how the code works; if you just know the paper from where the original author got the code, you can just say you implemented the detailed requirements there and validated it on the available test vectors.
This only works for copyright. Patents don’t protect parallel inventions. And trademarks cover anything that could confuse a potential customer, even unintentionally.
Yeah definitely doesn't cover trademarks as they are for something else like you said.
To my understanding patents do allow for either "improvements" or anything that achieves the same result as long as it's not the same solution as the patent?
They would certainly be a lot trickier then copywrite/IP but I think LLM's would still be able to generate possible solutions? One thing I'm thinking of for example is medication analogs - there's common substitutes you can make that achieve the same or better results that you can make.
I'm not so sure it's going to break. Google may have a hard time enforcing this one thanks to Alice, but we will see if they try. Given that there has been no lawsuit against OpenAI, I'm guessing that they aren't planning to.
It’s cool, I got -2’d and youre still up, I misread the room. Just hope it’s not the future here, no reason why other people shouldn’t post ones as a wizard, orc, crypto ceo…
There's a sort of circular logic there: the alchemist one is fine because people wanted it, the other ones aren't fine because people don't want it. I prefer an HN that ends up skipping "explain [ARTICLE] as $X" altogether unless $X is on-topic
Honestly this kind of stuff is already boring. It's an extremely simplified explanation, and overall a pretty low effort post. Don't get me wrong, ChatGPT is amazing, and I had it write me some recipes in Plato's voice. But now the novelty has worn off and this is essentially throwaway content. It's not contributing anything to the discussion.
Looking over the patent, I think they would really struggle to enforce this against someone using a decoder only transformer. Which is basically everyone at this point.
EU? what are you talking about? The patent is registered in USA. the only country missing from the list as far as i see is actually the EU. Or did you mean USA instead of EU?
As the lawyer behind the one-click patent once personally explained to me, it's the claims that matter most. And the first claim matters more than all of the other sub-claims.
This patent specifically covers ONLY transformers in which there is an encoder and a decoder.
Claim 1 of the patent contains the following:
"...the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a respective encoded representation of each of the network inputs in the input sequence... and a decoder neural network configured to receive the encoded representations and generate the output sequence."
Claims 29 and 30 (the only other independent claims) also specify an encoder and a decoder. So long as your transformer network does not make use of an encoder in combination with a decoder, this patent does not apply to you.
"1. A method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each of a plurality of generation time steps:
generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence."
101 comments
[ 2.3 ms ] story [ 154 ms ] threadAttention based models don't necessarily need to be sequence to sequence. They can be classifiers, decoder only, etc. Attention is just one tool in the ML architecture toolkit.
2019-10-22 - Application granted
Should probably put a (2019) in the title. Furthermore considering how fast the ML space moves, the fact that google hasn't used this to create a model significantly better than competitors seems to show that the patented architecture did not perform better than others.
Because it seems pretty clear that the criticism is that Google is an advertising company that uses search as leadgen, and is struggling to ship AI products.
One has to ask: was that to the benefit of society?
Google may make good use of this tech. But it would be better for all of us if everyone did and didn't have to pay them a fee for the privilege.
This is reverse logic because then Google would not have published the tech and would have kept it a trade secret.
In this case, google might not have funded the development of transformers which could dramatically reduce costs of everything for humanity.
Same goes for drug development.
That said, I think there’s a question around software patents and how long they should last. Perhaps it should just be to recoup costs, plus some multiple. I’m not sure.
That doesn't seem likely to me. And if it were true that Amazon was so vulnerable that checkout flow click parity would wipe them out, I hardly think the law should intercede to save them - that seems like a scenario where they had a bad business model.
I went to a talk done by Amazon's longest serving IP lawyer and he explicitly called out that Amazon has only gone to court (or asserted its patent, can't remember the phrase) over a kindle related patent.
edit: They do also site non-patent prior are regarding attention. Whether this covers self attention I'm not clear and of course their claims have been reviewed by the examiner in light of the art so they're presumed valid until re-exam.
There is an investor's grace period, as long as your own public prior art disclosure is the earliest, you get 1 year to file even under first to file. First to file refers to people filing for undisclosed inventions, disclosure acts as prior art against anyone else filing, along with the grace period for delaying your filing.
Often "wow the company detailing internals at this conference is so generous!" is actually them getting an extra year on the of patent expiration date by taking advantage of the grace period.
- cornered the market on AI researchers
- has the most researchers
- the best researchers
- has a head start on the GPU issue with TPUs
- has patents on the models everyone else uses
- already has distribution
- developed the models everyone else uses
- has models years ahead of everyone else (e.g video ones)
They sure are likely to be a filure, they're just late to the party. Keep in mind Apple is always late to the party.
While I would agree with you the CEO of Google doesn't know what he's doing, Google has the best of everything to succeed in the race. It is incredible they are doing this all openly and sharing their research with everyone while doing it.
The complaint with OpenAI is they are going too fast (the 6 months letter..) with Google they were going at a responsible speed - which is where in a game theory fight with someone else going faster, would leave them in the position they are today in terms of perception, but not the loser.
Whole areas will be off limits and dominated by a few companies — the patent system is a system for the olden times.
The patent system unfairly enforced incumbent advantage and needs to be reformed.
Given the recent determinations, it almost sounds like since it won't be human made, it might not be protectable? Not to mention the exceptions generally made for modifications to the IP to improve it.
AFAIK Courts generally have determined it must be "transformative" for fair use, and for patents - but like a substantive change. You can't just add a pixel in a corner and call it good.
https://www.justia.com/intellectual-property/patents/types-o...
Slightly different (was for getting a patent for ai work) - https://cdn.arstechnica.net/wp-content/uploads/2023/02/AI-CO...
If the code's algorithm is patented, you cannot get around the patent by using different function and variable names and perturbing the code organization.
I bet you're correct and we wont be able to use ML models to strip code of its patents the way we use them to strip code of its copyright protection. But I wouldn't have expected the copyright stripping trick to work either, yet here we are.
You can disguise borrowed code such that, in the first place, nobody will suspect that it's derived from anything, and even then, they won't have evidence that it's derived.
Changing a few names and superficial organization details, on a large and significant work, will likely not fool anyone.
If it's a small module or just one function, quite probably easily so. You don't even have to understand how the code works; if you just know the paper from where the original author got the code, you can just say you implemented the detailed requirements there and validated it on the available test vectors.
To my understanding patents do allow for either "improvements" or anything that achieves the same result as long as it's not the same solution as the patent?
They would certainly be a lot trickier then copywrite/IP but I think LLM's would still be able to generate possible solutions? One thing I'm thinking of for example is medication analogs - there's common substitutes you can make that achieve the same or better results that you can make.
To my understanding redbull actually did this (without ai) to modafinil with this patent - https://patents.google.com/patent/US20210380545A1/en
EDIT: Modafinil might have expired but it looks like redbull filed their patent before the expiration.
It's obviously math.
This whole "computer implemented invention" workaround is a complete sham.
To think that the EU wastes billions annually on this broken institution, while completely failing to properly fund startups is simply infuriating.
See: https://patents.google.com/patent/EP3542316B1/en
You can look at their supplied diagrams and general summary to confirm.
This patent specifically covers ONLY transformers in which there is an encoder and a decoder.
Claim 1 of the patent contains the following:
"...the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a respective encoded representation of each of the network inputs in the input sequence... and a decoder neural network configured to receive the encoded representations and generate the output sequence."
Claims 29 and 30 (the only other independent claims) also specify an encoder and a decoder. So long as your transformer network does not make use of an encoder in combination with a decoder, this patent does not apply to you.
"1. A method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence."