"While we built next-generation experiences for developers, our business failed in two important ways.
First, we failed to deliver our vision of AI-assisted programming because we were 10+ years too early to market, i.e. the tech is not ready yet.
We built the most-advanced AI for helping developers, but it fell short of the 10× improvement required to break through because today’s state of the art for ML on code is not good enough. You can see this in Github Copilot, which is built by Github in collaboration with Open AI. As of late 2022, Copilot has a number of issues preventing it from being widely adopted.
The largest issue is that state-of-the-art models don’t understand the structure of code, such as non-local context. We made some progress towards better models for code, but the problem is very engineering intensive. It may cost over $100 million to build a production-quality tool capable of synthesizing code reliably, and nobody has tried that quite yet.
Nonetheless, we could have built a successful business without 10×’ing developer productivity using AI, and we did not do that.
We failed to build a business because our product did not monetize, and it took too long to figure that out."
Netscape open-sourcing their code is what led to Firefox and an open Web, as a counterweight to closed source browsers. Safari took WebKit from Konqueror
Mind that they continued working on it and built Netscape versions on top of Mozilla/Phoenix/Firebird/Firefox. This people with knowledge of the code base and domain pushed it along.
While this maybe true for FAANG and other tech giants, it is unfortunately still a lot of money for most startups that are working their way to IPO.
Tbh even at tech giants that make tens of billions of dollars, a $100 million investment is likely a lot, I'm guessing this sort of investment will require sign-off by CEO or at least VP level along with a solid business plan.
How many working hours, tons of materials will it take?
Could they be used on more relevant needs to humanity?
If so, why use them on a moonshot we will die before humanity achieves? Essentially approving burning up resources on unverifiable, perhaps unachievable outcomes?
We still fund a lot of agency to iterate on high minded potential as if future humans have an immutable obligation to carry on the work.
Secular norms and justifications are not sacrosanct. The dollar is a made up token of power.
Kite rejected me for a position years ago which motivated me to go raise $1M from the NSF to research AI-based dev tools before I moved on to Microsoft.
They seemed like a really cool team, I wish them the best.
How did you go about that NSF fund raise? Was it a SBIR? Did you already know how to fundraise through SBIR or whatever vehicle you used? Were you funding just yourself or a team? What ended up happening to the thing you fundraised from NSF?
> it fell short of the 10× improvement required to break through because today’s state of the art for ML on code is not good enough. You can see this in Github Copilot, which is built by Github in collaboration with Open AI. As of late 2022, Copilot has a number of issues preventing it from being widely adopted.
True, AI assisted coding does not deliver 10x. But as a user of another AI assistant, I feel that it gives me ~1.25x to ~2x improvement for the keyboard typing when I code. And that is respectable too :) AI for me currently allows me to tab complete some things that previously an IDE on its own was not able to.
> The largest issue is that state-of-the-art models don’t understand the structure of code, such as non-local context.
Depending on how local he's talking, this isn't really true of Copilot. In my experience it will use context all the way up to the top of the file, even in very long files. And at least the Rust version even seems to look at the imports—if you have a use declaration it will actually correctly build and use structs in other files regardless of whether you've yet used them in the current file.
I had a similar thought. I think in this instance, they mean something like a labeled/supervised training approach where the model is given not just the tokenized code, but also perhaps is grounded in the possible structure (indentation vs. semicolons, function-scope, etc.).
My understanding is that copilot is largely a self-supervised approach. They feed a massive body of (somewhat noisy) code into the model. The model really does learn a lot of structure on its own and this is a testament to deep learning on noisy datasets.
I'm guessing the "hooks" that they have already from IDE's, language-servers, etc. _are_ quite "structure-aware" - so they want the predicted structure as well as the code, so they can improve the typing experience beyond line-completion.
I think the estimation of 100 million for such a task is maybe too high? I don't know - it feels like you could actually get quite close to such a system by simply using thousands of custom prompt engineering tricks that prepend structure examples to the prompt?
Yes, this was precisely what I was referring to. In small-enough programs (e.g. one file) Copilot has all the context. The other extreme would be something like the Chromium codebase. Because of this, Copilot looks better in quick demos than real-world use. (Though of course it is very impressive and this tech will get there, hopefully very soon!)
But what I'm saying is that it does use imports, at least in Rust. I'm assuming that somehow behind the scenes they're concatenating the contents of the imports into the prompt.
I can imagine this is easier in a language like Rust that has a really strict module system, and to be fair the project that I've been using it on is a side project that isn't over 10,000 lines of code yet. If I were up to 30 imports per file I can imagine concatenating would become much less effective.
Does it seem to only understand imports of public libraries? If so, it's likely that, rather than understanding the contents of those libraries, it's learning from others' use of those library APIs. If not, it is likely just understanding the words in the API at a shallow depth.
No, it's imports from other files in my project. It's either using the import or the fact that I have another tab open.
There are definitely times where it produces a close approximation that's obviously just statistical, but there are other times where there's no question that it picked up something from a different source file that couldn't have possibly been in its training set.
I haven't yet decided if it's using imports or opened files in the editor, but it's definitely not just using the single file I have active.
It could be doing some "fine tuning" based on the repo. That would be cool! That said, what I meant when referring to 'understanding' the non-local nature of code was in a more principled way.
For example, if an object defined in another file has a function called `rename` that takes zero arguments, when calling it from another file Copilot will likely suggest arguments if there are variables like `old` and `new` near the cursor, even though `rename` actually doesn't take any, just because functions called `rename` typically take arguments. This behavior is in contrast to a tool like an IDE that can trace through the way non-local code references work.
Condolences to the Kite team. But, congratulations, too - you have some of the highest value engineering experience in the world. I'm sure you'll land somewhere great; try and take some time off if you can afford it!
Mulling over business models, and noticing the 'devs won't pay' narrative in the blog post, it's interesting to see the existing business models in AI; basically they seem to be:
* API-driven cloud calls (this is a way to get high value out of your existing cluster if you're AWS, MS, etc.)
* Platform play + possible eventual lock-in: OpenAI/Microsoft
* Subscription service for very specific needs (Grammarly, writing support)
I wonder if engineers would pay $9.99/month (or even $49.99/month) for a 'grammar checker for PRs' - essentially: "Avoid embarrassing bugs before you commit". That is, I wonder if Kite could have been successfully sold as the third tier - sub service for something very specific.
I guess if it's a good idea, someone could pull the Kite repos and launch it -- but my guess is there may be a market in there.
Devs almost always lack any kind of purchase authority. Any tool that appeals to devs needs to appeal to their management more, either showing some kind of cost savings over existing tools, increased dev productivity, or the new fangled "dev experience" where this tool, by shear awesomeness, will let devs put aside the low salary, process hell, and keep them employed.
I understand that's Kite's perspective (and yours -- "purchasing won't pay for this"), but devs are not paid meager salaries in general, and definitely might care about their code quality when it's put out in 'public' whether that be internal repos, or github.
Payscale estimates average engineering salary as having between $3,000 and $7,000 a month more in disposable income over writers -- and I would guess almost every professional writer pays for grammarly.
But, I agree that this is a new concept, and just spitballing -- right now, these sorts of linters and code formatting tools are mostly open source, so it would be some product marketing work to see if the market would actually pay.
Like everything in most companies, it's never about money, it's about control/power. It's inevitably someone's job to manage all the SaaS shit at a company, and damnit we just bought a JetBrains license why do you need Copilot?
> I understand that's Kite's perspective (and yours -- "purchasing won't pay for this"), but devs are not paid meager salaries in general
Perhaps an under-appreciated perspective – most of us don't come from families that were paid the sort of salaries we get; and consequently we are more likely to be irrationally stingy about money as a result of having grown up (relatively) poor. I know that $200/year is not a significant cost at my salary; but I do remember a time when that would have been totally out of reach for me.
And also there's the fairness aspect – do I want to be participating in a system where only rich devs can get access to good tools? Yes, I understand that the world is not fair and already works that way, but I should probably not perpetuate that system; and instead, make work pay for it.
If you don't have a massive platform (ie if you aren't Amazon or MS) the vertical end to end solution is much more easy to sell - ie sell the benefit directly.
Thank you for open sourcing your startup. I’m sorry it didn’t work out. I think you deserve a big congratulations for being the first to really go after this problem. It’s a correct problem — it’s a big market and the solution will come eventually — I’m just sorry it turned out to be too gnarly to solve for you right now! I would have loved for it to have worked out better.
I agree that Kite didn’t deliver the 10x. I was an early user and tried hard to use it but didn’t find the benefit compelling enough to drop into my workflows, but it was very exciting.
I’m sure I speak for all of HackerNews when I wish you the best for whatever is next for the team.
> We failed to build a business because our product did not monetize, and it took too long to figure that out.
This is the one-sentence summary about why the business failed, but it's kind of a strange way of putting it.
I am dead sure that there were plenty of advisers along the way who told the company's executives that its monetization plan was weak and unlikely to succeed. But everyone assumes that they'll be the exceptional case.
"It took too long to figure that out" makes it seem like the most likely scenario wasn't staring them in the face the whole time.
> take over (something) and use it for a different purpose.
You are misleading readers in order to promote your agenda. You clearly speak perfect English, so you know what hijack means. "take over (something) and use it for a different purpose." is not found as a definition of "hijack" in any dictionary. "Hijack" implies "unlawfully" or "without having a right to do so".
Of course, every word can be used in a slightly different meaning; for example, in software can (harmlessly) hijack an entity (circumventing the usual API for expediency or performance). Such broadened semantics is perfectly fine when there's no confusion about the meaning. Very clearly in the case of OP, there was a clear intention to imply "unlawful" or "without having a right", so this exception doesn't apply.
The sad thing is that I actually support your agenda. I just don't support promoting it through misleading statements.
> : to take or take control of (something) as if by hijacking
> often, specifically : to change the topic or focus of (something, such as a conversation) : REDIRECT
It wasn't my statement by the way, I just figure if you're going to nitpick you should at least be correct about the nit.
Hey if we're going to talk language, maybe you should just use 'Kife', it looks like Kite and means to steal. (Allegedly derived from Old English word 'kip', net says it's British slang, but I've heard it a few times in northeast US.
"Our diagnosis is that individual developers do not pay for tools. Their manager might, but engineering managers only want to pay for discrete new capabilities, i.e. making their developers 18% faster when writing code did not resonate strongly enough."
I never used Kite, but I've tried Github Copilot twice, and found it marginal at best (and distracting at worst - which is why I turned it off both times). If Kite was similar, the reason I'm not paying is that coder AIs are not providing any value.
Developers are somewhat reluctant to pay for tools but I think you can get them to pay for things that are worth it. I've been paying for code editors for years.
Loads of devs I’ve spoken to, from junior to principal level, absolutely love Github Copilot though. Don’t know who is paying it for them, nor if Kite was significantly worse, but I think that at least Copilot has a brilliant future ahead of it.
I _love_ Copilot but the only reason I use it is because I qualify under their open source developers program, I just can't justify paying $10/month for it.
It saves a bit of time, but doesn't seem to make a difference on time to market of features, products, improvements or bug fixes.
In my experience, it's a quality of life improvement, but the things that dictate actual time to market is bottlenecked by things that aren't solved by copilot, such as overall design, decision making, requirements gathering, code structure/architecture, solution ideation, user acceptance, infrastructure setup, etc.
I think if it eventually could help with those other tasks, you'd see time to market gains, and that would start to make it really valuable.
If Copilot generates the exact same code as the source, I don't see how that process could be exempt, it's like using the clipboard on your PC with extra steps.
1) It's the code license that matters, and the depends on what license the code the developer is building is using
2) License compatibility is undecided (see eg the different views of Apache Foundation and FSF on the compatibility of ASF 2.0 and GPL)
3) It's easy for someone to deliberately produce code that violates a license. MS is a big target and you can bet license trolls would chase it on that.
3) is not a good reason. "Big fish" is not an excuse to not provide protection. The burden to protect others from harm as a direct result of a feature of a product is on the owner of that product.
Do you mind explaining why? What do you think will happen?
This is a serious question, I'm apparently just unaware of the horror-stories that can come out of breaking copyright. (Not from the US so..)
For me its most useful for helping with bash scripts and small simple stuff, just saves a huge amount of time I would spend googling and checking small things. Not sure if copyright is relevant there or not, it certainly isn't something I am worried about. Interested to hear what your fears are based on.
ML text and image generators have a habit of regurgitating training data, especially if you happen to use certain inputs or prompts that also match said data even superficially. For example, type in "fast inverse square root"[0] and get out a bunch of Quake source code.
If you use code or art generated by an AI that was regurgitating training data, you can be sued for copyright infringement.
The way that AI gets training data these days is... questionably ethical. It's all scraped off the web, because the people who make these AIs saw court precedent for things like Google Books being fair use and assumed it would apply to data mining[1]. Problem is, that does nothing for the people actually using the AI to generate what they thought were novel code sequences or images, because fair use is not transitive.
[0] This won't work in current Copilot because a) I'm misremembering the comment phrasing and b) they explicitly banned that input from generating that output.
So, this makes sense from the perspective of writing code professionally for a company. I don't buy that the risk is significant for an individual programmer, but I suppose it is plausible that lawsuits could pose a threat to a large organization.
But I find I get most pleasure out of using CoPilot for personal tasks, everyday stuff that isn't for work. Surely there is no issue there?
And to counter the note about the scraping sources-- I have recently (out of work) been experimenting with using CoPilot to translate open-source MATLAB code into python and zig. If I make something useful in this way, it will be open source and I will credit those who wrote the original code.
Surely a use-case like that last would have to tick all the moral and legal boxes?
I use it. It's good for boilerplate code or basically anything where you can avoid looking at the docs. For example, I was writing an ML training loop and it correctly filled in the rest of the function after I wrote the first few lines. The code is basically what's in the pytorch docs, just fit to my model and scenario.
It doesn't block your ability to learn any more than any auto suggestion systems. But I guess it depends on what you value in terms of learning, for me, copilot allows me to focus on the larger architectural problems while not having to worry about the exact syntax of certain things (DSL query language, middleware express, typescript def annotations, etc).
Every time I don't have to context switch to look up some technical errata in my browser is a complete win for me.
I observed one guy become very resistant to reading the docs as they believed copilot obsoleted them. Programming in Ruby, the copilot suggestions will run but since you didn't read the docs you won't know that the output doesn't actually work or doesn't do what you think it does since it was intended for a different situation.
Everyone I've spoken to who has actually used it thinks it works extremely well. This includes pretty experienced developers.
I've never heard anyone claim it blocks the ability to learn - if anything it's the opposite. Many people like how it shows you APIs you weren't aware of.
But I did find that I need to turn it off at certain stages when learning (or re-learning) a programming language. It's seemed counterproductive until you have a good grasp on the basic syntax of the language. But the "showing you new APIs" does seem to be a thing that actually helps.
In general, you should not be accepting code completions that you don't understand. I'm usually stricter, in that I only typically accept completions that line up with what I was planning to type anyway.
It's quite poor for learning a totally new set of programming language grammar.
If you're a Javascript developer of 5+ years of experience, it's probably absolutely fantastic. Not so great if you're learning a new programming language.
I used it for about two weeks total, over two different periods. Mostly Kotlin, Java, Typescript, small amounts of Groovy. I ended up turning it off. In my experience, it has moments of utility, but most of time it felt like it was getting in the way. The kinds of things it completed well were not the kinds of things that cost a lot of time or mental energy. I found wasting more time trying to fix things that when it got close but not quite there, than it saved in spitting out boilerplate.
A coworker of mine uses it, it almost seems like an unfair advantage but I can also use it too if I want to. Specifically writing hard to decipher code eg. functional composition but Copilot will spit it out.
I haven't wanted to use it personally. I'm also not a senior dev or anything so my opinion is not worth much.
Please remember that Guido van Rossum is now employed by Microsoft. Yes, I am a fan of Guido van Rossum and his work, but I am always suspicious when famous people recommend their own company's products. It feels like excellent marketing/PR that is hard to resist.
That's why I tried it twice. I've been hearing people say they liked it. But I haven't found it very helpful, and often distracting, so I ended up turning it off. I'll probably try again next year when the models are improved to see if I feel any differently then.
Yeah, I understand. I just see a lot of people on here who seem to be deliberately looking for reasons not to like Copilot.
You don't fit that stereotype, of course. So feel free to ignore the following.
Developer tools have learning curves. One doesn't simply open vim/emacs for the first time with a full understanding of how to use it (or why it's a good tool to use, even). Historically, we have had _no_ problem with the steepness of this curve. But, when it comes to Copilot, there's a lot of "tried it and it output an obvious bug! how did this make it past quality assurance?? such a liability!" Just very reactionary and all-that.
I don’t see how that’s relevant to anything in my comment, but since you brought it up I assume you take both of those notions as a given - but they aren’t necessarily concerns of all programmers.
The difference between other Dev tools and GitHub is the probability of getting things wrong. Like when intellij types out boilerplate or a compiler generates code it's 100 percent correct and if it fails, it fails predictably. Copilot is impressive sometimes but there's no guarantee of correctness or failure mode. I cannot trust such tools for anything serious. If you're in the habit of copy / pasting code from the internet then I can see why it might help speed that up. But imo that too is dangerous and I avoid it unless absolutely necessary
Yeah, where it seems very good for me is just writing flask or fast api boilerplate, esp if I have a good function sig + doc string.
I kind of hoped that it would propose good tsx for my components (react+mui) but instead it’s basically useless. Maybe problem is that I chose “filter out exact code reproductions.”
Paying for copilot :) at least in go it’s great to write tests and sometimes some smaller functions :) totally worth paying for it, even from your own pocket if the company wouldn’t allow expensing it
If the company wouldn't pay for it then better think twice because you could get in hot water with legal. That's not a tool one's job or even company's business is worth risking over.
Copilot has a ton of still unresolved legal and compliance issues (copyright violation problems, sending proprietary code to Microsoft as you are writing it, etc.) and most larger businesses won't touch it with a 10 foot pole for that reason. There is even a class action lawsuit against Microsoft over Copilot already.
I pay for copilot. It saves me a modest number of minutes of time per week. That's worth a small fee.
And before someone jumps in: I and my other co-founder who also uses copilot (We are the only two in the company who do, I think, without checking) are the compliance team. We're both very senior and use copilot basically a line or three at a time as a smart autocomplete. It's still worth it.
It's really just perfect for remembering obscure things and can easily be prompted to generate the boilerplate. If you surround it with your style you will see it try to use the same techniques, however if you work on large code bases it gets annoying when it starts copying the bad habits you are trying to get rid of. In those cases it's actually kind of good for bringing to your attention that the building next door is still on fire.
> [we] use copilot basically a line or three at a time as a smart autocomplete.
I think this is the best way to think of CoPilot. GitHub is selling it like its going to write all your code for you, but in reality it is just next-generation auto-complete.
That's not a bad thing. In some ways I'd argue its actually better. GitHub needs to change its marketing because even most developers seem to think that its out there to take away our jobs. Its not and can not. But it provides the smartest auto-complete you've ever seen and that can be useful, especially when wading through mundane parts of your codebase.
I found copilot to be less useful than autocomplete. Typical autocomplete suggests things that actually exist and work in the codebase. While Copilot would suggest things that look superficially like names that exist or ways I might have named it, but very often just wrong.
I find typed languages like Rust or Typescript make VS Code super powered and provides much more value than copilot.
Not surprisingly (Google's internal code analysis tooling is quite sophisticated), Google has added a post-processing filter to remove results that don't compile, but that's not publicly available: https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...
My experience with copilot has been very different. It easily pays for itself, and getting my employer (seed stage startup) to spring for it for the entire team was an easy sell.
Yeah it's pretty dumb most of the time. But I know that, and I don't use code from it without carefully checking it out and modifying it. But it's still a huge help. Just the time saved writing tests alone pays for it. And I've had a few spooky experiences where it feels like it knows the bug fix before I do. Think of it as a smarter auto-complete.
The technology has a long way to go, but I completely disagree with Kite here. It's already good enough to pay for. If my company didn't pay for it, I would. I already pay for JetBrains, and it costs more than Copilot. I would give up JetBrains before I give up Copilot.
My guess here is Kite positioned themselves as a free alternative to Copilot and then couldn't monetize. There very likely is more to it though.
Test are always mostly boilerplate and rarely include anything crafty.
95% of tests are: instantiating a class, running a method, and then asserting that the result. Tests do not or should not be crafty creative code snippets. They are boring functional code blocks by design and most are very similar, only changing out inputs and assertions between tests.
I'd go so far as to say if your test is doing something crafty, you're doing tests wrong. Maybe in a mock or fixture, but that's a write-once sort of affair.
I also don't apply DRY (don't repeat yourself) to tests. Tests should be independently readable beginning to end, no context needed. After all, the true value of a unit test is to take a block of code too complicated to easily fit in your mind, and break it down into a series of examples simple enough to fit.
The best part about it for me is just the Intellisense (in Typescript). I'm using it on probably 3/5 lines that I write as a smarter version, but I rarely use it to do more than finish the current line I am writing.
It is amazing for typing out mock data. Say you're testing parsing of XML - it can easily suggest the the assertions over the data parsed from the XML.
Example test that was 95% coming out of Copilot:
https://github.com/dotnet/arcade-services/blob/61babf31dc63c...
It also predicts comments and logging messages amazingly well (you type "logger." add 7/10 times get what you want, sometimes even better), incorporating variables from the context around. This speeds up the tedious parts of programming when you are finalizing the code (adding docs + tracing).
Honestly, Copilot saves me so much time every week while turning chores into a really fun time.
I don't think it's reasonable to say either was copying. AI assisted tooling is obvious and people have been waiting decades for the tech to reach a point where they can build these tools. Kite tried to get in early - too early probably - but even if they were the very first they didn't invent the idea.
> Just the time saved writing tests alone pays for it.
This, so much. My code since using Copilot is easily ten times better tested than it was before, and I wasn't especially lazy when it comes to testing.
Given 1-2 hand-written unit tests, Copilot can start filling in test bodies that correctly test what's described in the function name. When I can't think of any more edge cases, I'll go prompt it with one more @Test annotation (or equivalent in another language) and it will frequently come up with edge cases that I didn't even think of and write a test that tests that edge case.
(One great part about this use case for those who are a little antsy about the copyright question is that you can be pretty darn confident that you're not running a risk of accidental copyright violation. I write the actual business logic by hand, which means copilot is generating tests that only interact with an API that I wrote.)
APIs are not copyrightable (Oracle vs Google). However, the code that interacts with an API might be.
Regardless, it is interesting to think about what domains are easier to generate effective models for. I would expect it to be easier to generate a supervised model <test description> => <test code>. My intuition is also, that it is easier to generate React component code, and harder to generate feature code.
Incorrect, the case was appealed to the supreme court and the appeal was denied, so the lower court ruling held.
What was ruled by the supreme court was that Google's usage of the API (which had already determined to be copyrighted) fell under fair use in copyright law.
> Incorrect, the case was appealed to the supreme court and the appeal was denied, so the lower court ruling held.
Kind of; the appeal denied was an interlocutory appeal (an appeal before final judgement), so the lower court ruling was left in place until final resolution of the case potentially to be settled in any final appeal.
However, while copyrightability of APIs was raised on the final appeal, the Supreme Court sidestepped it, ruling that because Google’s use was fair use even if the API was copyrightable, it was unnecessary to decide the copyrightability question at all. So the Federal Circuit decision remains in place on copyrightability.
On the gripping hand, though, that decision really doesn't matter much because Federal Circuit decisions on issues outside of those it has unique appellate responsibility aren’t binding precedent on trial courts (it is supposed to apply the case law of the geographic circuit that would otherwise apply, but its rulings don’t have the precedential effect that rulings of that circuit would have.)
So, basically, as far as appellate case law, Oracle v. Google provides no binding precedent on copyrightability of APIs, but precedent that at least one pattern of API copying is fair use if APIs are copyrightable.
Which isn’t encouraging for anyone looking to protect an API with copyrights.
Testimonials of this form are near worthless to a company. Maybe it's true for you. Statistically, it's highly likely to be misleading.
People overestimate their willingness to pay for something for a number of reasons, but one of the biggest is that they incorrectly visualise what the choice to pay or not looks like. They often imagine a moment of abstract choice after which everything remains exactly the same but some small amount of money magically vanishes from their bank account. In reality, paying for something is a tedious inconvenience, and not paying for it more often takes the form of never getting round to putting your card details in than consciously deciding "this isn't worth it".
It can be taken to questionable extremes, but there's truth in the idea that the only real evidence as to what customers will do is what they actually do, not what they say they will do. I don't know if their interpretation is correct, but it sounds like Kite at least has evidence of the former sort.
It's my personal anecdote, I make no claims that other people feel the same way. If they were logical, they should though. Paying $10 to get several hundred dollars in productivity gains is a no-brainer. It's the same reason I pay for a second monitor, a properly comfortable chair and desk, a top end computer, and JetBrains. At software engineer salaries, even small productivity improvements pay off handsomely.
I don't know about software engineering more generally, but I found it worse than useless for my work in data science (machine learning and ETL pipelines), spitting out code that was so wrong that it couldn't be salvaged. I suspect there's a wide variance in the degree to which Copilot will indeed pay for itself.
^ This, I may not use copilot as much on production code, but the testing code it produces makes it easily worth it from a time saved and coverage perspective.
> Developers are somewhat reluctant to pay for tools but I think you can get them to pay for things that are worth it. I've been paying for code editors for years.
Especially when you don't market to developers in general, but freelancers/contractors specifically. It might be hard to sell to salaried developers (they'll buy because it's nicer to work with good tools), but it's easy to sell tooling to anyone who makes more money when they get more done.
I’d recommend anybody thinking about building a devtool to read Neil Davidson’s “Don’t roll the dice”. It’s a pretty old book, but Neil has also made it available for free now and the general lessons still hold true today.
Some IC developers will pay for tools, it’s very hard to have that happen at a price point that supports the scale required. So feature discriminate on the things their boss needs, and charge for that. And then the next set of features for their bosses’ boss, and so on until you’re selling into the C-suite.
"Our diagnosis is that individual developers do not pay for tools."
I think this is probably true. If you need a tool for your day job, your company ought to be paying for it. Some companies have slush funds for small purchases like books, but subscription costs for services would normally need to be approved. If you're a solo consultant then perhaps you'd pay for tools that make you more productive. But for personal projects the value-add would have to be pretty high to be paying another O($10-20) a month on top of other subscriptions.
The big group of "hobbyist" coders are students, and they get copilot for free via Github's very generous edu package (and so does anyone with an edu email address I think). The bigger problem is that this is a very expensive project. It's better suited to a big company with money to burn and deep pockets to give it away to junior devs who will evanglise for it at their new companies (e.g. students) for nothing. See Matlab.
The sheer volume of subscription services I've signed up for as the CTO for a startup is mind-boggling. $8 here, $19 there, $49 for something important, $99 for something essential.
Some tools are easily worth it, especially when you see what is charged for other (less valuable) tools.
1) I need to justify these expenses each for what value they bring, some things are nice to have but bring so little value on paper.
2) You can't just enable tools for some people, there's huge overlap and that overlap gets greater
I get that people need to be paid, but these things very quickly add up. I'm paying about 7-13% of peoples salaries already in these subscriptions, and I feel like a total dick for saying no to people or trying to consolidate these.
Ones I excluded for bring "not worth it" are Lens ($20/user/mo), Snyk ($139/user/month) and Postman ($36/user/month) - contrast those with Gitlabs pricing to understand the value trade-off.
The weird thing is that 13% seems high. It is hard to imagine they are less than 13% more efficient with those tools.
It is weird that software engineers are the only engineer-types that are supposed to be able to do their job with just a computer and a built-in editor.
> It is weird that software engineers are the only engineer-types that are supposed to be able to do their job with just a computer and a built-in editor.
Not really. We aren't actually engineers. Someone appropriated the title and misused it, and now there's no putting that genie back in the bottle.
Engineering is expensive, and most software developers rightly default to a pragmatic craftsman's attitude. Build something people can use and iterate on it based on feedback.
When there aren't lives at stake, it's usually the right call.
Software development is the only field I can think of where the making of the tools of the trade is wholly encapsulated by the field itself. Maybe blacksmithing. In such an environment, it makes sense. We make our own tools, and for the most part the most powerful of those tools end up being given away for free. To other people who both use the tools, and have the capacity to improve them. Who then give them away for free.
It is not a coincidence that software development tooling is one of the only fields that has has been thoroughly eaten by open source. It's because the user-developer circle is complete. A mechanical engineer probably doesn't have the skillset to improve the CAD program they're running. A software developer probably has the skillset to improve VIM.
On the contrary, I find Github Copilot extremely helpful and saving me heaps of time. Yes, it's not writing the logic instead of me, but it acts like the best companion I could have in most of the cases.
The tool just has to be very well integrated and easy to use. That's why copilot is seeing adoption, because Microsoft owns VSCode and has built a very simple integration of Copilot into VSCode.
That said, I'm not even sure VSCode or Copilot is lucrative, if it wasn't owned by Microsoft, could they both be sustainable businesses?
When I tried kite over a year ago I was relatively unimpressed with it. Even though it ran as a plug-in to jet brains IDE it required a windows installed service and two separate executables running in the background (kite.exe, kited.exe), and that stuff continue to run after exiting my IDE which was unacceptable for me.
Kite may have been the first to market but copilot blew them out of the water in terms of overall functionality.
> I fell for this. I enabled it because I was curious about trying new development tools, only to find out later it uploaded all of the source code on my computer to their service. What the hell.
> Since we literally build all of this our B.S. detection meter is really high.
Oh no. The only thing that's high is our conviction that our BS meter is high. We fall prey to, come up with and promote as much BS as the next person.
I paid for intellij - damned near the entire architecture team where I worked had a copy, and the company sure as shit wasn't the one paying for it.
(I eventually stopped subscribing, in part because they were too slow distancing themselves from Russia, in part because of their movement away from open source with their newer tooling.)
Developers will pay for software, if the value proposition is there.
> in part because they were too slow distancing themselves from Russia
I'd cut them some slack here. They had to get their team out of there first—with the way Putin is running things, they sure as hell couldn't announce they were leaving Russia until everyone who was going to follow them was out of there.
On the day of the invasion they tweeted a statement condemning the attack, and within two weeks announced they were leaving Russia.
I'd be curious to hear about services/tools developers do pay for. The diagnosis that developers do not pay for tools seems off to me.
A few tools that I put on the company card when I worked at a Big Tech Co as an IC:
* DataGrip (Jet brains)
* Colab Pro (Google)
* Postman Pro
These were all small $ enough where I didn't need to justify the expense. It was just assumed that if I thought the tool was worth the $, it was.
For more expensive purchasing decisions, there was a longer purchasing/approval process. But the expense would have to be 5-6 figures per year before hitting this barrier.
I think “company card” is the differentiating point here. I’m not sure how many IC devs have access to that. Which makes me think you don’t fall into the group that they’ve defined as IC.
I started getting purchasing authority around the senior level, when I also had some amount of hiring authority. Even if you don't have a company card, there's typically a process where you can get reimbursed for expenses.
If you are an IC reading this and have never tried to expense a tool you find valuable, give it a shot. You might end up surprised at how much management appreciates the ability to trade money for business value.
Yeah I called BS when I read that line too. I pay for plenty of dev tools. Similar list to you.
* JetBrains (PyCharm professional, DataGrip, and Goland) ~$250/yr
* Lucidchart (Diagramming) ~100 /yr
* Paw (HTTP Client) ~$50 /yr
* Docker Pro ~$60 /yr
I think there's probably more, but I'm not at my work laptop to look, but those are the big ones. Those are only individual subscriptions. There's also huge costs when associated with things like Gitlab Premium ($20 - $100 /user/month), CI/CD, Code coverage tools, security scanners, etc. Companies pay A LOT for development tools.
If Kite thinks that the problem why no one will pay $9/mo for their service is because developers or their company's are cheap, they need to re-assess. The reason they couldn't sell their service is because it wasn't providing enough value to justify it. But companies are paying hundreds of dollars a month per developer in most cases for various tools. The extra $9 for Kite isn't the dealbreaker if there was enough value from it.
Yea.. i happily pay for several JetBrains tools and i'd love to pay for even more. I've got several problems that i don't want to spend time solving myself.
Frankly as a developer i've got more problems than i can count and if it involves a GUI i tend to prefer to pay for it. I love FOSS but UX is just not often a focus. I have better experience with paid products. Assuming the licensing isn't punishing.
Biggest one off the top of my head is in the category of developer productivity (which i pay a lot for already), and specifically it lets me use my editor of choice (Helix/Kakoune currently) while getting powerful new features.
Huge hurdle, obviously, but my thought was an expanded set of LSP features from JetBrains, disconnected from their IDEs. They spend quite a bit of time and money on developing DX but it's all inaccessible to those of us who prefer different editors.
I'd pay a lot for these sorts of features that go above the existing FOSS LSP, while retaining more integration to the FOSS tooling we've come to know and love (like editors or choice).
With Copilot it's important to configure your editor so that it only generates completions when you ask for them. Then it can be very useful at times, especially when you're doing something you know is routine but you don't recall off-hand how to do (e.g. opening and writing to a file in append mode in a language you only use occasionally) Having it suggesting stuff every time you hit enter quickly gets annoying.
Copilot had 400k paying customers within it's first month [1]. I'm not a fan of mass generalization about large cohorts of people. Will everyone use a tool? Of course not. You just need dedicated early adopters that see the value add.
Without having tried it I'm assuming either their product was not good enough or their marketing department isn't strong. Developer really tend to neglect the latter.
I feel the problem with discussions about Copilot is that they consist of roughly two groups of people talking past each other. The first group believes that Copilot should be able to write code for whatever you tell it to write code for. The second group thinks Copilot is a fairly overpowered autocomplete.
The first group gets annoyed all the time because Copilot fails to write most code when prompted with comments, or writes inaccurate code at best. They get upset when they see that Copilot can reproduce GPL code when prompted in a specific way.
The second group most prompt Copilot by allowing it to tab a complete a line or two at a time, and they are actually super happy because Copilot is way better than any other existing autocomplete; it's basically in a class of its own. To them, the GPL issue seems a bit more abstract, because they would never use Copilot to do that anyways.
I fall pretty firmly into the second camp (can you tell?). Allow me to soliloquize for a moment. Copilot is an incredibly powerful tool, probably the most powerful one I have, but, just like any tool, you need to really learn your way around it, and understand what it can and cannot do, before you start making judgments. I'm not surprised that you turned it off after using it twice. Imagine saying you stopped using React after making two components!
Maybe I should write up a bit more about how I use Copilot, but in a nutshell I feel that it falls somewhere between a 2x-better autocomplete and (and this bit is even more interesting) a tool similar to google search, but more tightly integrated with the coding environment. The second bit is why it's so good. Imagine if I were to continuously google search everything I was doing while coding, while I was coding it. Sure, most of the time it'd just confirm you were doing the right thing, but... every now and then, Google might turn up a better strategy than the one I was currently trying. That's how I feel Copilot works all the time; it's continuously "google searching" for alternate approaches, and every now and then it'll be like, "aha, did you think of [this thing]" and really take me aback, because I wouldn't have even thought to Google for that particular bit of code / problem / strategy.
Of course, you could continuously google search everything you did as you did it, but it would be a massive waste of time. Just imagine Copilot is doing it for you, and returning what it found. Most of the time I know what I'm doing, but every now and then, the result is remarkable.
I'm not a lawyer, but my understanding is that even small snippets of licensed code can be problematic. I don't know exactly what the cutoff is, but when I tried Copilot it often suggested to auto complete snippets of code that were long enough that if I was intentionally coping them from a licensed codebase, I would handle the license. I'm not talking about whole giant functions, but small functions or large chunks of a function.
It is true that more commonly it suggested at most a few lines of obvious code which could really only be written the way it suggested, but a number of people in the comments on this article mentioned using Copilot to come up with test cases, so I think people are actually using it to suggest larger snippets of code.
I pay out of pocket for a JetBrains license because it makes me a LOT more productive. I don't spend money on a lot of dev tools, but if it saves me non-trivial time, it's a no-brainer.
I tried Kite and didn't even keep using the free trial or whatever the free tier was at the time. I don't think the issue is that developers won't pay for tools. Companies like Jetbrains are incredibly successful selling tools to developers. The issue for Kite is that Kite wasn't worth paying for.
Tabnine, in emacs, with lsp mode gives you single line completions and predicts and fills out word by word in a very effective manner. It's an actual timesaver and worth the money (in emacs). The vscode experience is more problematic, but that's vscode's completions guis fault, not the tabnine server's.
I disagree, Github copilot makes me happy, helps a lot with guessing patterns in my own project base code, I just write the good function name and he guess what I want to do.
Yeah having tried kite years ago, copilot absolutely destroys them in terms of helpful suggestions both at a line level and at a code block level. Its contextual awareness of surrounding code is also fantastic.
Now whether or not that's due to the fact that copilot had the financial resources to train a significantly superior ML model is another question, but throwing shade at copilot is a fairly transparent move.
It failed because they did it exactly in reverse of how it should have been done. First they assembled the team, then they outlined the product then marketing and then only then they realized nobody would pay for that. You're supposed to first sell your product and then build it! I wonder whether anyone raised this issue in the early stage...
Indeed. I find it very exciting when a product category dominated by proprietary products gets a FOSS alternative. I wasn't aware of the others, thanks!
> Our diagnosis is that individual developers do not pay for tools.
Throwing salt on the wound here but that’s just false. I mean, there’s copilot and it’s alternative that I can’t think of the name right now. more broadly there’s Jet brains ides, visual studio, Productivity apps, etc. look at product hunt or appsumo or popular show hns. Devs pay for tools, just not Kite.
Edit: I should clarify, enough devs pay for tools to make the market sustainable. Not all devs pay for tools.
If you could actually prove it provided real benefits, it would still be worth it since you could know for sure that spending $5 on a tool will result in $10 extra earned. Sounds like people just didn't believe that there was that value being generated.
Most devs don't pay for their tools. Because they are employees and they need their bosses to sign off on expenses. I know some free lancers that pay for some tools but way more that don't. So, it's a small market that sustains a handful of really nice tools. Jetbrains is one of the more successful tool vendors in this space. But their tools are really essential to many developers.
As a CTO, most of my budget goes to paid services that add clear value with a clear value proposition. The value proposition with developer tools is usually quite murky. It's all very subjective and preference based. So, something like kite is a hard sell.
It's remarkable that they attracted so much investment. But of course that put them under enormous pressure to meet what were probably highly unrealistic revenue goals as well. That team might have made them a nice acquihire target at best.
Kite made me a very good (for a startup) job offer a few years back. They had a very friendly and welcoming bunch of people, and even Adam, the founder, came off as a typical human being in conversation. Easily the best job I've ever turned down, even knowing that Copilot would eat their lunch a year or two later.
Value generation in software doesn't equal profit generation. Is it a flawed business model to pursue growth first and profit later? No, as long as there is a good plan to get that future profit. If the 500k developers weren't driving business spending decisions enough to pay for Kite, either it isn't particularly useful or it's a sign of the times. I'm guessing from the rest of the context it's the former, no one seems to be crying out that this is a great product that will be widely missed. This sort of failure is good and a good decision by the business leaders. It keeps our economy healthy, you want the real winners to win, and not every bet works out.
What that says to me is: we were absolutely not capable of developing the required technology, went to market anyway, and feel spite because someone else developed tech that could have made us successful 10 years ago.
Because the incident should be fresh and impactful enough to remember, from my point of view it almost reach levels as bad as Sourceforge adware injection in open source software installers.
For a recap for people who weren't in the industry or have short memory.
Kite took ownership of some popular code editor plugins and injected some adware/tracking code.
The VCs are willingly fleeced - they aren't about companies generating revenue either. Hard to feel sorry for VCs when they're playing the same game Kite is - just hyping the market and flipping an asset, hoping the music doesn't stop before they get out.
545 comments
[ 2.1 ms ] story [ 280 ms ] threadFirst, we failed to deliver our vision of AI-assisted programming because we were 10+ years too early to market, i.e. the tech is not ready yet.
We built the most-advanced AI for helping developers, but it fell short of the 10× improvement required to break through because today’s state of the art for ML on code is not good enough. You can see this in Github Copilot, which is built by Github in collaboration with Open AI. As of late 2022, Copilot has a number of issues preventing it from being widely adopted.
The largest issue is that state-of-the-art models don’t understand the structure of code, such as non-local context. We made some progress towards better models for code, but the problem is very engineering intensive. It may cost over $100 million to build a production-quality tool capable of synthesizing code reliably, and nobody has tried that quite yet.
Nonetheless, we could have built a successful business without 10×’ing developer productivity using AI, and we did not do that.
We failed to build a business because our product did not monetize, and it took too long to figure that out."
That's not the same thing as being too early to the market. That simply means you didn't have a solution capable of solving a problem.
https://github.com/orgs/kiteco/repositories
I wish more projects would do this
$100 million is nothing tbh.
Tbh even at tech giants that make tens of billions of dollars, a $100 million investment is likely a lot, I'm guessing this sort of investment will require sign-off by CEO or at least VP level along with a solid business plan.
How many working hours, tons of materials will it take?
Could they be used on more relevant needs to humanity?
If so, why use them on a moonshot we will die before humanity achieves? Essentially approving burning up resources on unverifiable, perhaps unachievable outcomes?
We still fund a lot of agency to iterate on high minded potential as if future humans have an immutable obligation to carry on the work.
Secular norms and justifications are not sacrosanct. The dollar is a made up token of power.
They seemed like a really cool team, I wish them the best.
True, AI assisted coding does not deliver 10x. But as a user of another AI assistant, I feel that it gives me ~1.25x to ~2x improvement for the keyboard typing when I code. And that is respectable too :) AI for me currently allows me to tab complete some things that previously an IDE on its own was not able to.
Tabnine for JetBrains CLion. It works with Rust and several other languages.
Depending on how local he's talking, this isn't really true of Copilot. In my experience it will use context all the way up to the top of the file, even in very long files. And at least the Rust version even seems to look at the imports—if you have a use declaration it will actually correctly build and use structs in other files regardless of whether you've yet used them in the current file.
My understanding is that copilot is largely a self-supervised approach. They feed a massive body of (somewhat noisy) code into the model. The model really does learn a lot of structure on its own and this is a testament to deep learning on noisy datasets.
I'm guessing the "hooks" that they have already from IDE's, language-servers, etc. _are_ quite "structure-aware" - so they want the predicted structure as well as the code, so they can improve the typing experience beyond line-completion.
I think the estimation of 100 million for such a task is maybe too high? I don't know - it feels like you could actually get quite close to such a system by simply using thousands of custom prompt engineering tricks that prepend structure examples to the prompt?
I can imagine this is easier in a language like Rust that has a really strict module system, and to be fair the project that I've been using it on is a side project that isn't over 10,000 lines of code yet. If I were up to 30 imports per file I can imagine concatenating would become much less effective.
There are definitely times where it produces a close approximation that's obviously just statistical, but there are other times where there's no question that it picked up something from a different source file that couldn't have possibly been in its training set.
I haven't yet decided if it's using imports or opened files in the editor, but it's definitely not just using the single file I have active.
For example, if an object defined in another file has a function called `rename` that takes zero arguments, when calling it from another file Copilot will likely suggest arguments if there are variables like `old` and `new` near the cursor, even though `rename` actually doesn't take any, just because functions called `rename` typically take arguments. This behavior is in contrast to a tool like an IDE that can trace through the way non-local code references work.
Mulling over business models, and noticing the 'devs won't pay' narrative in the blog post, it's interesting to see the existing business models in AI; basically they seem to be:
* API-driven cloud calls (this is a way to get high value out of your existing cluster if you're AWS, MS, etc.)
* Platform play + possible eventual lock-in: OpenAI/Microsoft
* Subscription service for very specific needs (Grammarly, writing support)
I wonder if engineers would pay $9.99/month (or even $49.99/month) for a 'grammar checker for PRs' - essentially: "Avoid embarrassing bugs before you commit". That is, I wonder if Kite could have been successfully sold as the third tier - sub service for something very specific.
I guess if it's a good idea, someone could pull the Kite repos and launch it -- but my guess is there may be a market in there.
Payscale estimates average engineering salary as having between $3,000 and $7,000 a month more in disposable income over writers -- and I would guess almost every professional writer pays for grammarly.
But, I agree that this is a new concept, and just spitballing -- right now, these sorts of linters and code formatting tools are mostly open source, so it would be some product marketing work to see if the market would actually pay.
Perhaps an under-appreciated perspective – most of us don't come from families that were paid the sort of salaries we get; and consequently we are more likely to be irrationally stingy about money as a result of having grown up (relatively) poor. I know that $200/year is not a significant cost at my salary; but I do remember a time when that would have been totally out of reach for me.
And also there's the fairness aspect – do I want to be participating in a system where only rich devs can get access to good tools? Yes, I understand that the world is not fair and already works that way, but I should probably not perpetuate that system; and instead, make work pay for it.
I agree that Kite didn’t deliver the 10x. I was an early user and tried hard to use it but didn’t find the benefit compelling enough to drop into my workflows, but it was very exciting.
I’m sure I speak for all of HackerNews when I wish you the best for whatever is next for the team.
Also, what are you good folks doing next?
This is the one-sentence summary about why the business failed, but it's kind of a strange way of putting it.
I am dead sure that there were plenty of advisers along the way who told the company's executives that its monetization plan was weak and unlikely to succeed. But everyone assumes that they'll be the exceptional case.
"It took too long to figure that out" makes it seem like the most likely scenario wasn't staring them in the face the whole time.
https://news.ycombinator.com/item?id=14836653
The common definitions have to do with stealing, but an equally valid definition of the word hijack is to:
> take over (something) and use it for a different purpose.
Taking over a project so you can have it to advertise your service is exactly that.
You are misleading readers in order to promote your agenda. You clearly speak perfect English, so you know what hijack means. "take over (something) and use it for a different purpose." is not found as a definition of "hijack" in any dictionary. "Hijack" implies "unlawfully" or "without having a right to do so".
Of course, every word can be used in a slightly different meaning; for example, in software can (harmlessly) hijack an entity (circumventing the usual API for expediency or performance). Such broadened semantics is perfectly fine when there's no confusion about the meaning. Very clearly in the case of OP, there was a clear intention to imply "unlawful" or "without having a right", so this exception doesn't apply.
The sad thing is that I actually support your agenda. I just don't support promoting it through misleading statements.
> to take control of or use something that does not belong to you for your own advantage:
And Encyclopædia Britannica: https://www.britannica.com/dictionary/hijack
>: to take or take control of (something) for your own purposes
And Merriam Webster: https://www.merriam-webster.com/dictionary/hijack
> : to take or take control of (something) as if by hijacking > often, specifically : to change the topic or focus of (something, such as a conversation) : REDIRECT
It wasn't my statement by the way, I just figure if you're going to nitpick you should at least be correct about the nit.
> many programmers would consider [this] a violation of the open-source spirit.
Instead of making the untrue statement above, just say
"They used, in my opinion, an unethical way to advertise their product; specifically, they bought OSS products and put their ads in there."
Hopefully the next project goes well!
> We failed to build a business because our product did not monetize, and it took too long to figure that out.
Yet people always defend telemetry in software, saying it's how they improve their product. 7 years of telemetry, and they couldn't figure it out?!
I never used Kite, but I've tried Github Copilot twice, and found it marginal at best (and distracting at worst - which is why I turned it off both times). If Kite was similar, the reason I'm not paying is that coder AIs are not providing any value.
Developers are somewhat reluctant to pay for tools but I think you can get them to pay for things that are worth it. I've been paying for code editors for years.
In my experience, it's a quality of life improvement, but the things that dictate actual time to market is bottlenecked by things that aren't solved by copilot, such as overall design, decision making, requirements gathering, code structure/architecture, solution ideation, user acceptance, infrastructure setup, etc.
I think if it eventually could help with those other tasks, you'd see time to market gains, and that would start to make it really valuable.
Never ever I'm risking breaking copyright, and I also don't like Microsoft not including their own code in the model.
From allegations I have read across the Internet, Microsoft might be doing those who use Copilot a favor.
[0]: https://www.theverge.com/2022/11/8/23446821/microsoft-openai...
1) It's the code license that matters, and the depends on what license the code the developer is building is using
2) License compatibility is undecided (see eg the different views of Apache Foundation and FSF on the compatibility of ASF 2.0 and GPL)
3) It's easy for someone to deliberately produce code that violates a license. MS is a big target and you can bet license trolls would chase it on that.
Probably more reasons, but there's a good start.
This is a serious question, I'm apparently just unaware of the horror-stories that can come out of breaking copyright. (Not from the US so..)
For me its most useful for helping with bash scripts and small simple stuff, just saves a huge amount of time I would spend googling and checking small things. Not sure if copyright is relevant there or not, it certainly isn't something I am worried about. Interested to hear what your fears are based on.
If you use code or art generated by an AI that was regurgitating training data, you can be sued for copyright infringement.
The way that AI gets training data these days is... questionably ethical. It's all scraped off the web, because the people who make these AIs saw court precedent for things like Google Books being fair use and assumed it would apply to data mining[1]. Problem is, that does nothing for the people actually using the AI to generate what they thought were novel code sequences or images, because fair use is not transitive.
[0] This won't work in current Copilot because a) I'm misremembering the comment phrasing and b) they explicitly banned that input from generating that output.
[1] In the EU, this practice is explicitly legal
But I find I get most pleasure out of using CoPilot for personal tasks, everyday stuff that isn't for work. Surely there is no issue there?
And to counter the note about the scraping sources-- I have recently (out of work) been experimenting with using CoPilot to translate open-source MATLAB code into python and zig. If I make something useful in this way, it will be open source and I will credit those who wrote the original code.
Surely a use-case like that last would have to tick all the moral and legal boxes?
"GitHub Copilot blocks your ability to learn." Is a common refrain.
I don't see ANY industry-wide consensus on whether GitHub Copilot truly helps developers right now.
The only scenario I can get anyone to agree on is generating templates. Aka, JSON or CSS files that you then edit.
Every time I don't have to context switch to look up some technical errata in my browser is a complete win for me.
I've never heard anyone claim it blocks the ability to learn - if anything it's the opposite. Many people like how it shows you APIs you weren't aware of.
But I did find that I need to turn it off at certain stages when learning (or re-learning) a programming language. It's seemed counterproductive until you have a good grasp on the basic syntax of the language. But the "showing you new APIs" does seem to be a thing that actually helps.
In general, you should not be accepting code completions that you don't understand. I'm usually stricter, in that I only typically accept completions that line up with what I was planning to type anyway.
It's good for learning new packages in Python.
It's quite poor for learning a totally new set of programming language grammar.
If you're a Javascript developer of 5+ years of experience, it's probably absolutely fantastic. Not so great if you're learning a new programming language.
- learning a new code base
- learning a new (popular) library
- learning a new language
You could compare it to, say, eslint but for other languages .
What is an idiomatic way of X with Y?
Well, copilot will give you the answer in 10-30 seconds less time than opening a browser and searching.
Im not going to argue it’s legal merits, but as a learning tool, it’s very much like having a smart linter.
The larger scale code generation is less obviously useful and usually wrong, I agree.
On the contrary, it frequently suggests code that adhere to better practices.
I haven't wanted to use it personally. I'm also not a senior dev or anything so my opinion is not worth much.
[1] https://youtu.be/cdiD-9MMpb0?t=8723
Others have already pointed out the case as a reply.
You don't fit that stereotype, of course. So feel free to ignore the following.
Developer tools have learning curves. One doesn't simply open vim/emacs for the first time with a full understanding of how to use it (or why it's a good tool to use, even). Historically, we have had _no_ problem with the steepness of this curve. But, when it comes to Copilot, there's a lot of "tried it and it output an obvious bug! how did this make it past quality assurance?? such a liability!" Just very reactionary and all-that.
Anyway, sorry for the toxic response.
Thanks for the rigorous examination of my comment though. /s
I kind of hoped that it would propose good tsx for my components (react+mui) but instead it’s basically useless. Maybe problem is that I chose “filter out exact code reproductions.”
Copilot has a ton of still unresolved legal and compliance issues (copyright violation problems, sending proprietary code to Microsoft as you are writing it, etc.) and most larger businesses won't touch it with a 10 foot pole for that reason. There is even a class action lawsuit against Microsoft over Copilot already.
And before someone jumps in: I and my other co-founder who also uses copilot (We are the only two in the company who do, I think, without checking) are the compliance team. We're both very senior and use copilot basically a line or three at a time as a smart autocomplete. It's still worth it.
I think this is the best way to think of CoPilot. GitHub is selling it like its going to write all your code for you, but in reality it is just next-generation auto-complete.
That's not a bad thing. In some ways I'd argue its actually better. GitHub needs to change its marketing because even most developers seem to think that its out there to take away our jobs. Its not and can not. But it provides the smartest auto-complete you've ever seen and that can be useful, especially when wading through mundane parts of your codebase.
I find typed languages like Rust or Typescript make VS Code super powered and provides much more value than copilot.
Indeed, I payed for a debugger because MSVC is pretty terrible
Yeah it's pretty dumb most of the time. But I know that, and I don't use code from it without carefully checking it out and modifying it. But it's still a huge help. Just the time saved writing tests alone pays for it. And I've had a few spooky experiences where it feels like it knows the bug fix before I do. Think of it as a smarter auto-complete.
The technology has a long way to go, but I completely disagree with Kite here. It's already good enough to pay for. If my company didn't pay for it, I would. I already pay for JetBrains, and it costs more than Copilot. I would give up JetBrains before I give up Copilot.
My guess here is Kite positioned themselves as a free alternative to Copilot and then couldn't monetize. There very likely is more to it though.
(Don't have much experience with it)
95% of tests are: instantiating a class, running a method, and then asserting that the result. Tests do not or should not be crafty creative code snippets. They are boring functional code blocks by design and most are very similar, only changing out inputs and assertions between tests.
I also don't apply DRY (don't repeat yourself) to tests. Tests should be independently readable beginning to end, no context needed. After all, the true value of a unit test is to take a block of code too complicated to easily fit in your mind, and break it down into a series of examples simple enough to fit.
It also predicts comments and logging messages amazingly well (you type "logger." add 7/10 times get what you want, sometimes even better), incorporating variables from the context around. This speeds up the tedious parts of programming when you are finalizing the code (adding docs + tracing).
Honestly, Copilot saves me so much time every week while turning chores into a really fun time.
Once you get the hang of how to prompt it (mainly through clever use of comments) it can be a HUGE time saver.
This, so much. My code since using Copilot is easily ten times better tested than it was before, and I wasn't especially lazy when it comes to testing.
Given 1-2 hand-written unit tests, Copilot can start filling in test bodies that correctly test what's described in the function name. When I can't think of any more edge cases, I'll go prompt it with one more @Test annotation (or equivalent in another language) and it will frequently come up with edge cases that I didn't even think of and write a test that tests that edge case.
(One great part about this use case for those who are a little antsy about the copyright question is that you can be pretty darn confident that you're not running a risk of accidental copyright violation. I write the actual business logic by hand, which means copilot is generating tests that only interact with an API that I wrote.)
Regardless, it is interesting to think about what domains are easier to generate effective models for. I would expect it to be easier to generate a supervised model <test description> => <test code>. My intuition is also, that it is easier to generate React component code, and harder to generate feature code.
And was all over the tech news at the time.
I kind of suspect you are intentionally trolling…
What was ruled by the supreme court was that Google's usage of the API (which had already determined to be copyrighted) fell under fair use in copyright law.
Kind of; the appeal denied was an interlocutory appeal (an appeal before final judgement), so the lower court ruling was left in place until final resolution of the case potentially to be settled in any final appeal.
However, while copyrightability of APIs was raised on the final appeal, the Supreme Court sidestepped it, ruling that because Google’s use was fair use even if the API was copyrightable, it was unnecessary to decide the copyrightability question at all. So the Federal Circuit decision remains in place on copyrightability.
On the gripping hand, though, that decision really doesn't matter much because Federal Circuit decisions on issues outside of those it has unique appellate responsibility aren’t binding precedent on trial courts (it is supposed to apply the case law of the geographic circuit that would otherwise apply, but its rulings don’t have the precedential effect that rulings of that circuit would have.)
So, basically, as far as appellate case law, Oracle v. Google provides no binding precedent on copyrightability of APIs, but precedent that at least one pattern of API copying is fair use if APIs are copyrightable.
Which isn’t encouraging for anyone looking to protect an API with copyrights.
It bases this generated test cases on other similar test cases in other software, including GPL licensed
Testimonials of this form are near worthless to a company. Maybe it's true for you. Statistically, it's highly likely to be misleading.
People overestimate their willingness to pay for something for a number of reasons, but one of the biggest is that they incorrectly visualise what the choice to pay or not looks like. They often imagine a moment of abstract choice after which everything remains exactly the same but some small amount of money magically vanishes from their bank account. In reality, paying for something is a tedious inconvenience, and not paying for it more often takes the form of never getting round to putting your card details in than consciously deciding "this isn't worth it".
It can be taken to questionable extremes, but there's truth in the idea that the only real evidence as to what customers will do is what they actually do, not what they say they will do. I don't know if their interpretation is correct, but it sounds like Kite at least has evidence of the former sort.
Especially when you don't market to developers in general, but freelancers/contractors specifically. It might be hard to sell to salaried developers (they'll buy because it's nicer to work with good tools), but it's easy to sell tooling to anyone who makes more money when they get more done.
Some IC developers will pay for tools, it’s very hard to have that happen at a price point that supports the scale required. So feature discriminate on the things their boss needs, and charge for that. And then the next set of features for their bosses’ boss, and so on until you’re selling into the C-suite.
I think this is probably true. If you need a tool for your day job, your company ought to be paying for it. Some companies have slush funds for small purchases like books, but subscription costs for services would normally need to be approved. If you're a solo consultant then perhaps you'd pay for tools that make you more productive. But for personal projects the value-add would have to be pretty high to be paying another O($10-20) a month on top of other subscriptions.
The big group of "hobbyist" coders are students, and they get copilot for free via Github's very generous edu package (and so does anyone with an edu email address I think). The bigger problem is that this is a very expensive project. It's better suited to a big company with money to burn and deep pockets to give it away to junior devs who will evanglise for it at their new companies (e.g. students) for nothing. See Matlab.
The sheer volume of subscription services I've signed up for as the CTO for a startup is mind-boggling. $8 here, $19 there, $49 for something important, $99 for something essential.
Some tools are easily worth it, especially when you see what is charged for other (less valuable) tools.
Gitlab, Confluence, Jira, Asana, 1Password, co-pilot, codepen, sentry, jetbrains, gitlab plugins for jetbrains, Visual Studio, Docker Desktop, Perforce, Slack, etc;etc;etc;etc
Then there's things like Spacelift ($250!)!
The most frustrating thing is that:
1) I need to justify these expenses each for what value they bring, some things are nice to have but bring so little value on paper.
2) You can't just enable tools for some people, there's huge overlap and that overlap gets greater
I get that people need to be paid, but these things very quickly add up. I'm paying about 7-13% of peoples salaries already in these subscriptions, and I feel like a total dick for saying no to people or trying to consolidate these.
Maybe, but most of the tools you listed are not in that category IMNSHO.
Ones I excluded for bring "not worth it" are Lens ($20/user/mo), Snyk ($139/user/month) and Postman ($36/user/month) - contrast those with Gitlabs pricing to understand the value trade-off.
You either care about licenses of dependencies or just decide copyright doesn't apply to you :D Seems that using both is a contradiction.
They have a bunch of rules for IaC that prevent default behaviour from biting you in the butt.
It is weird that software engineers are the only engineer-types that are supposed to be able to do their job with just a computer and a built-in editor.
Not really. We aren't actually engineers. Someone appropriated the title and misused it, and now there's no putting that genie back in the bottle.
I have yet to meet or work with one.
Engineering is expensive, and most software developers rightly default to a pragmatic craftsman's attitude. Build something people can use and iterate on it based on feedback.
When there aren't lives at stake, it's usually the right call.
It is not a coincidence that software development tooling is one of the only fields that has has been thoroughly eaten by open source. It's because the user-developer circle is complete. A mechanical engineer probably doesn't have the skillset to improve the CAD program they're running. A software developer probably has the skillset to improve VIM.
That said, I'm not even sure VSCode or Copilot is lucrative, if it wasn't owned by Microsoft, could they both be sustainable businesses?
Kite may have been the first to market but copilot blew them out of the water in terms of overall functionality.
Sounds like everyone has dodged a bullet!
Since we literally build all of this our B.S. detection meter is really high.
Kite thought it can go after the up and coming new developers by doing slightly shady things.
However, developers also have an incredible allergy to such tactics and it forever taints your brand.
So overall, developers do pay for tools, just not useless ones with shady growth tactics.
I briefly tried Kite a few years ago. I didn't notice anything shady although maybe I just didn't stick around long enough.
What shady tactics are you referring to?
https://news.ycombinator.com/item?id=14857944
https://news.ycombinator.com/item?id=14836653
An old comment from another user. https://news.ycombinator.com/item?id=14837253
Developers easily fool themselves thinking they’ll save $9 p/m by building something from scratch in 3 weeks.
Oh no. The only thing that's high is our conviction that our BS meter is high. We fall prey to, come up with and promote as much BS as the next person.
(I eventually stopped subscribing, in part because they were too slow distancing themselves from Russia, in part because of their movement away from open source with their newer tooling.)
Developers will pay for software, if the value proposition is there.
I'd cut them some slack here. They had to get their team out of there first—with the way Putin is running things, they sure as hell couldn't announce they were leaving Russia until everyone who was going to follow them was out of there.
On the day of the invasion they tweeted a statement condemning the attack, and within two weeks announced they were leaving Russia.
https://twitter.com/jetbrains/status/1496786254494670851?lan...
https://blog.jetbrains.com/blog/2022/03/11/jetbrains-stateme...
Two weeks was impressive, just not impressive enough to get me to renew.
A few tools that I put on the company card when I worked at a Big Tech Co as an IC:
* DataGrip (Jet brains)
* Colab Pro (Google)
* Postman Pro
These were all small $ enough where I didn't need to justify the expense. It was just assumed that if I thought the tool was worth the $, it was.
For more expensive purchasing decisions, there was a longer purchasing/approval process. But the expense would have to be 5-6 figures per year before hitting this barrier.
If you are an IC reading this and have never tried to expense a tool you find valuable, give it a shot. You might end up surprised at how much management appreciates the ability to trade money for business value.
* JetBrains (PyCharm professional, DataGrip, and Goland) ~$250/yr
* Lucidchart (Diagramming) ~100 /yr
* Paw (HTTP Client) ~$50 /yr
* Docker Pro ~$60 /yr
I think there's probably more, but I'm not at my work laptop to look, but those are the big ones. Those are only individual subscriptions. There's also huge costs when associated with things like Gitlab Premium ($20 - $100 /user/month), CI/CD, Code coverage tools, security scanners, etc. Companies pay A LOT for development tools.
If Kite thinks that the problem why no one will pay $9/mo for their service is because developers or their company's are cheap, they need to re-assess. The reason they couldn't sell their service is because it wasn't providing enough value to justify it. But companies are paying hundreds of dollars a month per developer in most cases for various tools. The extra $9 for Kite isn't the dealbreaker if there was enough value from it.
Frankly as a developer i've got more problems than i can count and if it involves a GUI i tend to prefer to pay for it. I love FOSS but UX is just not often a focus. I have better experience with paid products. Assuming the licensing isn't punishing.
Huge hurdle, obviously, but my thought was an expanded set of LSP features from JetBrains, disconnected from their IDEs. They spend quite a bit of time and money on developing DX but it's all inaccessible to those of us who prefer different editors.
I'd pay a lot for these sorts of features that go above the existing FOSS LSP, while retaining more integration to the FOSS tooling we've come to know and love (like editors or choice).
There's more i'm positive, but i just woke up :)
https://github.com/community/community/discussions/7553#disc...
https://www.reddit.com/r/vscode/comments/qromfk/is_there_a_w...
Maybe the product is poor. Maybe people didn’t believe the claim. But nobody said, “nah I don’t want my devs to be 18% faster.”
Without having tried it I'm assuming either their product was not good enough or their marketing department isn't strong. Developer really tend to neglect the latter.
[1] https://www.ciodive.com/news/github-copilot-microsoft-softwa...
The first group gets annoyed all the time because Copilot fails to write most code when prompted with comments, or writes inaccurate code at best. They get upset when they see that Copilot can reproduce GPL code when prompted in a specific way.
The second group most prompt Copilot by allowing it to tab a complete a line or two at a time, and they are actually super happy because Copilot is way better than any other existing autocomplete; it's basically in a class of its own. To them, the GPL issue seems a bit more abstract, because they would never use Copilot to do that anyways.
I fall pretty firmly into the second camp (can you tell?). Allow me to soliloquize for a moment. Copilot is an incredibly powerful tool, probably the most powerful one I have, but, just like any tool, you need to really learn your way around it, and understand what it can and cannot do, before you start making judgments. I'm not surprised that you turned it off after using it twice. Imagine saying you stopped using React after making two components!
Maybe I should write up a bit more about how I use Copilot, but in a nutshell I feel that it falls somewhere between a 2x-better autocomplete and (and this bit is even more interesting) a tool similar to google search, but more tightly integrated with the coding environment. The second bit is why it's so good. Imagine if I were to continuously google search everything I was doing while coding, while I was coding it. Sure, most of the time it'd just confirm you were doing the right thing, but... every now and then, Google might turn up a better strategy than the one I was currently trying. That's how I feel Copilot works all the time; it's continuously "google searching" for alternate approaches, and every now and then it'll be like, "aha, did you think of [this thing]" and really take me aback, because I wouldn't have even thought to Google for that particular bit of code / problem / strategy.
Of course, you could continuously google search everything you did as you did it, but it would be a massive waste of time. Just imagine Copilot is doing it for you, and returning what it found. Most of the time I know what I'm doing, but every now and then, the result is remarkable.
It is true that more commonly it suggested at most a few lines of obvious code which could really only be written the way it suggested, but a number of people in the comments on this article mentioned using Copilot to come up with test cases, so I think people are actually using it to suggest larger snippets of code.
Now whether or not that's due to the fact that copilot had the financial resources to train a significantly superior ML model is another question, but throwing shade at copilot is a fairly transparent move.
lol
Throwing salt on the wound here but that’s just false. I mean, there’s copilot and it’s alternative that I can’t think of the name right now. more broadly there’s Jet brains ides, visual studio, Productivity apps, etc. look at product hunt or appsumo or popular show hns. Devs pay for tools, just not Kite.
Edit: I should clarify, enough devs pay for tools to make the market sustainable. Not all devs pay for tools.
(That manager getting the purchase order approved through corporate took months and months, but that's neither here nor there.)
Yes.
https://www.ciodive.com/news/github-copilot-microsoft-softwa...
As a CTO, most of my budget goes to paid services that add clear value with a clear value proposition. The value proposition with developer tools is usually quite murky. It's all very subjective and preference based. So, something like kite is a hard sell.
It's remarkable that they attracted so much investment. But of course that put them under enormous pressure to meet what were probably highly unrealistic revenue goals as well. That team might have made them a nice acquihire target at best.
They got wiped out by microsoft, github copilot, and litigation issues around AI provided code.
Good riddance to bad rubbish
https://news.ycombinator.com/item?id=14857944
https://news.ycombinator.com/item?id=14902630
https://news.ycombinator.com/item?id=19018037
For a recap for people who weren't in the industry or have short memory.
Kite took ownership of some popular code editor plugins and injected some adware/tracking code.