208 comments

[ 2.7 ms ] story [ 228 ms ] thread
Poor naming choice, considering CodeGuru.com has been around for what, decades?
I worked on the relaunch of Cloud9 --> AWS Cloud9 and there was an extensive naming process shortly before the launch of the service, both for AWS Cloud9 and for the term "environments" (which was previously workspaces).

I can tell you that there were lots of managers, PMs, directors, etc involved and they considered tons of naming options. They took into account third party services/products, first party services/products, and other things that might have overlap. This was likely the situation here and they accepted this as a drawback.

That said, you're free to disagree and that doesn't mean it was the right choice, just wanted to point out that this was not an oversight.

> just wanted to point out that this was not an oversight

With codeguru.com being a long established site, doesn't that make it worse?

I'm thinking Amazon doesn't care a whole lot about the IP of others. I say that based on reports of rampant counterfeit goods and complaints about Amazon copying product design for their in-house brands.
The code review feature seems too expensive to run on every PR automatically (to me): $0.75 per 100 lines of code. From their example pricing: "if you have a typical pull request with 500 lines of code, it would only cost $3.75 to run CodeGuru Reviewer on it." I wonder if it's actually good enough to justify that price.
I was thinking about the same thing. Even for a small project of 3 developers, it seems like this would rise easily to the $100+/month, for suggestions that may not even be that useful.
at which point you'd stop paying for it I would imagine?
There doesn't seem to be a point to start paying for it, and the time to stop paying for it seems mighty early
that's insanely expensive if you're doing any type of code generation.
This seems like a good incentive not to be generating thousands of lines of codes with each PR, which most would probably consider a feature as opposed to a bug.
exactly. IMO if you're generating code, it should happen at build/compile time not at checkin
What would the rationale be for that?
IMO it’s because generates code is not “source code”. It’s more similar to object files—both are generated by running a compiler.
Generated code is an artifact of the source code. If you need it for something specific, regenerate it from the source when you pull that from your version control system. You're not getting any benefit by storing something that can be generated alongside the means to generate it.
Thank you for your response. The advantages you get:

* Hermetic builds are faster because code-gen only occurs when changes occur in the base code

* Lots of docgen tools don't support incremental compilation

* Diffs in generated code show up as diffs when you change the code-gen tool, easier to isolate changes that occur if your code-gen tool is upstream (say you want entire org on Thrift 0.9.2 from Thrift 0.8)

Downsides I can see:

* Large repo.

* Source of truth is now the generated code, not the source, so someone else using the source could get a different result.

Essentially acting in an empirical mode of operation (i.e. does it provide benefits for cost), and ignoring any philosophical objections, this seems like it could go either way depending on the situation.

> Source of truth is now the generated code, not the source, so someone else using the source could get a different result.

This seems to apply to the other side, I think. Generating from source with different tooling or tool versions could create different results, whereas using the generated code guarantees consistent behavior.

One exception case in our project is documentation- parts of it, like the index, are generated prior to commit. We like having the docs updated in the same commit as the change so there's never an opportunity for mismatch.
I'm not a fan of this. My team does this a lot and invariably it leads to having stuff be unfindable because the source you really want is in your build directory and not in code search. Which is annoying but manageable if I build the project, but partners who don't will have an even harder time of things.

More recent efforts have us check in generated code alongside the "config" files, and automated processes ensure you check in the generated code if you touch the config file. It's much better this way.

We generate source code into a "src/main/generated" directory. IntelliJ picks it up like any other source code with proper Gradle configuration and we ignore it with .gitignore.

Only downside we've found is it can be a pain with searching for references in Github and you have to remember to generate the code, but for the most part it is seamless.

I guess how you manage it depends on your IDE, if you can configure it to work nicely, and how much the generated source changes/needs to be read.

If you split the same number of lines over two, three, ten, etc PRs it still costs the same. If anything it is incentivising code-golf via line minimization.
That’s incredibly cheap, assuming it provides good suggestions. How much time does it take you to review 500 lines of code change, and what’s your time worth? If it takes 10 minutes and your time is worth about $20/hour or more, this service will part for itself immediately.
You seem to assume that the code review tool can do everything that a human code reviewer can.
On the other hand, a machine will not get tired or bored where a human most definitely will if the diff is anywhere near that 500 lines
I disagree. I regularly review PRs with more than 500 changed lines/20+ changed files. I read every single line. I put the same amount of effort into reviewing code as I do writing it; every software engineer should.
(comment deleted)
I agree 100%: if it provides good enough suggestions, it could pay for itself pretty easily on regular day-to-day PRs. (Although: not all 500 line PRs are made equal.)

My original comment was definitely unclear. I actually had two separate thoughts (that I didn't communicate well at all):

(1) if your team has occasional large, automated PRs (code generation, automated refactors, etc), you probably don't want to run this tool on them because of cost, so anyone that has these large PRs and uses CodeGuru probably needs to build a way into their automation to suppress CodeGuru (or build a way to invoke it for specific PRs)

(2) I also wonder if it's good enough to justify the price on regular PRs

We don't have many situation (1) PRs where I work now, but they do come up occasionally. For example, I've used IntelliJ IDEA's structural find-and-replace to do very large automated refactors where CodeGuru would be very expensive and probably provide little value. We also do check in some generated code (we usually don't do this, but there are a couple exceptions where we weighed the tradeoffs and decided checking in the generated code was a better solution, in our eyes).

Although this could certainly supplement a human reviewer, I don't think it could replace one.
Our code reviews are far more "is this the right way to solve the problem?" than "hey, you never use that variable you declared." The latter would be picked up by our linter; I'm having a hard time seeing the value proposition here.

>It’s like having a distinguished engineer on call, 24x7

I don't believe that, regardless of how many times they sprinkle in the words "machine" and "learning".

I'd be very surprised if the service they've announced is a linter.

The announcement says it can even analyze parts your code that are more computationally expensive than they need to be. I'm not sure I understand the skepticism--surely they have among the largest code repositories in the world. Why couldn't they train models on it to look at best practices and even compare code practices to different metrics.

Isn't this just calculating the cyclomatic complexity?
No - OP meant computationally expensive, not cognitively expensive. Two nested for-loops can be O(nˆ2) but can have a cyclomatic complexity as low as 1.
It may not be as good as a human, but I highly doubt it's a linter either. It's somewhere in between. If their claims are true, and it has been trained on hundreds of thousands of their own reviews, the AI could have picked up common patterns that are beyond lint but still real mistake a good reviewer would spot.
Yeah or looking at it the other way around. If this would replace your human reviewer, then maybe you would do good to have some serious discussions with your HR department...
Only if CodeGuru gets a lot of the value of a code review. But I think finding actual bugs is a pretty small of it.

A good code base is a team-created intellectual work. For that to happen, you need a ton of collaboration, shared learning, evolution of norms, interpersonal bonding, and practice of key social behaviors (e.g., principled negotiation, giving good feedback, recognizing and rewarding good actions). Automated code review gets at none of that.

You don't replace the human code review. You supplement it. An AI that can replace the code review would need to be an AGI that "sat" down with your team and understood the architectures and meaning in your code. If it can't say "WTF?" it can't do a full code review.
You really believe this will free you from traditional code reviews? I would treat it as an advanced linter, and from the pictures looks like that's how it integrates itself in github.
“Time to hire some code golfers, we’ll put the whole app into one line of code.”
"CodeGuru says this should be separated into more lines..."
Really though, why charge by-the-line on this kind of product? Imagine if CodeCommit or Lambda charged you by the line too!
(comment deleted)
That sounds pretty terrible to be honest. I cannot imagine getting that kind of value out of it (that I would not get with a simple linter).
Don't accidentally commit `node_modules` - that'd be a costly mistake!
Or a 35,000 line XML configuration file.
Now that I think of it, if it's paid by lines of code, it perversely incentives people to minimize the lines of code, no? Does it count white space and comments? Can I minify my code before passing it to this, then unminify it?
Many languages can have code written in them minimized down to a single line. I guess they must have a character count number equals a line qualifier somewhere.
But even then, still pushes people to shorten variable names and other kind of minification.
Can you just remove newlines from files? In most languages they're optional.
Maybe it’s on recommended line formatting
If it is as smart as some people here think it would probably output: "are you f#$king with me?"
Or when a junior dev switches from tabs to spaces
heh.. I wish it were only junior devs.
Or when a junior dev switches from spaces to tabs ;)
If it’s trained on software written by Amazon it’s probably worth the $3.75 just so you can do the exact opposite of what they recommend.
I concur that Amazon's engineers suck. Source: They have rejected me twice.

Obviously have no clue what they're doing.

Definitely no bias there ;P
I don’t have much context, but I’ve never seen Amazon as a technical leader in the industry. They’re absolutely a business leader, and the services they provide can be good, but at a code level I’ve always thought of them as very MVP, if it works it’s good enough.

For code review services I’d expect a level far above this. Maybe they are able to do that, but I don’t have any existing positive bias towards this, and a few things against it.

It'll probably generate irrelevant stats on the engineers to send directly to their managers to use against them in their next review.
You have no idea how much this resonated ^^

Just needded to add an AWS library in my code base and BAM! here is how my console will look on every reload from now on :

:8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/aws-sdk/lib/react-native-loader.js -> node_modules/aws-sdk/lib/credentials/temporary_credentials.js -> node_modules/aws-sdk/clients/sts.js -> node_modules/aws-sdk/lib/react-native-loader.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle. metroRequire @ :8081/index.bundle?platform=ios&dev=true&minify=false:93 :8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/aws-sdk/lib/react-native-loader.js -> node_modules/aws-sdk/lib/credentials/cognito_identity_credentials.js -> node_modules/aws-sdk/clients/cognitoidentity.js -> node_modules/aws-sdk/lib/react-native-loader.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle. metroRequire @ :8081/index.bundle?platform=ios&dev=true&minify=false:93 :8081/index.bundle?platform=ios&dev=true&minify=false:28851 Warning: AsyncStorage has been extracted from react-native core and will be removed in a future release. It can now be installed and imported from '@react-native-community/async-storage' instead of 'react-native'. See https://github.com/react-native-community/react-native-async... reactConsoleErrorHandler @ :8081/index.bundle?platform=ios&dev=true&minify=false:28851 :8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/@aws-amplify/analytics/lib/Providers/index.js -> node_modules/@aws-amplify/analytics/lib/Providers/AWSKinesisFirehoseProvider.js -> node_modules/@aws-amplify/analytics/lib/Providers/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle. metroRequire @ :8081/index.bundle?platform=ios&dev=true&minify=false:93 :8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/@aws-amplify/predictions/lib/types/Providers/AbstractConvertPredictionsProvider.js -> node_modules/@aws-amplify/predictions/lib/types/Providers/index.js -> node_modules/@aws-amplify/predictions/lib/types/Providers/AbstractConvertPredictionsProvider.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle. metroRequire @ :8081/index.bundle?platform=ios&dev=true&minify=false:93 :8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/@aws-amplify/predictions/lib/types/Providers/index.js -> node_modules/@aws-amplify/predictions/lib/types/Providers/AbstractIdentifyPredictionsProvider.js -> node_modules/@aws-amplify/predictions/lib/types/Providers/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle. metroRequire @ :8081/index.bundle?platform=ios&dev=true&minify=false:93 :8081/index.bundle?platform=ios&dev=true&minify=false:93 Require cycle: node_modules/@aws-amplify/predictions/lib/types/Providers/index.js -> node_modules/@aws-amplify/predictions/lib/types/Providers/AbstractInt...

PR I sent yesterday, line changes: +2,703 −3,529. That's like 50 bucks just for that PR.
dude I hope that was the result of updating yarn.lock or similar, otherwise good luck to your reviewer!
Nope, it's mostly code. It's a gnarly PR that atomically delivers a feature (and removes the feature that the new one replaces), but this looks about right for my weekly productivity overall, except I usually submit it in smaller PRs, and the delta is mostly lines added.
If another developer reviewed your code, how much time it would it take them, and how much is that time worth?

If you divide the 50 bucks by that number, you get a cost ratio. If it's lower than the ratio of (benefit expected by automatic code review) / (benefit expected by manual code review), it's worth using.

I guess we can speculate all we want; in the end, only experience will show if the service is worth it or not.

> I wonder if it's actually good enough to justify that price.

If it can spot a lot of issues (performance, security, bug, etc), $3.75 is definitely a good deal to do it once a while but not on every single changes (e.g. fixing a typo in the code comment)

Tools like this should be built into your IDE. No developer ever wants automated feedback at the end of the process in a code review.

There are lots of academic ML review/suggestion tools. Those people come to the table with trials and statistics to assess the quality of their results. Amazon probably copied one of those papers, added a rules-engine to recommend their own APIs, and slapped a hefty price tag on it.

"Amazon CodeGuru is a machine learning service for automated code reviews and application performance recommendations. It helps you find the most expensive lines of code that hurt application performance..." I suspect if AWS is using customers code bases to train its AI models? Another source is to scavenge open source repositories.
"CodeGuru’s machine learning models are trained on Amazon’s code bases comprising hundreds of thousands of internal projects, as well as over 10,000 open source projects in GitHub" - from the article.
Oh no, the code quality is going to be shit.
They might be superficial, but if they did any sort of supervised training (I'm assuming they did), then they probably won't be wrong.
Could Amazon really have that many internal projects? That doesn't seem right.
Maybe you should have continued reading the second paragraph as well?

> CodeGuru’s machine learning models are trained on Amazon’s code bases

> comprising hundreds of thousands of internal projects, as well as over

> 10,000 open source projects in GitHub. Tens of thousands of Amazon

> developers have contributed to CodeGuru’s training based on decades of

> experience in code review and application profiling.

$0.75 per 100 lines of code scanned per month. Wonder if it automagically ignores new lines and javadocs?
I found what it generates.

https://github.com/pediredla/Algorithms/pull/3/files

It looks like a linter, but maybe there is more.

I've never seen a linter tell me problems with code in this detail before:

> You are using a `ConcurrentHashMap`, but your usage of `get()` and `put()` may not be thread-safe at lines: 110, 113, 135, and 137. Two threads can perform this same check at the same time and one thread can overwrite the value written by the other thread.

Many linters are state-aware, for example to catch use-before-init bugs in various languages.

This one could be a fairly simple rule ConcurrentHashMap.get() followed by some code that branches on the result, followed by put() is unsafe. These warnings can be very helpful, but no fairy ML magic needed.

Pretty sure you can turn on that inspection in IntelliJ. The other ones are much more impressive. The 'waiters' one for instance is gold.
I wanted to try it on a single repository, but it requested access to all repositories, public or private and also needs admin access for webhooks. No thanks.
Please do us a favor and consolidate all these Amazon announcements into a single announcement page link. This is ridiculous.
AWS Reinvent is going on. Announcements are done piece meal.
This always comes up when the big tech cos do their annual conference day thing. We don't consolidate the posts, but we do downweight some of them, so you're actually getting less (Amazon|Google|Apple|Microsoft)iness then the system would otherwise be letting through.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I love Salesforce for this.Every release, there's one place with all the changes and new features.No need to play a detective trying to figure out
Was there a list of supported languages somewhere? I couldn't find it.
Only java, it's in the FAQ.
Next step is to provide automated fixes. I've have a side project that does it for Go source code: https://fixmie.com (have plans for other languages and protocols).

But due my Visa situation here in the US (H1B), I'll be never able to monetize it as it's illegal to have a side income. But I think this is just the start and there is an huge opportunity for new startups and projects.

> Next step is to provide automated fixes.

That's a pretty deep rabbit hole. But considering that old "IDEs" with crappy "Intellisense", "Quickfix" or similar were widely sold, there's potential there.

It's not only about the code. For example it also could fixe your import paths if one of your libraries has a CVE and a new version was released. In the case of Fixmie, all the fixes are "suggestion" and GitHub nicely allows you to batch them all and submit with them in a single commit.

(Disclaimer: I'm working for GitHub, but on a different project)

Wouldn't that create worse programmers?
How so? There is so many things that we sometimes forgot. Even experienced developers will make mistakes.
If you relieve the programmer of thinking where his error is and give him the fix, the programmer will not bother to reason out what the solution is, he will simply expect it from you.
Do you really think that what we do and don't have to think about today is at some holy division of things that are best left automated (e.g. garbage collection, platform independence, serialization) and things we have to do by hand? Why is this particular point in time special?

It's a spectrum. Now isn't special.

I think if you provide an IDE that solves everything it will become like a calc, when people stoped making mental excersices in favor of typing the problem and get the insntant result of it.
It is not impossible for you to earn second income on H1-B, it's just that the secondary source would need its on visa petition.

DISCLAIMER: I am not an attorney. More importantly, I am not your attorney. The above is not legal advice. If you desire legal advice, consult a competent, licensed attorney in your area.

What's the probability that such an unsponsored visa would be approved, and within even a remotely relevant timeframe?
Zero. H1B, by definition requires a specialty and a sponsor who can fire you (I don't know the exact phrasing but this is what prevents someone with an H1B from starting their own company)
Before I proceed...are you sure and how sure are you? Would you like to entertain a bet against that nonsensical zero of yours, as if you actually know this.

I am always fascinated by deducive ignorami who pretend to have done the research, like yourself.

(comment deleted)
My hot take is that if you can automatically detect meaningful bugs or author fixes, you need to level up your abstraction.

I think these things make the most sense for Java and Go where there tends to be lots of repetition and lower-order programming patterns.

Unlike say, Python, Lisp, or Rust.

I’ve never programmed in Java or Go in any serious context. What do some of those repeated patterns look like?
Error handling in Go.

Lots and lots of getters and setters in Java.

If you squint this looks like a baby step towards computers writing code.
Not even close. We'll have to nail down how to write unambiguous design specifications in a format that the AI can consume, first.
Maybe we could come up with a consistent language to tell machines what to do first.
I got to preview this service (the code review service) a few weeks ago.

The best thing about it was the recommendations on how to use the AWS SDK better as that's probably got the most potential to drift or make mistakes on

considering how it can be a driver for AWS sales, they should give the service for free though
Code review if done right is a place to learn. Criticizing (and automatically fixing) code style issues and nit-picky can be done by a machine.
Code review is not linter. Code review is a chance to discuss design, scaling, trade-offs and mentor others. I don't think this solution will offer it.
I think this does what you mention and not the former. I would imagine this works best when you have a codebase that heavily utilizes the AWS SDK so it can internally 'paint a picture' of what's going on and provide better architectural decisions and other best practices.

How well it works is beyond me though

Bullshit. You are vastly overestimating the "intelligence" of this overpriced linter. It mechanically detects patterns. See this example: https://d1.awsstatic.com/re19/Screenshot_Catch-Code-Issue_2%... The kind of human-level artificial intelligence that you're suggesting this would have, is science fiction.
Well I stand corrected, would've expected more from a company that knows all the best practices for their own services
> Code review is a chance to discuss design, scaling, trade-offs and mentor others.

Trade offs sure but design and scaling need to be considered _before_ the code review. Maybe an architecture review of sorts? Once you hit code review it's a little too late to reconsider design and scale unless it's a serious issue.

> mentor others

Mentoring is mostly outside of a code review. Sure it can help with that but I don't think that really counts. IMO anyway.

> Mentoring is mostly outside of a code review

Strongly disagree, at least for remote teams.

Working remotely, I've personally found code reviews to be a great way of mentoring less experienced team members.

I also encourage junior team members to review code of more experienced team members.

For big changes, we discuss proposal/API/code reviews as a team.

I've had several people provide feedback that they've learned a lot from reviews like this, and honestly I wish I'd had this kind of mentoring when I started out (I was basically a one-man cowboy-coder for the first 5 years or so of my career).

I know mentoring can be seen as a chore for many, but it can be seriously rewarding too!

If you're discussing design and scaling at code review, you have a serious, SERIOUS, problem. That's what design docs are for.
Fair enough about the design discussions, but I also think this is quite a little bit more than a linter.
Neat idea and a good place to start. But most of the time, people are willingly and sometimes violently opposed to automated free tooling like linters, formatters etc. I’m not holding my breath for that cohort (majority of devs).
I'm wondering did anyone actually tried it? It's not impossible for an automated tool to give valuable feedback over a PR guys. Should be easier than a self driving car I guess
Chess masters have long been combining human and computer analysis... even before computers were able to actually beat humans at chess.
Why do this in the code review stage as opposed to in the code editor linting stage? I'd wanna have these suggestions before pushing.
They should compute the SHA512 hash of lines of code or code blocks from well-known open source projects and then just give you pre-computed "reviews" for those lines/blocks, and then only charge for "novel" code. Otherwise you would need to waste time segregating your original code from the various packages you use. And it seems unfair to charge customers for canned results that can be cached and served at very low cost.
I think you can set it to only scan when new pull requests are made. So you could commit your libraries etc without asking for review and then turn it on only for code you have written.

I might be wrong though.

Yes obviously you would just choose not to submit those irrelevant PRs to this extremely overpriced linter (it's not a code reviewer)
If it was minimally not garbage, they would have run it on any high profile open source project and promoted the results.

Hence, it is pure garbage.

Many (most?) of the checks appear to be specific to Amazon libraries.
It seems good for performance, and probably only that. I don't see anything in the pitch about readability, maintainability, extendability, security, usability/accessibility or portability.
Disclaimer: I work at AWS on an unrelated team. I was not involved in development of this product. Opinions stated are my own, and not necessarily a reflection of my employer. Nothing here is being posted in any sort of official capacity.

There's lots of focus here in the comments on the code reviewer portion, but one of the things I'm most excited about is the profiler - https://aws.amazon.com/codeguru/features/

I do a lot of performance engineering work, and one of my go to tools for visualizing where programs are spending their time is flamegraphs. While you can certainly create them with profilers besides CodeGuru (and I do not work with Java, so I haven't yet had the chance to check out CodeGuru for any of my use cases), I'm super excited about anything that gets more people using them. They make it very easy to see where your optimization opportunities are, and I have personally found them very useful when working with our customers - they're way easier, in my opinion, to go through and explain than just looking at raw perf output or similar.

Seconded. I used them a lot when I worked for myself on old-school single server apps but have struggled to convince my team now that I work on something spread across AWS instances. I'd just brought the concept up again this week for a hackathon project but this looks like we could buy our way to what I want for cheap (compared to overall hosting). I suspect it may pay for itself.
A profiling tool I want to try out—it seems almost magical—is Coz. It can estimate the effect of speeding up any line of code. It does this by pausing (!) other threads, so it gives a 'virtual' speed up for that line.

What's interesting is that this technique correctly handles inter-thread effects like blocking, locking, contention, so it can point out inter-thread issues that traditional profilers and flame graphs struggle with.

Summary: https://blog.acolyer.org/2015/10/14/coz-finding-code-that-co...

Video presentation: https://www.youtube.com/watch?v=jE0V-p1odPg&t=0m28s

Coz: https://github.com/plasma-umass/coz

JCoz (Java version): http://decave.github.io/JCoz/ and https://github.com/Decave/JCoz

I have never heard of this kind of profiling before, thanks for sharing
Yes, indeed! Especially when paired with an continuously running profiler, one can learn quite a bit about one's code. It's actually rather surprising to me that they have not quite caught on earlier.

A bit of an (almost) shameless plug is a project I have been working on at https://blunders.io. A bit similar to the Code Guru profiler, but with a different feature set.

Does anyone know of a good flame graph visualizer for callstacks? Particularly one that allows you to drill down into a stack. Bonus points if you can diff two data sets. I recently built an in-app profiler and am trying to work out the analysis side of things to make life easier for the other developers on my team.
Assuming trust in the AWS code reviews (I mean, that dataset is huge), I suspect this has use in the review portion even without considering profiling. Hoping there is more detail on the ML used, as it appears more adaptable than current rule based code reviewing solutions... here come more and more Dev-focused integrations coming to the code level