Show HN: Crocodile - Better code review for GitHub (crocodile.dev)
I've been working on a code review app for GitHub called Crocodile for about a year. I used to work at Microsoft where we used a tool called CodeFlow for reviewing code and I missed it after I left. I know many other ex-Microsoft engineers feel the same. Here are some of the distinguishing features of Crocodile that are inspired by CodeFlow:
* Comments float above the code instead of being inline. Long discussions that are displayed inline make it really hard to review the code.
* Comment on any text selection in the file, even a single character.
* Comments don't get lost when code changes. I hate it when comments become "outdated" because I rebase or the line is edited.
I also implemented lots of features that I wish CodeFlow had which you can read more about on the blog. [1]
For those curious about the tech stack: it's mostly written in Go with Alpine.js, HTMX, and Tailwind CSS for the frontend. For storage I use PostgreSQL, S3 compatible object storage, and Redis for caching. I use Pulumi for infrastructure provisioning and Kubernetes deployments. Everything is hosted on DigitalOcean.
Feedback is welcome!
61 comments
[ 3.0 ms ] story [ 139 ms ] threadIf you told me that your solution help my team ship faster and saves an hour per engineer per week then that's easy math: your product pays for itself.
Suggestions:
Pricing can be based on the average of hours saved.IMHO if the one-month free trial isn’t enough to know whether a SaaS solution is worth an $8/head price, it’s probably not worth that price.
The trial has always just been a cherry on top.
I haven’t ever used the entire trial for the sake of “trying” and then choosing to not use the product further.
Imagine trying to sell a better WYSIWYG editor for WordPress for example. You can be better, save time, etc. - it's still always going to be a tough sell to charge more than a fraction of WP itself which offers a WYSIWYG editor and everything else.
Wordpress is a great example, rather than build another Wordpress, solve one problem Wordpress doesn't solve (example: Elementor) and do it far better (depth vs breadth).
We are using this Github Action (https://github.com/flowwer-dev/pull-request-stats) for now, it adds stats to every PR, it has helped the team visualize their contributions and how critical it was for us moving forward and as a manager/tech-lead it helps me identify areas of improvements at the individual level. I think it's a simple idea with a lot of depth.
If someone adds a lot of comments, are they nitpicking too much or providing more valuable feedback?
Does lack of reviewing PR hints lack of contribution or person being focused on something?
We've mostly looked at "Total reviews" and "Total comments". When I put the action in place, I told the team that the goal was for everyone to contribute to the process.
I was expecting to look at the stats, make some observations and action items and bring it up during our 1:1. I didn't even need to do that. Everybody took it to heart and contributed in different ways. Our most junior engineer ended up contributing in the most meaningful way and set a new standard for how far we take testing.
At this point, it's pretty clear that I see this more as a tool for individuals. For me, I am a bit constrained on time, I can check the table, notice a variation in the number (the action doesn't show variation so I do it out of memory) and decide to take an hour to explore the contributions. The goal being: how can I help them grow?
As an IC, I perhaps could project myself a bit based on my experience, if I am a junior engineer then I can see who comments the most, explore their comments and learn from them. As a tech lead, I'd want to make sure the tone of the comments are constructive (ie. definitely not toxic).
Another feature I've thought about is "show me all of person X's reviews and comments across all repos". That would have been super useful when I was on promo panels to go through and see if a promo candidate's review comments were useful, constructive, kind, etc.
Thanks for the feedback. I really appreciate it!
I think when it comes to per-seat pricing, it's good to remember that engineers don't put the credit card in. The person with the credit card will want to know ROI before investing. The engineers will want to try out first as a group and if their experience is positive (some of it might be subjective) then they'll need to build a case for why the company should pay for your product and that's where analytics can be leveraged for sales.
Is it still better? Maybe it is.
I don’t think that turning a PR comment into a floating box is a value-add… anyone can do that with a userscript or inline bookmark.
And ultimately…you can’t solve a people problem with tech.
- It sounds like you’re micromanaging your team’s metrics at a rate that’s fractional to one contributor’s hourly cost even though you don’t understand what helps them to be productive. Don’t do that.
- You’re asking for a Show HN to eliminate its price tag and give you everything the poster finds valuable for free, and offer an entirely different product. I’m pretty sure you’re not the audience for this product.
You are seeing micro-management where I am looking for visibility and automation. I physically cannot micro manage my team.
> - You’re asking for a Show HN to eliminate its price tag and give you everything the poster finds valuable for free, and offer an entirely different product. I’m pretty sure you’re not the audience for this product.
Best of luck!
Compliance is a PITA but it's a real thing for companies. How are you planning on clearing that hurdle? I wouldn't want to put you through the vendor survey gauntlet and SOC2 is a lot to ask of a solo dev but the aforementioned IT team could likely be persuaded if you did Type 1 at least.
Do you support features like github suggestions which can be committed easily by author?
Given your history with Microsoft and their internal tool which this is inspired by, how long until github copies some of the ideas?
How many active users does this have?
Are there potential problems you see outside of current product?
I remember a startup providing paid code reviews as a service launched on HN a while ago. That could be a pivot for example in providing more value.
Not right now. The suggestion feature is something I want to add in the future if there's interest though!
> Given your history with Microsoft and their internal tool which this is inspired by, how long until github copies some of the ideas?
No idea! But CodeFlow had been around for many years before they acquired GitHub so they've had plenty of time to do it! I think this is an existential fear that anyone building code review tools has. For Crocodile, I think the long term strategy will be to decouple this from GitHub so that it can work with any source control provider. I architected it this way from the beginning but decided to target GitHub first because they have mindshare and because it made it really easy to dogfood since I host my code on GitHub.
> Are there potential problems you see outside of current product?
There's the one you already mentioned, which is GitHub improving their PR system, but there's not much I can do about that. Current problems I'm looking at are roadmap related. I have a few ideas of features I'd like to add, but no real indication on what to do next. Hence, posting on HN to get feedback. :)
Thanks for taking a look and the feedback!
> * Comment on any text selection in the file
> * Comments don't get lost when code changes
This addresses my main pet peeves with GitHub/Bitbucket reviews!
2 questions:
With regards to comments not getting lost, how well does this work across rebases and force pushes?
Are you considering supporting other products like Bitbucket and Gitlab?
Supporting other VCS providers, even non-Git ones, is on my list. The underlying data model is agnostic to GitHub so it's just a matter of having demand for it and doing the work!
https://app.crocodile.dev/reviews/rwsfSKbgZoSt?change=README...
The best code review tool I've ever used was a tool at Google called Critique.[0] They've open-sourced it as Gerrit[1], but there are sadly no hosted versions available for under $15k/yr, and it's complicated to self-host.
I've been using Reviewable, and my experience has been good not great. Github's native code review has caught up a bit, but Github's review tool falls apart if your review lasts more than one round.
Here are my gripes with Reviewable:
Steep learning curve - Every new developer who joins the team spends their first few reviews being confused and frustrated by Reviewable.
Performance - Reviewable has awful performance. It takes about 10 seconds to load a code review. It seems like it's doing some odd websockets stuff where sometimes my "connection" to Reviewable will drop and I can't add comments. I've never experienced this with any other web app. It's gotten better over the last few years, but it's still annoyingly frequent.
Complicated configuration - I just want the reviewer to be able to hit an "LGTM" button to mark their approval. Reviewable's decision about when a PR is approved is based on this complicated function combining whether the reviewer typed the text ":lgtm", how many people looked at the review, whether they also hit the approve button. Each repo has its own configuration, and I can't make org-level changes without changing every repo one at a time.
Excessive permissions - This might be a Github thing, but you can't grant Reviewable permissions to a particular private repo - you have to grant it permissions to all of your private repos. Several developers who join my team need to create a dedicated Github account to avoid exposing their other private repos to Reviewable.
Thread state is unclear - The options are "discussing", "satisfied," "blocking," or "working," and it's not obvious who's supposed to move the thread to what state at what point.
No development - I've been a paying customer of Reviewable for about 7 years, and I can remember only 1-2 minor features that have been added during that time. They haven't updated their blog[2] in 6 years, and they've never communicated with me as a paying customer to tell me anything they're doing.
I checked out Crocodile, and it looks like it has potential. I'm not sure I'd pitch it to my team to switch yet. Here are some of my thoughts:
* When do the reviewer's comments become visible to the author? One of the must-have features for me is that both author and reviwer(s) can prepare a set of notes, but they're not visible to anyone else until they hit "publish" to share them with the team. Sometimes I make comments in one spot, and then as I read more of the code, I revise a previous comment. If all my comments publish immediately, I can't revise comments like that. Github, Reviewable, and Gerrit all support a flow of preparing comments and then committing them in a separate step.
* Crocodile touts the floating thread thing, and I've never used a tool that has it, but it doesn't seem better to me. Inline comments do break the flow, but floating comments actually cover up the code and prevent me from reading it. I see I can close threads, but I can't figure out how to get them back.
* Being able to comment on character-level granularity is cool!
* I think your thread state is better than Reviewable's, but I'd prefer an even simpler model where threads are either "open" or "resolved." When an author responds to a comment, the default action is to resolve it, but the author can override the default and leave it "open" if their comment is asking for clarification rather than declaring a fix. The reviewer can reopen a thread if they feel that the author has misunderstood the note. 95% ...
I've made a couple notes based on this to help me prioritize next. Thanks for taking the time to provide all this amazing feedback!
> Steep learning curve
Guilty as charged. I think it's cohesive and efficient once you learn it but the learning curve is quite steep. We keep trying to think of ways to flatten it a bit but haven't had any great ideas so far. If you have thoughts on this -- or could point us to a great UX designer with dev tools experience -- it would be much appreciated!
> Performance
There are some obvious things we're working towards to improve it but these days most reviews load in 3-4 seconds for me, even on a not-very-awesome laptop. What kind of platform are you seeing 10 second load times on, and how's your Internet latency to us-central? Please open an issue and we can work through it together.
> Complicated configuration
Yeah, there's a lot of support for legacy and highly customized workflows in there. But these days the primary and default integration is with GitHub's approve/request changes workflow, which gives you one-click approve -- have you tried using that? (Efficient multi-repo config updates are also on the todo list, but never quite rose to the top.)
> Excessive permissions
This is an unavoidable side-effect of using GitHub's OAuth authorization mechanism. GitHub Apps allow finer-grained permissions but they didn't exist when we created Reviewable and don't have a good answer for listing PRs, which would make our dashboard significantly less useful. We do have a transition planned out but at this point our main customers are enterprise folks who run their own GHE Server and don't much care about fine-grained permissions, so the ROI doesn't pencil out.
> Thread state is unclear
Huh, I'm surprised by this one. The basic flow is: 1. Reviewer creates a discussion (disposition defaults to Blocking). 2. Author responds with questions/comments (disposition defaults to Discussing). 3. Reviewer clarifies (disposition unchanged) or clicks the big Resolve button (disposition changes to Satisfied, discussion is resolved). 4. Author addresses the comment and clicks the big Done button (disposition changes to Satisfied). 5. Reviewer checks and, if satisfied, clicks Resolve (disposition changes to Satisfied and discussion is resolved).
Basically, at every step you either reply to keep the discussion going, or click the button to indicate that you're fine with closing it out. You shouldn't even need to care about the specific disposition unless you're trying to run more advanced workflows.
> No development
Yeah, I pretty much abandoned the blog and should probably take it down. However, we do ship new features regularly and post updates on Headway [1]. These should also pop up as the "red circle new stuff counter" in the UI, but perhaps your browser is blocking that third-party connection. (Headway only gets major feature posts, but there's a lot more work going on in the background that's only reported on the enterprise changelog [2].) I'm not a fan of spammy email newsletters so we don't send those.
Again, thanks for the feedback, and don't be a stranger -- it's easy to reach us through email, chat, issues, etc., and we respond promptly to every message. Even if we have to hunt them down on HN. :)
[1] https://headwayapp.co/reviewable-changelog [2] https://github.com/Reviewable/Reviewable/blob/master/enterpr...
>>Thread state is unclear
>Huh, I'm surprised by this one...
>Basically, at every step you either reply to keep the discussion going, or click the button to indicate that you're fine with closing it out.
I think there are two problems here:
1) It's not obvious enough that a user is supposed to declare a state after writing a response 2) There are more states than necessary
For (1), the UI flow doesn't hint to the user that they're supposed to do anything after they type their reply. I just tried it and I see this:
https://i.imgur.com/DimycTB.png
It sounds like you're saying the expectation is that users click the circle at the upper right and then choose a state, but that hasn't been obvious to anyone I've introduced to Reviewable.
For (2), this is more personal opinion, but I think comment threads are similar to Github issues in that there only need to be two states: closed or open. The states Reviewable offers to me as the author are:
(1) Discussing, (2) Satisfied, (3) Blocking, (4) Working, (5) Pondering
1, 3, 4, and 5 are all "open" for my purposes, and (2) is closed. It's rare in my experience for a developer to respond halfway through a round of review to say they're still working on or pondering a comment, so I don't need dedicated states for that. They can just keep it open and write a comment like "still working on this one!" I don't see a difference between (1) and (3), as all notes are blocking until they're resolved.
My somewhat controversial opinion is that the author and reviewer should trust each other enough that the author can respond to a note and mark it resolved without awaiting confirmation from the reviewer who wrote the note. It's rare in my experience for an author to incorrectly resolve a note, and when it happens, the reviewer can just reopen it and say, "Hey, I think there was a misunderstanding on this one, and the changes haven't addressed this note." I think forcing the flow into "Discussing" -> "Pending resolution" -> "Confirm resolution" just adds needless friction.
>You shouldn't even need to care about the specific disposition unless you're trying to run more advanced workflows.
We don't, but I see it as a missed opportunity because there's useful information there.
In Critique, it was easy to see at a glance whether a note was active or resolved (active notes had an orange background and resolved notes had a gray background IIRC). In Reviewable, we have to read all the notes more closely to see which are active and which are resolved.
>I'm not a fan of spammy email newsletters so we don't send those.
I don't like those when it's baldly trying to wring more money out of me, but I like it when vendors tell me what's going on and how they're improving the product.
> 1) It's not obvious enough that a user is supposed to declare a state after writing a response
That's because you don't need to declare a state after writing a response. :) The basic workflow is simple: click the button if you're in favor of ending the discussion, otherwise write a response. For more advanced workflows, yes, you can explicitly select a disposition from the dropdown, or prefix your message with a magic keyword to do so automatically.
> For (2), this is more personal opinion, but I think comment threads are similar to Github issues in that there only need to be two states: closed or open.
You're conflating two separate things: each discussion has a state (open -> in progress, closed -> resolved in Reviewable parlance), but so does each participant. The discussion's state is a function of all the participants' states, allowing for a consensus to develop in multi-party reviews without giving any one person the power to unilaterally close the discussion. There are three basic participant states:
Blocking -> prevent discussion from resolving Satisfied -> in favor of resolving discussion Discussing -> neutral; not my monkeys, not my problem
The other two states you mentioned are specialized and affect other features besides the discussion state:
Working -> like Blocking, but also "keeps the ball in your court", so Reviewable knows you're still responsible for further progress on this discussion and won't list it as needing a reply from other participants; it's usually employed by a PR author who agrees with a request but wants to reply with further details before they've actually done the work.
Pondering -> this prevents a draft from being sent when you publish; useful for "notes to self" as you read through the code that may or may not become actual issues you want to raise, and that you don't want to accidentally publish as-is.
It's not a simple system but we figure you've got GitHub for that, so we choose to err on the side of powerful.
> My somewhat controversial opinion is that the author and reviewer should trust each other enough that the author can respond to a note and mark it resolved without awaiting confirmation from the reviewer who wrote the note.
You can do that too! If you initiate a discussion as Discussing, then if the author clicks the Done button (switching to Satisfied) the discussion will automatically be resolved, with no further input from you. You can change your default disposition for discussions created as a reviewer if that's how you prefer to work; see https://docs.reviewable.io/discussions.html#resolution-workf... for details.
> In Critique, it was easy to see at a glance whether a note was active or resolved
In Reviewable, resolved notes are collapsed into the title bar (if just resolved) or into a gutter icon (afterwards), so usually you won't even see them. I'm confused when you say that you can't tell which are active and which are resolved, but perhaps we differ in our understanding of what "resolved" means.
> "Being able to comment on character-level granularity is cool!" This is something that I really like too - I'd go so far as to say PR titles, file names, etc, nearly everything should be commentable.
Have you checked out graphite.dev? Its iterating rapidly and incorporating a lot of what Phabricator and Gerrit did well.
Not sure if you've gotten this feedback, but the demos on your website are illegible. They're squeezing a terminal session into a tiny 350px width box:
https://i.imgur.com/UBnNWNT.png
Also, would be good to disclose that you're the co-founder if you're commenting about it publicly.
Crocodile has its own organization and member list. Billing is based on the max number of users in the Crocodile org for the month.
I end up using open unresolved comments as the basis for changes requested now. It's hard to filter for those though.
1. More comment states. Comments start as "open". The workflow I had in mind was that once the author addresses it, they would mark it as "resolved". The reviewer can go to the comments view (in the meta section on the left nav) where they can filter to resolved comments. If it looks good, they can mark it as "closed" or "open" again if they want more changes. I have this as a suggested workflow in the docs. [1]
2. The iterations section in the left nav lets you compare any two iterations. You can use this to see what has changed since your last review. Crocodile even indicates which iteration your last review was on to make this easy.
[1] https://www.crocodile.dev/docs#commenting
One of my favorite features was a panel with every comment on the PR, sorted and organized by status. Because all files in the PR were preloaded clicking a comment instantly took you to the specific code and version. So good!
A couple improvements I made:
* You can see the whole discussion. The last time I used CodeFlow it only showed one line of the comment which often wasn't enough for me to remember what it was so in practice I always had to click on every single one to see the whole thing. Not a big deal with CodeFlow since it was so snappy as you mentioned.
* It shows a preview of the code that was commented on to provide context.
Thanks for taking a look! Great to see other CodeFlow fans out there.
It does seem that the demo review is broken in Safari, I get a JS error and the UI doesn't seem to work completely:
> [Error] TypeError: e.connect is not a function. (In 'e.connect(l,s)', 'e.connect' is undefined)
Also, is there any way to keep up-to-date on the project? I don't see a newsletter or Twitter link on your homepage.
No newsletter yet, but you can follow me on Twitter for now @jameslao.
The files you've chosen to use don't appear to show any difference between iteration 1 and 2, so one of your major features doesn't do anything. Is that a bug, or accidental? (I'm using Firefox 101 on MacOS 11.4).
My personal dumb suggestion: give a few demos, showing off the very worst points of code review hell, and how crocodile fixes each one. Make it a game. E.g.
"You're halfway through a large code review, and Sally has just added 2 new commits, ugh. Challenge: find the button to see the new additions, then decide if you want to include them in your current review, or review this iteration first and the new additions separately."
"Simon has just added a merge commit that fucks everything up, all the files look weird. Challenge: there's a way you can trivially see that the PR before this extra commit was looking great."
"You can spot a new method that was introduced, and you'd like to see the places where it's being called, but that's a lot of scrolling back and forth. Challenge: find how to show code hints on a selected piece of text"
One thing I've always wondered is why all these review tools use centralised databases. Git itself is a distributed model and reviews tend to boil down to code comments on set of lines or characters. I'm always surprised no one has created a review tool that ships around reviews like patch files. Even if there was a server as an option, like github, I could then work offline and build little tools to help make my review process more efficient.
I suppose it's not quite as easy to monitise as it's decentralised, but I'd love to see one crop up some day. Then my review process can match up with my coding process.
Either way this looks like a big improvement in some areas over the GitHub tools so I'll definitely be checking it out.
For any PR that is nontrivial I will pull it locally so that I can more easily navigate to functions/data types that are used by or changed in the PR. It would be nice if the review ui provided a way to click through to the definition of a symbol that appears in the code. (I think gh does this when browsing code for some languages.)
A related helpful feature would be the ability to see “what calls this”. Currently I have to do this kind of review with ‘git grep’, after pulling locally.
I’d also love to be able to toggle into ‘git blame’ for a given bit of code, in order to better understand why the code is in its current state
I’ve been looking for this feature forever. A changed line might impact code that wasn’t changed. There wasn’t a convenient way to comment on those. Looking forward to trying this.