> Are you a software engineer looking for ways to make your workflow more efficient, or tired of manually sifting through long commit logs to understand pull requests? Introducing the GPT summarizer GitHub action: a powerful tool that leverages OpenAI’s latest and greatest large language model to generate concise and informative summaries of the entire pull request, as well as descriptions of the changes to individual files and individual commits.
This kind of stupid.
1. At best, all this can do is look at what you actually committed, but that's not going to tell you the most useful stuff that should be included in a commit message, like why you're making the change in the first place and what your intent was (which may deviate from what you actually did).
2. If you wrote the damn commits, you should be able to describe them yourself. If you can't, something's wrong.
The only case where I can see a tool like this being useful, instead of a bad habit, is summarizing old commits, where the committer didn't put anything useful in the commit message.
It isn’t stupid if it works. If this allows an engineer to spend 15 mins less time while generating good enough PR descriptions, that’s a huge win, especially if the developer isn’t a native English speaker. IME people after getting whatever they’re working on to pass tests don’t feel like writing prose about their work. This would significantly reduce the burden of work which is perceived ‘low value’ and ‘boring’ by most (yes I do review code and yes I will reject a PR without a summary).
> It isn’t stupid if it works. If this allows an engineer to spend 15 mins less time while generating good enough PR descriptions, that’s a huge win...
It can't work, unless it's hooked up to a brain scanner and can read your thoughts.
It seems more like dumb thing that people who don't know what they're doing will think is smart.
You see a tool which doesn’t do all you want it to do and say it’s useless where it clearly can make the act of actually doing what you want simpler and faster. I don’t understand why you think it’ll do 100% if 80% is already good enough and very helpful to get the last 20% done.
It's a path to laziness and skipping the 20% that's actually helpful in favor of a useless 80%.
I see a useful use case for using a tool like this after the commit messages are written (e.g. auto-generate these summaries on the fly when browsing history), but using it to write commit messages themselves is a terrible idea.
The reality is that writing a decent commit message, by some reasonable standard involve information not encoded in the contents of the commit (or any linked data). The contents of the commit are almost a side-effect of the actual work.
As such, no amount of “gpt magic” will solve this.
I think the biggest problem is GPT is very confident in its mistakes. If it writes something incorrect, and you don’t know it’s doing it it’ll go completely unchallenged.
It’s correct often enough that you can get complacent.
Those 15 saved minutes you will have to spend to validate that generated message actually makes sense. And if you are not careful, you will have to spend much more time 'rewriting history' later.
Also if an engineer has troubles describe change than maybe you don't want that change to end up in you repo.
Information that can be automatically extracted from the delta is the least important part of a good commit message. I can figure that part out from looking at the commit later. Instead, what I really want to know is why you made the change. What is the context? What problem were you solving? Why this way and not other ways you might have solved the problem?
A summary of what the change does is still useful, and I'm glad this can now be automated, but it's not the main benefit of a well written commit message.
What I have always been told for writing good commit messages is that the message should tell you what the commit does when applied to the code. The place for these what/why messages would appear to be in a ticketing system of some kind, referenced by ID in the commit message.
What you're asking for would result in extremely bloated messages.
There's no meaningful limit on message length for a reason. Don't bury the context in some external system that may change or go offline, let it live in the commit where it is safe and easily accessible.
On the contrary, git commit messages and checked in docs are the only places that are guaranteed not to be lost to history. Ticketing systems change, get migrated to with data loss or are access limited. Git is forever. The commit message is a much better place for explaining what was done why than some obscure Jira instance.
> The place for these what/why messages would appear to be in a ticketing system of some kind, referenced by ID in the commit message.
No. It's best to assume that all information in your ticketing system will be totally lost.
The commit message should have the "what/why" directly in them, because that's where you'll first look when you want to find that out (why add an extra hop), and it would require extreme incompetence (which is not unknown) to lose information stored there.
Walk through a hypethetical ticketing system migration with your tech lead. Assume all external links are broken, and see how expensive it becomes to answer questions about your own codebase. "why did we add special case X? is it still applicable? It interferes with new feature request Y, can we remove it to reduce total risk/cost of implementing Y?"
Integrating with another tool is fine, but not at the expense of your commit history!
> I agree at a gut level with this. Do you have a source that this is more widely believed? I would like to bring this to my tech lead.
This was learned from hard experience.
During my tenure, we've migrated corporate wikis/"collaboration systems" four times, and we've used at least as many ticket systems. We also migrated from webex to Teams a few years ago. Each migration resulted in at best a severe partial loss, and in some cases a total loss. The most stable resource is the codebase.
Basically, whenever anyone does a migration for you, they will lose things, and they might even think losing them is a "clever" shortcut that will make them more efficient. Our recent documentation system migration only covered documents modified in the last year as a corporate cost-saving measure, and we were on our own for everything else. Weak/understaffed teams did a bad job. They retroactively implemented a retention policy on Webex that wiped out a lot of valuable recordings, and then the coup-de-grace was the migration to MS Teams, which wiped almost everything else (basically everything that the original recorder didn't conscientiously download and save). We recently migrated to MS ADO as our work tracking system, and that was the only time I've seen old stories/bugs migrated at all, and even then we still lost all embedded images.
Here's a list of things in order of least to most likely to be lost:
1. Head/tip of your project in source control
2. Source history
3. Documentation stored in a central location as files you can easily copy
4. Documentation stored in some system (like a wiki)
Please read a few commit messages for the git project itself.
That is how git was designed to be used. Maybe that provides some perspective of just how bloated commit messages can be, but also how useful they then become.
Haven't tried this solution but my first thought was of those (.NET) libraries that would generate comments using a functions signature (function name, parameter name, etc.). Remember some senior devs at the company I worked for at the time being mesmerize and overusing this. We ended up with comments that could easily be inferred from reading the signature...
I think you would be surprised how PRs work in many companies. We are encouraged to write small PRs and many will have automated tools that points to a Jira ticket in the description or as a comment. For this reason many leave empty descriptions. In such cases this would still help.
From my experience people rarely read the commit message s, so it is better to put the why part into comments in the code, not git history. I mostly write the detailed why part in the commit only for refactoring changes.
My experience is that many changes span several blocks of code in a repo, so there's not always one single location to put the comment. There's also a blessing in that git commits don't need to be maintained like comments. Comments can drift if not maintained. A commit is only observed in-context.
Most developers seem to be poor "commit historians" by default, I'll admit. If you have the energy and consistency to enforce good history, it eventually becomes self-sustaining.
If I'm adding something weird and funky that requires a long explanation of context, it goes in a comment (or even an ADR). If I'm removing it, it goes in the commit message.
Why write meaningful commit messages at all then? Should we perhaps “git commit -m foo” all the time and call it a day?
My experience here varies wildly; I’ve worked on projects where commit history was carefully maintained and documented, as well as on projects with random squashing or random commit messages. Both had their own benefits and came with a cost too.
For larger, more complex or more mature projects, going through git history can be a common task. So here I’d prefer meaningful commit messages; git blame/bisect/etc gets easier, and sometimes any additional piece of docs has immense value. For smaller, simpler projects or prototypes, where history isn’t used that often, I don’t care that much.
Not proud of it, but honestly > 50% of my commit messages are "wip" or similar.
Source backup is one of my primary uses for SCC, and when I step out for lunch, or need to knock off for the day, I commit where I am, and push with a "wip" (work in progress) message.
The feature branch name I merge is probably the most useful. It's an aggregate of all my commits and what I'm working on.
I do this too, but when I come back from lunch I git reset HEAD~1 and keep going till I have a meaningful piece of work to commit on its own. You can also use interactive rebase to reorder or squash commits if you realize that for readability reasons it makes more sense for a refactoring you started later to appear earlier in the stack or for a subpar commit and its subsequent fixup to be combined. This makes the final history more easy to read for reviewers, and (after it's merged) more easy to read for future you and other devs on the project too.
interactive rebase is (IMHO) the golden ticket for this. Unfortunately, if you're using git as a backup then that means you pushed the commits, so a `git push -f` will be required. I do that all the time and think in general we are unnecessarily scared of it. As long as you're the only person working on the branch, a push -f is NBD. Never do it to master though or a shared branch unless you plan to bring donuts or something in later as an apology.
Yeah where I work, we strongly encourage interactive rebase on solo feature branches.
In fact, even the hairiest long-lived branches can get cherry-picked apart into several "layers" of branches to get reviewed and merged piecemeal. It's really easy if you're starting from too many commits. It gets harder (not impossible) if individual commits commingle said layers.
The commit message is also for future me. Future me doesn't remember what today me can easily recall about the last week of debugging and reverse engineering about how the system works. A thorough commit message can really help when trying to figure out bisect bugs point to a particular commit or to better understand individual lines of code when reviewing code with git blame.
The main reason i read commit messages, is when i change some code i want to know why it was introduced in the first place to ensure i am not regressing some feature. The why is important, even if its just to know that there is no big story.
Well, there's one particular exception that I find super relevant -- my own commit messages. There are a number of times when I know I made a code change, and can't remember why, and my own commit messages in the git log remind me.
Hopefully as a side benefit others also find them helpful, but my present self is always grateful to past-self's conscientiousness!
From my experience, people rarely write meaningful commit messages or comments in code, and they expect you to just glean everything you need to know by simply reading the code itself.
When you find a bug later on related to a specific commit, the "why" in the commit can help explain the "what" that resulted in a bug.
It's all in addition to the changelog and other documentation if following a defined release methodology.
Personally I like having the "why" in a comment (with a pointer to the bug tracker item) if adding code. That doesn't work as well when deleting code, and for that, the commit message works as well as anything else.
Design decisions are not the "why"s. Commits are point in time checkpoints why development took a certain path. Let's take an example.
If a what looks like this: Store user full name in session table
The why can look something like this:
The full name is authoritatively stored in the user database and should be looked in using the user name. However, during deploys of the Raticate system, the cache gets invalidated and we need to quickly refresh it. This leads to a thundering herd system, as in incident 5244. By caching the full name in the session table we sidestep this problem entirely.
The reason why the Redis cache was not chosen for this data is because the session object is handed to the checkout system for processing, which can for security reasons, as described in document 7745, not access Redis.
That's one way but it requires a lot of discipline, and even then you'd have to manually go thru them to fish any "commit A introduced something, commit Z reverted that 40 commits after coz reasons" out of it.
I disagree. Unless there’s some smarter way to do this, I use the commit messages to find the kinds of historical changes I’m curious about. I want to know what changed. I’m not sure how one would do this unless they’re looking at every commit.
The “why” goes into the PR and more importantly, engineering documentation and inline comments.
Plus, a commit message seems woefully plain and too short to properly communicate these things.
The beauty is that this isn’t prescribed. We can all do what makes sense for our contexts.
I’ve used 35-year-old commit messages to help understand a long-standing issue, decades after all other related organization tooling and data had disappeared.
I'd be more excited to use GPT to draft a summary of release notes by scanning all the new PRs in a release, summarizing what they are, and dividing them up into categories (bug fix, feature, breaking changes, etc.)
While we're making tools for developers, make it so that I can yell at my computer, have it run voice recognition on it, and have ChatGPT interact with Jira for me.
In addition to the other issues (eg. it can only explain the what) I'm low-key disappointed that the repository isn't using itself. It would undoubtedly have lead to an increase in commit message quality here.
One problem with this I can see is that it's quite easy to write a delta that's larger than 4096 tokens minus prompt length (the maximum amount of context gpt-3 can process), and if the delta's so short it's less than that then it's not particularly necessary to use the AI to generate the commit anyway.
Is there a name for the phenomenon where new automation is disparaged by comparing it to the best human outputs, rather than considering what the average human output looks like and asking if it's a net improvement on that?
Like, in this case, I would guess that the vast majority of commit messages are something like "foo", "fixed", "argh", "stupid CI thing try 10", etc. Pretty sure GPT can improve on those.
Wait, if the tool can derive the commit message from the delta, then why does it need to be part of the commit? You could improve the algorithm behind the commit message later.
This is where my thoughts went as well: it's like a data structure that includes 2 numbers and their sum. Then I thought that this particular operation is not just a sum and having the memoized value could make sense: using GPT to generate the commit message for a few thousand commits would be pretty expensive.
Nice! Thanks for sharing. This feels far more useful to me than OP. The idea of having to push your code to github to have it generate the commit message makes me immensely sad (even for repos that are already on github, but especially when they are on gitlab).
I can absolutely see how a tool like this could provide a useful hook, to reject any commit messages that do not provide information above what GPT3 can do.
I get the appeal of offloading toil to AI. But gotta be careful to not mistake thinking for toil. Sometimes thinking is hard and not fun but that doesn't make it toil.
Writing good commit messages is thinking, not toil. 90% of the time is even easy, one line description, good to go. The other 10% is when you gotta do the thinking, though. Extracting the relevant changes from the incidental ones, summarizing the whys, the pros and cons, the weird interactions that lead to the specific choices made in the code. This is not toil, it's thinking. It's not even a matter of whether AI can do it well (for this 10%, I'm *very* skeptical it can), the issue is that this is the kind of thinking you should *want* to do, that you should want to get better at.
TL;DR: this feels like asking the computer to jog for you and expect to loose weight.
These things are going to drive me crazy. I think stuff generated by an AI like this should have to be tagged. It’s infuriating to waste one’s time trying to read an interpret a message (or some code), trying to understand the author’s intent, only to realize the author merely pressed a “generate a bunch of code/text” button and I’m spending time trying to interpret shit the author didn’t even spend time writing.
My experience with this hitherto relates to terrible scaffolded/generated code. I’m scratching my head wondering “why did s/he do it this way?” only to realize the nonsense code was lazily churned out by some generator. I wish this stuff was labeled clearly.
It seems to me that a lot AI is used as either tool to solve made up problems or as a bad solution to real problems. This seems like the former. Who is the target audience? I'm baffled.
Please add functionality so that you can show some sort of badge on your GitHub profile based on your use of this. It would be a great tool to sort out who actually understands the purpose of a commit message.
88 comments
[ 3.0 ms ] story [ 140 ms ] threadThis kind of stupid.
1. At best, all this can do is look at what you actually committed, but that's not going to tell you the most useful stuff that should be included in a commit message, like why you're making the change in the first place and what your intent was (which may deviate from what you actually did).
2. If you wrote the damn commits, you should be able to describe them yourself. If you can't, something's wrong.
The only case where I can see a tool like this being useful, instead of a bad habit, is summarizing old commits, where the committer didn't put anything useful in the commit message.
It can't work, unless it's hooked up to a brain scanner and can read your thoughts.
It seems more like dumb thing that people who don't know what they're doing will think is smart.
I see a useful use case for using a tool like this after the commit messages are written (e.g. auto-generate these summaries on the fly when browsing history), but using it to write commit messages themselves is a terrible idea.
As such, no amount of “gpt magic” will solve this.
It’s correct often enough that you can get complacent.
Communication is part of the job. How many well-paid and creative professions get to just "nope" out if their responsibilities like that?
Also if an engineer has troubles describe change than maybe you don't want that change to end up in you repo.
A summary of what the change does is still useful, and I'm glad this can now be automated, but it's not the main benefit of a well written commit message.
Starting point. After that it should up to the committer to make modifications and upstream the message. For those cases, seems good enough for me.
What you're asking for would result in extremely bloated messages.
No. It's best to assume that all information in your ticketing system will be totally lost.
The commit message should have the "what/why" directly in them, because that's where you'll first look when you want to find that out (why add an extra hop), and it would require extreme incompetence (which is not unknown) to lose information stored there.
Integrating with another tool is fine, but not at the expense of your commit history!
This was learned from hard experience.
During my tenure, we've migrated corporate wikis/"collaboration systems" four times, and we've used at least as many ticket systems. We also migrated from webex to Teams a few years ago. Each migration resulted in at best a severe partial loss, and in some cases a total loss. The most stable resource is the codebase.
Basically, whenever anyone does a migration for you, they will lose things, and they might even think losing them is a "clever" shortcut that will make them more efficient. Our recent documentation system migration only covered documents modified in the last year as a corporate cost-saving measure, and we were on our own for everything else. Weak/understaffed teams did a bad job. They retroactively implemented a retention policy on Webex that wiped out a lot of valuable recordings, and then the coup-de-grace was the migration to MS Teams, which wiped almost everything else (basically everything that the original recorder didn't conscientiously download and save). We recently migrated to MS ADO as our work tracking system, and that was the only time I've seen old stories/bugs migrated at all, and even then we still lost all embedded images.
Here's a list of things in order of least to most likely to be lost:
1. Head/tip of your project in source control
2. Source history
3. Documentation stored in a central location as files you can easily copy
4. Documentation stored in some system (like a wiki)
5. Tickets/stories in your work tracking system
Please read a few commit messages for the git project itself.
That is how git was designed to be used. Maybe that provides some perspective of just how bloated commit messages can be, but also how useful they then become.
There is a great blog post about it: https://dev.to/jacobherrington/how-to-write-useful-commit-me...
It boils down to the following commit message template:
Most developers seem to be poor "commit historians" by default, I'll admit. If you have the energy and consistency to enforce good history, it eventually becomes self-sustaining.
(Those are not the only situations, of course.)
My experience here varies wildly; I’ve worked on projects where commit history was carefully maintained and documented, as well as on projects with random squashing or random commit messages. Both had their own benefits and came with a cost too.
For larger, more complex or more mature projects, going through git history can be a common task. So here I’d prefer meaningful commit messages; git blame/bisect/etc gets easier, and sometimes any additional piece of docs has immense value. For smaller, simpler projects or prototypes, where history isn’t used that often, I don’t care that much.
Source backup is one of my primary uses for SCC, and when I step out for lunch, or need to knock off for the day, I commit where I am, and push with a "wip" (work in progress) message.
The feature branch name I merge is probably the most useful. It's an aggregate of all my commits and what I'm working on.
In fact, even the hairiest long-lived branches can get cherry-picked apart into several "layers" of branches to get reviewed and merged piecemeal. It's really easy if you're starting from too many commits. It gets harder (not impossible) if individual commits commingle said layers.
I find the latter quite common.
Hopefully as a side benefit others also find them helpful, but my present self is always grateful to past-self's conscientiousness!
Out of three (commits, comments, docs), the commits are least approachable, just few changes in the code will erase original commit in the blame.
By all means write good commit messages, but crystallize the info in them into changelog or other docs every bigger release
It's all in addition to the changelog and other documentation if following a defined release methodology.
Personally I like having the "why" in a comment (with a pointer to the bug tracker item) if adding code. That doesn't work as well when deleting code, and for that, the commit message works as well as anything else.
If a what looks like this: Store user full name in session table
The why can look something like this:
The full name is authoritatively stored in the user database and should be looked in using the user name. However, during deploys of the Raticate system, the cache gets invalidated and we need to quickly refresh it. This leads to a thundering herd system, as in incident 5244. By caching the full name in the session table we sidestep this problem entirely.
The reason why the Redis cache was not chosen for this data is because the session object is handed to the checkout system for processing, which can for security reasons, as described in document 7745, not access Redis.
* What problem were you solving?
* Why this way and not other ways you might have solved the problem?
Or do you think those aren't usually needed for a very good commit message?
That the ticket was open
> * Why this way and not other ways you might have solved the problem?
This is first one that worked.
There, that's those answers covered for like 80% of the developers
The “why” goes into the PR and more importantly, engineering documentation and inline comments.
Plus, a commit message seems woefully plain and too short to properly communicate these things.
The beauty is that this isn’t prescribed. We can all do what makes sense for our contexts.
This just ensures that the “why” is lost when someone comes looking years later.
From experience, SCM history is far more durable than just about any other work product we produce.
Over five decades later, commit history was still available for the Unix sources and could be reconstructed: https://github.com/dspinellis/unix-history-repo
I’ve used 35-year-old commit messages to help understand a long-standing issue, decades after all other related organization tooling and data had disappeared.
`npx add-gpt-summarizer@latest`
The CLI will walk you through all the setup steps. source code: https://github.com/soof-golan/add-gpt-summarizer
Like, in this case, I would guess that the vast majority of commit messages are something like "foo", "fixed", "argh", "stupid CI thing try 10", etc. Pretty sure GPT can improve on those.
Writing good commit messages is thinking, not toil. 90% of the time is even easy, one line description, good to go. The other 10% is when you gotta do the thinking, though. Extracting the relevant changes from the incidental ones, summarizing the whys, the pros and cons, the weird interactions that lead to the specific choices made in the code. This is not toil, it's thinking. It's not even a matter of whether AI can do it well (for this 10%, I'm *very* skeptical it can), the issue is that this is the kind of thinking you should *want* to do, that you should want to get better at.
TL;DR: this feels like asking the computer to jog for you and expect to loose weight.
My experience with this hitherto relates to terrible scaffolded/generated code. I’m scratching my head wondering “why did s/he do it this way?” only to realize the nonsense code was lazily churned out by some generator. I wish this stuff was labeled clearly.