Storing changes as text is a fundamentally bad idea. Not only does it make merge conflicts more likely to happen, as transforms (me adding a function, you renaming something I use in the function) more likely to occur, it also means that formatting is committed and discussed rather than being a matter of personal preference.
Hopefully the next item in this RCS CVS Subversion Git chain is just storing ASTs and transforms on top of them so we can spend less time fixing basic conflicts and discussing formatting.
I've seen the suggestion of VCS/SCM storing ASTs rather than plaintext for a decade now. Are you aware of any projects that are trying to address this?
They are being downvoted for not understanding how git works. If you have a 3-way diff/merge tool for ASTs you can plug it into git and use it today, and you can use it on all existing branches and historical changesets.
The "problem" is no one actually wants to resolve merges that way.
I don't understand how this relates to what I said without more details about what "lost" means, sorry.
If you mean Git had issues finding some specific code motion to show in a diff, you can try one of the other diff algorithms, or adjust the threshold for rename/copy detection. AST-based differs would also suffer this issue; a "nice diff" is not a formal problem and does not have a universal solution.
If you mean you once had a mis-merge that dropped some code you didn't want to drop, this won't go away with AST-based diffs. It will just happen at the token level instead of the line level.
If you mean it's generally hard to deal with collapsing lots of branches with shared history, that's true but would also be true with AST-based approaches. This is the situation something like pijul could help with, but also raises all the other tradeoffs of snapshot vs changeset based approaches.
You’re still committing formatting, and our whole ecosystem of tools is focused around text. You’re proposing a workaround rather than fixing the problem.
This isn't specific to VCS, but I'll drop a link to this thread about structural editors which PaniczGodek has been maintaining for a few years now, just to give it more exposure: https://twitter.com/PaniczGodek/status/1195784199250284545
Git stores snapshots, not diffs, and therefore could just as well be considered storing ASTs. The trick is writing a useful diff/merge for them. Programmers also don’t think in ASTs, arguably even less so than lines. The problem is not as formal as it looks.
Which also means the Lisp AST is not great to diff with.
First, it's too weak - you need to at least recognize special top-level defun-style forms, or you'll generate some minimal diff between two totally different functions just because they both use the same cond pattern or whatever.
Second, reader macros mean you can't really work on the source AST in the first place, unless you also teach the diff tool all your reader macros.
Does that mean git would have to support all known programming languages, and any language syntax change would require a backwards incompatible upgrade of the VCS?
In a sibling comment, I was wondering about projects that are trying ASTs. I haven't read deeply about it, but I recalled that [pijul][1] might be a way forward.
Pijul's patch algebra, and storing diffs rather than snapshots generally, makes it less amenable to these kinds of experiments than Git. Changing your diff/merge strategy would be akin to rewriting your entire project history.
Git stores more than just code, though. The tool already supports custom merge tools, so why not apply your favorite AST mergetool to the relevant source files?
Your suggestion (what comes after git) is throwing out the baby with the bathwater.
I agree with the author re Git as it exists right now, but this feels like an unsolved technical problem that ideally should be handled in your VCS than something to be worked around by social conventions not to do cleanups.
git blame, for instance, has the "--ignore-revs" and "--ignore-revs-file" options which let you specify some commits that can be ignored. This is a little helpful for the sort of code spelunking the author does (which I do very frequently), but it's a manual process. There's no built-in convention to Git for what this file should be named and how to have git blame pick it up automatically. Moreover, you have to create the commit in order to know what its revision is, which means that in squash/rebase workflows, you have to land the style cleanup on the target branch and only then can you add it to the ignore list.
One option that occurs to me is a special marker string to put in the commit message like "Git-Ignore-For-Blame: yes" or something. I'm not sure if there's a better way to do it.
More generically, if the commit is only swapping out tabs and spaces, "git blame -w" ought to take care of it. For more involved reformatting, git diff and git merge both have "ignore-all-space" modes, which will handle things like line breaks.
But really what you want for blames and cherry-picks, I think, is blames at the syntax level. The Git commands only know about lines - git blame runs on lines, git cherry-pick constructs diffs based on lines and tries to re-apply them, etc. But at the repository level there's nothing special about line breaks. Why can't we run git blame or cherry-pick on an AST, so that it operates regardless of formatting?
Are there tools that do this? (They don't need to be built into git; a third-party tool ought to be able to do this just fine.)
i think what you (and the OP) are describing goes beyond making merges more robust to nonfunctional changes by using something like an AST representation. making text edits apply cleanly is one part of the problem, but as the OP notes, the harder part of these merges is actually when interfaces stay the same but behavior changes. i think that to solve that, you'd need some kind of static analyzer that could jump from changed lines/syntax to execution paths and then to diffs of them.
i bet there are some cool papers out there on trying to do this... but, this is essentially a restatement of the halting problem, so building something that is guaranteed to be correct using only static analysis may be impossible. (but this does not preclude a solution that might be good enough)
On this topic, is there a JavaScript lint tool that integrates with Git to enable you to gradually apply style changes to code as it evolves? E.g. if the change is tabs-to-spaces the lint tool would require spaces in new or changed lines of code, but not complain about tabs in unchanged lines of existing code.
You can usually take the output of the lint tool and filter it by the lines modified by the patch (which you get from the diff).
Few tools do it out of the box (in the python world flake8 supports diff filtering out of the box, and there's diff-cover which can filter analysis beyond just test coverage) but it's easy enough to do with a wrapper, especially if the linter has a configurable or even parseable (e.g. json) output option.
This still leads to frustrating behavior, where changing one line of a file might end up reformatting all of it (and this often obscures the actual change itself).
I can't remember if this package specifically was one of the ones we tried (it was 3-4 different packages in the JS ecosystem), but anything that pretends files and not changes are the units of staging will cause problems for advanced git workflows.
At one job I pushed back against a drive to introduce a pre-commit "format to standard" hook. Aside from the fact that nobody was really advocating for it other than one programmer who had, shall we say, a certain outlook, I have never wanted to see a commit that had both the relevant change and formatting fixes. While the "reformat lines that have changed" kind of works, you can end up with code that is half one style and half another, which is worse for comprehension than having it all one style, even if that style is a bad one. This wasn't Python, so at least we didn't have to worry that an automatic reformat could literally change the behavior of an entire block. There are still edge cases in most languages, around control structures, i.e. the infamous braceless "if".
I think the implementers of the Go language got it right: there's one style, for everyone. Like it or hate it, there's no Holy War over it.
I guess one way this might work is the lint tool could run against the whole file but then cross-check the line numbers of any errors found with the git diff. If the diff shows the line is unchanged then ignore the error. That doesn't sound too hard.
On IntelliJ, if you use the Save Actions plugin, you can set it to only reformat lines that were changed. Although I'm not sure if would work for using Prettier
On one hand, I agree that these are all good points. I haven’t ever played with kernel code, but I have tried to backport years-old Glibc commits and it’s an absolutely miserable experience because “minor” things like small-scale reformatting would frequently be bunched together with unrelated changes. They also maintained their configure script in version control and apparently even merged it manually, because I’ve found that at many points in its history it would be composed of pieces generated by different versions of autoconf.
On the other hand, I don’t think that the conclusions of this story are as inevitable as it makes them seem. Much of the pain it describes is due to the fact that Git is, as it proudly itself, stupid: it doesn’t understand that your spacing or line breaking or bracketing changes are incidental to what the code is trying to accomplish. I’m not saying that structural or otherwise language-aware editing or source control is the silver bullet because I’m well aware that so far most attempts at it suck, but I think it’s important to remember that whitespace pedantry is to a large extent a tooling issue.
Could a merge tool coupled less than completely to clang-format (or yapf, or black, or gofmt ...) use its settings to improve the merges, I wonder?
(Tangentially related: Monticello, Paredit, the “skeleton syntax tree” idea from Dylan.)
Yeah absolutely. My personal measure for a the "cohesion-prowess" of a society is how much they screw around with plain text, opaque binary data, etc. vs take the time to properly create rich data structures for interfaces.
Taking the time and deep breath to write nice interfaces rather than scrambling and putting all the effort into implementations ignoring the larger picture of how human endeavors compose seems very fundamental. It reminds me also of the slogan
> A developed country is not a place where the poor have cars. It's where the rich use public transportation.
I’m not sure it’s quite as clear-cut, for three reasons:
- Text (or binary) streams are the ultimate unnormalized representation, and you don’t always want your data to be normalized. When storing it, maybe (though I’m sure the DBAs among us will disagree even on this point), but the intermediate values during processing, almost certainly not.
- Most representations of structured values are all or nothing, whereas in reality most types of structured data are usually more conveniently viewed as having a hierarchy of supertypes, possibly with things like “chunk of text” on the top, maybe “list of chunks of text” below that, and so on; otherwise your tools are much less reusable. (Compare “generic programming” in the Haskell/Scala not Java/C# sense, i.e. metaprogramming over algebraic datatypes not parametric polymorphism. See also the “nanopass” approach to compilers.)
These two (largely overlapping) points are how I explain to myself the (empirically if not theoretically evident) advantage of Unix’s “getta byte” approach to IPC (as Cutler sarcastically put it) over the myriads of structured approaches that came before and after. By not structuring your data exchange, it doesn’t force you into a framework that’s more rigid than it needs to be. People barely understand how to make structured data amenable to processing by composable tools now, back in the 80s I don’t think anybody even knew to ask the question.
This is why I mentioned Paredit and Dylan’s “skeleton syntax trees”: they are, respectively, a structural editor and a macro system that work where others fail by the virtue of being expressed in terms of more than text or even tokens, but less than a full syntax tree.
Finally, in this specific case,
- We’re talking about programming languages, about making a computer understand the programmer’s intent to some degree. This is a maze where making even the slightest wrong turn has Gödel and Turing together slapping you across the face with a giant NOPE sign. It’s kind of endearing to see the luminaries on the Algol committee (essentially all of CS at the time) state that they want, like, pseudocode, from their papers, except, y’know, with a compiler, and not suspect anything can go wrong with that. But we are hardly smarter than them, so we should probably remember the bitter lessons they’ve been forced to learn and keep in mind the possibility that there might not be a good solution to our problem.
> and you don’t always want your data to be normalized.
I think "normalized" is doing a lot of work in this sentence. In the context of data keeping, normalization usually doesn't mean something like "like give a unit vector", where they both have value and the normalization is lossy, but rather cleanups of messy stuff that ought not to occurred in the first place.
What I am saying is not the original input has no value, but we should take the steps to make those problems actually not happen in the first place. Rich interfaces do that, and are an investment that pays for itself eventually so this isn't manifestly unreasonable.
> ...metaprogramming over algebraic datatypes not parametric polymorphism...
I worked on a little toy "tree editor" where we did just that. The use of a good type system to both operate "generically" without "upcasting" and loosing the structure for ever is key.
> See also the “nanopass” approach to compilers.
I do like that a lot :) but I also like that there is an IR for every pass, like with CakeML. I thought those went hand in hand?
> (empirically if not theoretically evident) advantage of Unix’s “getta byte” approach to IPC
I would argue the (over-)prevelance of HTTP is effectively empirically the opposite, namely that people do prefer structured things in a vacuum, but coordination failures have repeated gotten us stuck in local maxima of already widely established things like Unix's various binary streams and then HTTP.
> This is why I mentioned Paredit and Dylan’s “skeleton syntax trees”: they are, respectively, a structural editor and a macro system that work where others fail by the virtue of being expressed in terms of more than text or even tokens, but less than a full syntax tree.
I do agree the jump from text to sexprs / json / token trees (Rust's name) etc. is the most important step. Fist get rid of flatness, then worry about higher invariants.
> We’re talking about programming languages, about making a computer understand the programmer’s intent to some degree.
And surely text doesn't help with that. Too much structure makes some edits hard if the user can't temporarily mid-edit ignore the rules, but computer should love it.
> It’s kind of endearing to see the luminaries on the Algol committee (essentially all of CS at the time) state that they want, like, pseudocode, from their papers, except, y’know, with a compiler, and not suspect anything can go wrong with that.
This is about Algol 1968 not Algol 1960? My understanding was that was too many features, not too many invariants.
Next generation of version control will diff the AST of the parsed code, so line endings don’t show up in the diff ;) Bonus, developers will be able to each see the code with their own code style or open a Java file in Scala ;)
I’d love for that to be true, but it’s not like people haven’t had that idea before (if maybe not with source control, because it’s become ubiquitous only relatively recently). Pushing code beyond a textual representation has been slooow and I haven’t yet seen a breakthrough large enough to change that.
The Scala thing isn’t happening, though, — it’s a difference of style and approach, not syntax.
Agreed. I’m pretty sure I have it solved with Tree Notation though. I’ve got a thing in the works which I call 3D git. Tree languages get it for free. The trick was simplifying the syntax so there’s nothing left but space, and then your syntactic shape matches the semantic shape.
> At a previous company, there was an “infamous” commit in our main repository. The commit was about 10 years old, and it replaced every tab with 4 spaces.
If have commits like this, add the ids to a file `ignorerevs`, and then tell git about it:
You can specify a file with of such commits with --ignore-revs-file, and of course check in that file. Naming that file .git-blame-ignore-revs seems to be a convention I've seen more than once.
Still need to specify the file yourself, so hopefully someone can point out the missing magic to have git pull in .git-blame-ignore-revs on its own.
ah, fixing a bug, ok. I was trying to think of two hypothetical correctly functioning programs identical but for whitespace, and thinking that was probably pretty rare.
I don't see any sane reason you'd intentionally access the first element only of a list by using a for loop with a non-conditional return. But anyway okeydoke. I do wonder if there are any realistic non-bugfix refactors/enhancements that would result in a whitespace-only change, or in general how often in a real codebase you are going to get a whitespace-only commit diff. But yeah, I understand it is possible, cool.
> 10 years ago would probably long enough ago to not care for me.
Hm, especially for the case of blaming (mostly to figure out why something was done a certain way) I frequently happen upon 10+ year old commits. This happens both in open-source projects, as well as at work (where we don't use git, but the same concepts apply).
Depending on the size of the team and how much agreement you can get on the importance of such a change, I think the better way to do this on an older repository is to get everyone to nuke their checkouts and use git-filter-branch to rewrite the history so that nobody ever used tabs.
Wow, this is incredible! I convinced my team to adopt Prettier for many of our repositories. The only real criticism was that our git history would be cluttered since it would reformat all of our code. This seems like the perfect solution.
When our code was well formatted and reviewed by professionals to show intent, I do not want a robot tampering with it for no good reason, especially if it blindly makes big diffs out of small changes.
Who gives a shit about a single commit that converts tabs-to-spaces? How is this a problem in any way?
The real problem here is that “git blame” is a garbage tool. Perforce has “timelapse view” that is radically better than any historical view I’ve seen in Git/Hg.
With respect to this issue, git just needs to suck less.
You answered your own question. Git blame is a pretty crappy tool but it's a tool that a lot of people use nevertheless. If you have a commit that converts tabs to spaces, then the blame for pretty much every single line will be lost before that point.
That says to list all the annoying commits in a file, then use a new-ish (Git version 2.23) Git feature to ignore all those commits when doing git blame.
Notice that one of the comments in that article says how to get everyone to use that ignore file, when using GitLens with VSCode.
It is a shame that we don’t have better tools, and that we are still hand-editing text files in order to write programs.
Imagine having an editor which automatically applied your local preferences around tabs/spaces, code formatting, variables up top vs nearest use, function definitions nested to minimize top-scope surface area vs all functions flat at top scope, etc etc etc
And when you are done editing, all of these local changes are reversed and you submit the minimal possible diff.
(and if we want to really talk pipe dreams, the dev only sees an AST editor and the underlying text is never even exposed in the first place)
Our tools are so far behind mostly because everyone’s thinking is still chained to 1970’s hand editing text files mentality. This is the flying car which isn’t being worked on because everyone is still thinking about making better bicycles.
Agreed. It seems silly that we still have to deal with things like tabs vs spaces or formatting in different ways. This should be handled by IDEs and editors.
The benefit is minimal, the task is hard, there's decades worth of tools that won't work, and it's easy to mess up introducing "impossible bugs" with behind-the-scenes transformations.
Also - AST isn't THAT important when reading the code. Let's say I give you this:
X (X X X X; X X X; X X) {
X (X X X) {
X(X X X X X)
}
}
Do you know what this code does? How about this:
for int i = 0 i < 10 i ++ if i % 2 printf " %d " , i
I've used several graphical languages professionally (not AST-based, graph-based, but the problem remains) and the main problem was - structure wasn't fully describing what happens - the "meat" of the behaviour was still in text form and was hidden behind the pretty graphic form - in case of both of these languages the meat was in the names of the subprocesses called and in the mapping of process variables <-> subprocess parameters.
And there were A LOT of these, so you couldn't show them at once on the same screen as the graphical view of the process. So programming with both of these languages was very frustrating - you had to click on each node and look through long lists of x:y substitutions to track how parameters flow through the system.
Our tools are so far behind mostly because everyone’s
thinking is still chained to 1970’s hand editing text
files mentality.
People IN THE LITERAL 1970s were talking about this exact idea. We still don't have this, for anything other than highly-specialized applications (for instance, equation editors in word processors).
It's almost like it's way, way, way harder to deliver software than it is to handwave about how much better things would be if people just listened to YOU.
One thing people never realize when they talk about this kind of technology is the amount of time code spends in a non-compiling state. Think about what your dream tool does what you cannot parse the code into an AST because you're in the middle of an hours-long refactor. That's a huge reason why it's so useful having the lowest-level representation as text.
Another problem is that virtually all of these efforts completely misunderstand what makes programming hard.
This isn't good general advice. The vast majority of projects aren't going to need to cherry pick commits to such old branches, and overly limiting style fixes and refactorings is a good way to ossify a code base.
> The commit was about 10 years old, and it replaced every tab with 4 spaces.
Reminds me of a project I worked on a while back - whenever you looked at the git history of a file, if you went back far enough you’d hit a dead end at the point where every source file was renamed from *.c to *.cpp.
Not really. Git tracks content and not files. File renames doesn't exists in git. When git tell you that a file is renamed it's because an algorithm guesses that a file is renamed (if the content is similar).
Exactly, so any rename should be done without other edits in the same commit, so the content hashes exactly the same. Any bulk rename in a repo would satisfy that and be a non-issue.
Why is having an obviously-incompilable intermediate state in your history a good idea, rather than using a diff algorithm that will find the move anyway?
Not sure I follow. So long as git understands the move so a single file history (using the standard command or standard IDE tooling) doesn’t “stop” before the rename, then I’m happy. And I’m very happy to accept one commit in the history that doesn’t build in order to get that.
What holds true for the Linux kernel may not hold true for other repos.
In this case, the OP is supporting old Linux releases and remarking how naturally cherry picking can work. LLVM by comparison, doesn't have long-term stable releases. So this conservative approach doesn't necessarily apply for LLVM.
So I like the analysis but I disagree with the generality.
> So I like the analysis but I disagree with the generality.
Any organization that releases software revisions has this issue. Yes, we do live in a world where much software is now subject to "continuous release" (either applications which are auto-updated to latest, on phones and some laptop OSes or web-only applications where the latest is pushed to deployment). And indeed the specific issue described here is much less important.
However I suspect the majority of software in use is not like that. On the Enterprise side, OSes and applications are updated carefully; an ERP upgrade can be a major undertaking. All sorts of machine control software is ether never upgraded, with limited patching, or treated the same way as other enterprise software. Etc.
The difference between the two is why there is both "git flow" and "GitHub flow" -- it's not like one is absolutely superior to the other.
has always served me better than blame.
It can better jump file boundaries and find e.g. prior code that the code in question was copied from.
I've always thought though that we need better conflict resolution that is code aware. And also better changeset specification, e.g. if a commit is equivalent to a s/foo/bar/ then this information should be included in the commit, or if a function signature has changed, then record that fact, rather than the dumb line by line diffs.
It sounds like the real advice here is “have a strong coding style guide that is rigorously enforced so nobody ever has to go make style fixes to existing code”.
What was the point the author tried to get across with the first paragraph bringing up the large whitespace commit? It didn't add anything to the story, it just ended up as an initial distraction.
Also, is he implying that thousands of commits (one for each file) would be a better way to convert that large code base? Why not solve this problem with the right tools, such as ignorerevs?
The whitespace commit breaks cherrypicking. What seemed like a harmless change could cripple automated backports of fixes. I thought it was the entire crux of his story.
Yes but the whitespace commit - a change that unavoidably affects almost every single line of its target - is possibly the worst example one can come up with.
I believe his point is that the commit shouldn't have happened at all, keeping the old style instead (or maybe applying the new style together with other changes ... Yuck. Don't do this)
It kinda serves as a hook to describe why these changes are bad. I think it's a good example for the rest of the article.
I feel it's a very contrived example because there's no compromise to how such a change is to be done. If there's an argument to make it's about making the decision to switch whitespace tactic at all.
It reminds me that one of my grievances with whitespace significant languages (hello Python) is that I can't effectively just skip commits that are pure whitespace changes in code review. I am curious how people approach this ...
123 comments
[ 2.6 ms ] story [ 227 ms ] threadHopefully the next item in this RCS CVS Subversion Git chain is just storing ASTs and transforms on top of them so we can spend less time fixing basic conflicts and discussing formatting.
Never looked at it myself, so not an endorsement. Just came up in the Lobsters discussion[2] on this last week where someone mentioned it.
[1]: https://www.unisonweb.org/docs/tour/
[2]: https://lobste.rs/s/b9pddy/when_it_comes_git_history_less_is...
The "problem" is no one actually wants to resolve merges that way.
If you mean Git had issues finding some specific code motion to show in a diff, you can try one of the other diff algorithms, or adjust the threshold for rename/copy detection. AST-based differs would also suffer this issue; a "nice diff" is not a formal problem and does not have a universal solution.
If you mean you once had a mis-merge that dropped some code you didn't want to drop, this won't go away with AST-based diffs. It will just happen at the token level instead of the line level.
If you mean it's generally hard to deal with collapsing lots of branches with shared history, that's true but would also be true with AST-based approaches. This is the situation something like pijul could help with, but also raises all the other tradeoffs of snapshot vs changeset based approaches.
Citation needed
First, it's too weak - you need to at least recognize special top-level defun-style forms, or you'll generate some minimal diff between two totally different functions just because they both use the same cond pattern or whatever.
Second, reader macros mean you can't really work on the source AST in the first place, unless you also teach the diff tool all your reader macros.
[1]: https://pijul.com/manual/why_pijul.html#comparisons-with-oth...
https://arxiv.org/pdf/1703.01192.pdf
Your suggestion (what comes after git) is throwing out the baby with the bathwater.
git blame, for instance, has the "--ignore-revs" and "--ignore-revs-file" options which let you specify some commits that can be ignored. This is a little helpful for the sort of code spelunking the author does (which I do very frequently), but it's a manual process. There's no built-in convention to Git for what this file should be named and how to have git blame pick it up automatically. Moreover, you have to create the commit in order to know what its revision is, which means that in squash/rebase workflows, you have to land the style cleanup on the target branch and only then can you add it to the ignore list.
One option that occurs to me is a special marker string to put in the commit message like "Git-Ignore-For-Blame: yes" or something. I'm not sure if there's a better way to do it.
More generically, if the commit is only swapping out tabs and spaces, "git blame -w" ought to take care of it. For more involved reformatting, git diff and git merge both have "ignore-all-space" modes, which will handle things like line breaks.
But really what you want for blames and cherry-picks, I think, is blames at the syntax level. The Git commands only know about lines - git blame runs on lines, git cherry-pick constructs diffs based on lines and tries to re-apply them, etc. But at the repository level there's nothing special about line breaks. Why can't we run git blame or cherry-pick on an AST, so that it operates regardless of formatting?
Are there tools that do this? (They don't need to be built into git; a third-party tool ought to be able to do this just fine.)
i bet there are some cool papers out there on trying to do this... but, this is essentially a restatement of the halting problem, so building something that is guaranteed to be correct using only static analysis may be impossible. (but this does not preclude a solution that might be good enough)
Few tools do it out of the box (in the python world flake8 supports diff filtering out of the box, and there's diff-cover which can filter analysis beyond just test coverage) but it's easy enough to do with a wrapper, especially if the linter has a configurable or even parseable (e.g. json) output option.
I think the implementers of the Go language got it right: there's one style, for everyone. Like it or hate it, there's no Holy War over it.
On the other hand, I don’t think that the conclusions of this story are as inevitable as it makes them seem. Much of the pain it describes is due to the fact that Git is, as it proudly itself, stupid: it doesn’t understand that your spacing or line breaking or bracketing changes are incidental to what the code is trying to accomplish. I’m not saying that structural or otherwise language-aware editing or source control is the silver bullet because I’m well aware that so far most attempts at it suck, but I think it’s important to remember that whitespace pedantry is to a large extent a tooling issue.
Could a merge tool coupled less than completely to clang-format (or yapf, or black, or gofmt ...) use its settings to improve the merges, I wonder?
(Tangentially related: Monticello, Paredit, the “skeleton syntax tree” idea from Dylan.)
Taking the time and deep breath to write nice interfaces rather than scrambling and putting all the effort into implementations ignoring the larger picture of how human endeavors compose seems very fundamental. It reminds me also of the slogan
> A developed country is not a place where the poor have cars. It's where the rich use public transportation.
This is brilliant, stealing it for my transit advocacy.
- Text (or binary) streams are the ultimate unnormalized representation, and you don’t always want your data to be normalized. When storing it, maybe (though I’m sure the DBAs among us will disagree even on this point), but the intermediate values during processing, almost certainly not.
- Most representations of structured values are all or nothing, whereas in reality most types of structured data are usually more conveniently viewed as having a hierarchy of supertypes, possibly with things like “chunk of text” on the top, maybe “list of chunks of text” below that, and so on; otherwise your tools are much less reusable. (Compare “generic programming” in the Haskell/Scala not Java/C# sense, i.e. metaprogramming over algebraic datatypes not parametric polymorphism. See also the “nanopass” approach to compilers.)
These two (largely overlapping) points are how I explain to myself the (empirically if not theoretically evident) advantage of Unix’s “getta byte” approach to IPC (as Cutler sarcastically put it) over the myriads of structured approaches that came before and after. By not structuring your data exchange, it doesn’t force you into a framework that’s more rigid than it needs to be. People barely understand how to make structured data amenable to processing by composable tools now, back in the 80s I don’t think anybody even knew to ask the question.
This is why I mentioned Paredit and Dylan’s “skeleton syntax trees”: they are, respectively, a structural editor and a macro system that work where others fail by the virtue of being expressed in terms of more than text or even tokens, but less than a full syntax tree.
Finally, in this specific case,
- We’re talking about programming languages, about making a computer understand the programmer’s intent to some degree. This is a maze where making even the slightest wrong turn has Gödel and Turing together slapping you across the face with a giant NOPE sign. It’s kind of endearing to see the luminaries on the Algol committee (essentially all of CS at the time) state that they want, like, pseudocode, from their papers, except, y’know, with a compiler, and not suspect anything can go wrong with that. But we are hardly smarter than them, so we should probably remember the bitter lessons they’ve been forced to learn and keep in mind the possibility that there might not be a good solution to our problem.
I think "normalized" is doing a lot of work in this sentence. In the context of data keeping, normalization usually doesn't mean something like "like give a unit vector", where they both have value and the normalization is lossy, but rather cleanups of messy stuff that ought not to occurred in the first place.
What I am saying is not the original input has no value, but we should take the steps to make those problems actually not happen in the first place. Rich interfaces do that, and are an investment that pays for itself eventually so this isn't manifestly unreasonable.
> ...metaprogramming over algebraic datatypes not parametric polymorphism...
I worked on a little toy "tree editor" where we did just that. The use of a good type system to both operate "generically" without "upcasting" and loosing the structure for ever is key.
> See also the “nanopass” approach to compilers.
I do like that a lot :) but I also like that there is an IR for every pass, like with CakeML. I thought those went hand in hand?
> (empirically if not theoretically evident) advantage of Unix’s “getta byte” approach to IPC
I would argue the (over-)prevelance of HTTP is effectively empirically the opposite, namely that people do prefer structured things in a vacuum, but coordination failures have repeated gotten us stuck in local maxima of already widely established things like Unix's various binary streams and then HTTP.
> This is why I mentioned Paredit and Dylan’s “skeleton syntax trees”: they are, respectively, a structural editor and a macro system that work where others fail by the virtue of being expressed in terms of more than text or even tokens, but less than a full syntax tree.
I do agree the jump from text to sexprs / json / token trees (Rust's name) etc. is the most important step. Fist get rid of flatness, then worry about higher invariants.
> We’re talking about programming languages, about making a computer understand the programmer’s intent to some degree.
And surely text doesn't help with that. Too much structure makes some edits hard if the user can't temporarily mid-edit ignore the rules, but computer should love it.
> It’s kind of endearing to see the luminaries on the Algol committee (essentially all of CS at the time) state that they want, like, pseudocode, from their papers, except, y’know, with a compiler, and not suspect anything can go wrong with that.
This is about Algol 1968 not Algol 1960? My understanding was that was too many features, not too many invariants.
The Scala thing isn’t happening, though, — it’s a difference of style and approach, not syntax.
Agreed. I’m pretty sure I have it solved with Tree Notation though. I’ve got a thing in the works which I call 3D git. Tree languages get it for free. The trick was simplifying the syntax so there’s nothing left but space, and then your syntactic shape matches the semantic shape.
A sneak peak:
https://twitter.com/breckyunits/status/1408546408911695873?s...
If have commits like this, add the ids to a file `ignorerevs`, and then tell git about it:
Then at least `git blame` will still give useful results. (This is a relatively new git feature, added a year or two ago.)Still need to specify the file yourself, so hopefully someone can point out the missing magic to have git pull in .git-blame-ignore-revs on its own.
[0] https://git-scm.com/docs/git-config#Documentation/git-config...
banana_giraffe was asking for a way to set it up automatically so users don't have to do anything.
But wouldn’t the standard -w be enough to ignore most of this commit.
But I used ignoreRevs in repositories that messed up their history by using an automatic indenter with atrocious settings.
Hm, especially for the case of blaming (mostly to figure out why something was done a certain way) I frequently happen upon 10+ year old commits. This happens both in open-source projects, as well as at work (where we don't use git, but the same concepts apply).
https://stackoverflow.com/questions/58042532/how-can-i-clang...
The real problem here is that “git blame” is a garbage tool. Perforce has “timelapse view” that is radically better than any historical view I’ve seen in Git/Hg.
With respect to this issue, git just needs to suck less.
My company has a web tool for browsing the repo that allows this. It’s still inferior to Perforce timelapse view. But it’s not a hard blocker.
That says to list all the annoying commits in a file, then use a new-ish (Git version 2.23) Git feature to ignore all those commits when doing git blame.
Notice that one of the comments in that article says how to get everyone to use that ignore file, when using GitLens with VSCode.
Imagine having an editor which automatically applied your local preferences around tabs/spaces, code formatting, variables up top vs nearest use, function definitions nested to minimize top-scope surface area vs all functions flat at top scope, etc etc etc
And when you are done editing, all of these local changes are reversed and you submit the minimal possible diff.
(and if we want to really talk pipe dreams, the dev only sees an AST editor and the underlying text is never even exposed in the first place)
Our tools are so far behind mostly because everyone’s thinking is still chained to 1970’s hand editing text files mentality. This is the flying car which isn’t being worked on because everyone is still thinking about making better bicycles.
Also, cities where one can get around on bicyles are much nicer than where one needs to use cars. Flying or otherwise. https://www.youtube.com/watch?v=ul_xzyCDT98
To summarize both points: what looks like progress superficially may actually not be progress at all.
Also - AST isn't THAT important when reading the code. Let's say I give you this:
Do you know what this code does? How about this: I've used several graphical languages professionally (not AST-based, graph-based, but the problem remains) and the main problem was - structure wasn't fully describing what happens - the "meat" of the behaviour was still in text form and was hidden behind the pretty graphic form - in case of both of these languages the meat was in the names of the subprocesses called and in the mapping of process variables <-> subprocess parameters.And there were A LOT of these, so you couldn't show them at once on the same screen as the graphical view of the process. So programming with both of these languages was very frustrating - you had to click on each node and look through long lists of x:y substitutions to track how parameters flow through the system.
It's almost like it's way, way, way harder to deliver software than it is to handwave about how much better things would be if people just listened to YOU.
Not really the tone I had intended, but it looks like that's the tone which came across.
To be more explicit: "It's a shame there's no money in next-gen tooling"
Another problem is that virtually all of these efforts completely misunderstand what makes programming hard.
To me the solution is to migrate to trunk based development and hide new features behind feature flags.
The biggest issue with this transformation seems to be implementing feature flags in the code.
Reminds me of a project I worked on a while back - whenever you looked at the git history of a file, if you went back far enough you’d hit a dead end at the point where every source file was renamed from *.c to *.cpp.
We have a few files at work that change name with every commit…
In this case, the OP is supporting old Linux releases and remarking how naturally cherry picking can work. LLVM by comparison, doesn't have long-term stable releases. So this conservative approach doesn't necessarily apply for LLVM.
So I like the analysis but I disagree with the generality.
Any organization that releases software revisions has this issue. Yes, we do live in a world where much software is now subject to "continuous release" (either applications which are auto-updated to latest, on phones and some laptop OSes or web-only applications where the latest is pushed to deployment). And indeed the specific issue described here is much less important.
However I suspect the majority of software in use is not like that. On the Enterprise side, OSes and applications are updated carefully; an ERP upgrade can be a major undertaking. All sorts of machine control software is ether never upgraded, with limited patching, or treated the same way as other enterprise software. Etc.
The difference between the two is why there is both "git flow" and "GitHub flow" -- it's not like one is absolutely superior to the other.
Then there last paragraph would tell them he’ll ignore replies that don’t mention plaid.
has always served me better than blame. It can better jump file boundaries and find e.g. prior code that the code in question was copied from.
I've always thought though that we need better conflict resolution that is code aware. And also better changeset specification, e.g. if a commit is equivalent to a s/foo/bar/ then this information should be included in the commit, or if a function signature has changed, then record that fact, rather than the dumb line by line diffs.
Someone just has to build it.
Also, is he implying that thousands of commits (one for each file) would be a better way to convert that large code base? Why not solve this problem with the right tools, such as ignorerevs?
It kinda serves as a hook to describe why these changes are bad. I think it's a good example for the rest of the article.