YAML anchors may be a sharp tool but no one is forced to use them. I have written many verbose Github workflows that would have benefited from using anchors, and I am relieved to learn I can clean those up now.
I think YAML anchors in GitHub Actions are very welcome, for example for DRYing the push/pull_request 'paths:' filters [1].
Now only if they supported paths filter for `workflow_call` [2] event in addition to push/pull_request and my life would be a lot easier. Nontrivial repos have an unfortunate habit of building some sort of broken version of change detection themselves.
The limit of 20 unique workflow calls is quite low too but searching the docs for a source maybe they have removed it? It used to say
> You can call a maximum of 20 unique reusable workflows from a single workflow file.
but now it's max of 4 nested workflows without loops, which gives a lot of power for the more complex repos [3]. Ooh. Need to go test this.
Anchors will be exceptionally useful for a few workflows for me. We have what is essentially the same setup/teardown for three jobs within one workflow. I’d love to be able to factor that stuff out without introducing yet another yaml file to the repo, this will be a big help.
Just because it is expressed in YAML doesn't make YAML the party to blame here. I would say one of the main concerns I have with anything in GitHub Actions related to the word "merge" has to do with identifying the last commit for a merge, not merging of objects in YAML.
If you have two workflows... one to handle a PR creation/update and another to address the merge operation, it is like pulling teeth to get the final commit properly identified so you can grab any uploaded artifacts from the PR workflow.
Author's replacement for anchors is to use global syntax, like a top-level "env:" block.
This is a terrible advice from security endpoint - given that env variables are often used for secrets data, you really _don't_ want them to set them at the top level. The secrets should be scoped as narrow as possible!
For example, if you have a few jobs, and some of them need to download some data in first step (which needs a secret), then your choices are (a) copy-paste "env" block 3 times in each step, (b) use the new YAML anchor and (c) set secret at top-level scope. It is pretty clear to me that (c) is the worst idea, security wise - this will make secret available to every step in the workflow, making it much easier for malware to exfiltrate.
Can YAML go away entirely and instead allow pipelines to be defined with an actual language? What benefits does the runner-interpreted yaml-defined pipeline paradigm actually achieve? Especially with runners that can't be executed and tested locally, working with them is a nightmare.
A JSON-like language with functions is the answer here. When it comes to describing large, complex, and sometimes repetitive data, having a declarative language with proper tools for abstraction helps so much with readability, writability, and maintainability.
I've seen few thousands-line YAML files with anchors riddled all over the place. It was impossible to deal with. Rewriting it in Jsonnet paid off immediately.
Another example is Nixpkgs. It's quite pleasant to deal with despite the size of its codebase.
Obviously they are very useful. I still don't think they should exist in this usage of yaml.
Once you allow setting and reading of variables in a configuration file, you lose the safety that makes the format useful. You might as well be using a bash script at that point.
Honestly, everything about GH actions/AzDO pipelines is infuriating. The poor tooling with poor write-time assertions are just so frustrating.
Give me a proper platform that I can run locally on my development machine.
but, if those anchors are a blessed standard YAML feature that YAML tools will provide real assertions about unlike the ${{}} stuff that basically you're doing a commit-push-run-wait-without any proper debug tools besides prints?
I disagree. Instead of anchors we had to rely on third party file change actions to only trigger jobs on certain file path changes, instead of using the built in mechanism, because each job required the list, and the list was long.
Using anchors would have improved the security of this, as well as the maintenance. The examples cited don't remotely demonstrate the cases where anchors would have been useful in GA.
I agree that YAML is a poor choice of format regardless but still, anchor support would have benefitted a number of projects ages ago.
Ok, now make the 'redundancy' argument with anything other than `env` or `permissions`.
I think they should be supported because it's surprising and confusing if you start saying 'actually, it's a proprietary subset of YAML', no more reason needed than that.
A very pedantic point, but merge keys are not part of the YAML spec [1]! Merge keys are a custom type [2], which may optionally be applied during the construction phase of loading. I definitely wouldn't say that merge keys are integral to anchors.
(Also, as a personal bias, merge keys are really bad because they are ambiguous, and I haven't implemented them in my C++ yaml library (yaml-cpp) because of that.)
First, he can just not use the feature, not advocate for its removal.
Second, his example alternative is wrong: it would set variables for all steps, not just those 2, he didn't think of a scenario where there are 3 steps and you need to have common envs in just 2 of them.
Wrote a new yaml grepping tool this past weekend and just realized thanks to this that I have a whole new can of worms to keep in mind. Ugh.
Turns out it does report values at their targets (which is desirable) but doesn't know or indicate that they're anchors (undesirable).
Also tested something with yq - if you tell it to edit a node that is actually from a yaml anchor, it updates the original anchor without warning you that that's what you're doing. Yikes.
Anchors are so, so useful. Buildkite (which has its own CI pipelines syntax) is a good example. Let’s say I want every pipeline step to run on my custom agents (on my self-hosted infra). I could either copy/paste an identical “agents” property across however many hundreds or thousands of CI steps I have.
I don't think anchors' primary function is to allow global definitions (of variables or whatever), rather it's more like arbitrary templates/snippets to be reused through the YAML file.
In GitLab, where YAML anchors have been supported for years, I personally find them very useful —it's the only way of "code" reuse, really. In GitLab there's a special edtor just for .gitlab-ci.yml, which shows the original view and the combined read-only view (with all anchors expanded).
I agree that it's hard to point to the specific line of the source code, but it's enough — in case of an error — to output an action name, action property name, and actual property value that caused an error. Based on these three things, a developer can easily find the correct line.
Common Lisp has this (and other dialects that imitate the Circle Notation; I think Scheme has it now, Emacs Lisp, TXR lisp, ):
E.g.
(#1=(a b) c d e #1#)
encodes
((a b) c d e (a b))
where the two (a b) occurrences are one object. It can express circular structures:
#1=(a b c . #1#)
encodes an infinite circular list
(a b c a b c a b c ...)
The object to be duplicated is prefixed with #<decimal-integer>=. This associates the object with the integer. The integer is later referenced as #<decimal-integer># to replicate it.
The thing is, you don't see a lot of this in human-written files, whether they are source code or data.
This is not the primary way that Lisp systems use for specifying replicated data in configurations, let alone code.
Substructure sharing occurs whether you use the notation or not due to interned symbols. (Plus compilers can deduplicate strings and such.) In (a a a) there is only one object a, a symbol.
If you feed the implementation circular source code though, ANSI CL says the behavior is undefined. Some interpreters can handle it under the right circumstances. In particular ones that don't try to do a full macro-expanding code walk before running the code. Compilers, not so much.
28 comments
[ 3.0 ms ] story [ 54.0 ms ] threadOPs main argument seems to be "I don't have a use for it and find it hard to read so it should be removed".
One day we might even see for-loop in CSS...
Now only if they supported paths filter for `workflow_call` [2] event in addition to push/pull_request and my life would be a lot easier. Nontrivial repos have an unfortunate habit of building some sort of broken version of change detection themselves.
The limit of 20 unique workflow calls is quite low too but searching the docs for a source maybe they have removed it? It used to say
> You can call a maximum of 20 unique reusable workflows from a single workflow file.
but now it's max of 4 nested workflows without loops, which gives a lot of power for the more complex repos [3]. Ooh. Need to go test this.
[1] https://docs.github.com/en/actions/reference/workflows-and-a...
[2] https://docs.github.com/en/actions/reference/workflows-and-a...
[3] https://docs.github.com/en/actions/how-tos/reuse-automations...
If you have two workflows... one to handle a PR creation/update and another to address the merge operation, it is like pulling teeth to get the final commit properly identified so you can grab any uploaded artifacts from the PR workflow.
This is a terrible advice from security endpoint - given that env variables are often used for secrets data, you really _don't_ want them to set them at the top level. The secrets should be scoped as narrow as possible!
For example, if you have a few jobs, and some of them need to download some data in first step (which needs a secret), then your choices are (a) copy-paste "env" block 3 times in each step, (b) use the new YAML anchor and (c) set secret at top-level scope. It is pretty clear to me that (c) is the worst idea, security wise - this will make secret available to every step in the workflow, making it much easier for malware to exfiltrate.
I've seen few thousands-line YAML files with anchors riddled all over the place. It was impossible to deal with. Rewriting it in Jsonnet paid off immediately.
Another example is Nixpkgs. It's quite pleasant to deal with despite the size of its codebase.
Once you allow setting and reading of variables in a configuration file, you lose the safety that makes the format useful. You might as well be using a bash script at that point.
Give me a proper platform that I can run locally on my development machine.
but, if those anchors are a blessed standard YAML feature that YAML tools will provide real assertions about unlike the ${{}} stuff that basically you're doing a commit-push-run-wait-without any proper debug tools besides prints?
Then yes, they should use them.
Custom YAML anchors with custom support and surprise corner cases: bad.
- The complaint is Github using a non-standard, custom fork of yaml
- This makes it harder to develop linters/security tools (as those have to explicitly deal with all features available)
- The author of this blogpost is also the author of zizmor, the most well-known Github Actions security linter (also the only one I'm aware of)
Using anchors would have improved the security of this, as well as the maintenance. The examples cited don't remotely demonstrate the cases where anchors would have been useful in GA.
I agree that YAML is a poor choice of format regardless but still, anchor support would have benefitted a number of projects ages ago.
I think they should be supported because it's surprising and confusing if you start saying 'actually, it's a proprietary subset of YAML', no more reason needed than that.
(Also, as a personal bias, merge keys are really bad because they are ambiguous, and I haven't implemented them in my C++ yaml library (yaml-cpp) because of that.)
[1]: https://yaml.org/spec/1.2.2/
[2]: https://yaml.org/type/merge.html
First, he can just not use the feature, not advocate for its removal.
Second, his example alternative is wrong: it would set variables for all steps, not just those 2, he didn't think of a scenario where there are 3 steps and you need to have common envs in just 2 of them.
Turns out it does report values at their targets (which is desirable) but doesn't know or indicate that they're anchors (undesirable).
Also tested something with yq - if you tell it to edit a node that is actually from a yaml anchor, it updates the original anchor without warning you that that's what you're doing. Yikes.
(For anyone who wants to test it: https://pypi.org/project/yamlgrep/)
Or I could use a YAML anchor.
In GitLab, where YAML anchors have been supported for years, I personally find them very useful —it's the only way of "code" reuse, really. In GitLab there's a special edtor just for .gitlab-ci.yml, which shows the original view and the combined read-only view (with all anchors expanded).
I agree that it's hard to point to the specific line of the source code, but it's enough — in case of an error — to output an action name, action property name, and actual property value that caused an error. Based on these three things, a developer can easily find the correct line.
E.g.
encodes where the two (a b) occurrences are one object. It can express circular structures: encodes an infinite circular list The object to be duplicated is prefixed with #<decimal-integer>=. This associates the object with the integer. The integer is later referenced as #<decimal-integer># to replicate it.The thing is, you don't see a lot of this in human-written files, whether they are source code or data.
This is not the primary way that Lisp systems use for specifying replicated data in configurations, let alone code.
Substructure sharing occurs whether you use the notation or not due to interned symbols. (Plus compilers can deduplicate strings and such.) In (a a a) there is only one object a, a symbol.
If you feed the implementation circular source code though, ANSI CL says the behavior is undefined. Some interpreters can handle it under the right circumstances. In particular ones that don't try to do a full macro-expanding code walk before running the code. Compilers, not so much.