It still works if you turn off SIP. This is the same on Apple Silicon and Intel. However, for these purpose of tracking file accesses, I recommend using `eslogger` instead, as it doesn’t require disabling SIP and is faster, among other advantages.
It’ll show all file events. No need to disable SIP. SIP is doing a lot of good work for users and unless you’re doing kernel work or low level coding I’d keep it enabled. Obv. There are other cases but for the general public keep it on.
It’s personal preference. If the security people had their way we’d all be developing on iPads. If SIP interferes with your work: turn it off. Linux doesn’t have SIP and it’s just fine to develop on Linux as well.
100% agree. We would put everyone on chromebooks if we had our way. I don’t think it’s good for productivity and generating new ideas for a company so I never advocate for it.
There was a man who never existed named Thomas Covenant, created by Stephen R Donaldson. Decades ago this character said something that has stayed and will stay with me my entire life:
"The best way to hurt somebody who's lost everything is to give him back something that is broken."
For us, this thing is MacOS. I miss dtrace every damn day.
DTrace allowed you to ask the damn os what it was doing, since the man pages are random and do not match the command line help, new daemons constantly appear with docs like this:
NAME
rapportd – Rapport Daemon.
SYNOPSIS
Daemon that enables Phone Call Handoff and other communication features between Apple devices.
Use '/usr/libexec/rapportd -V' to get the version.
Users should be aware that `sudo apt install sysdig` may require reconfiguration of UEFI Secure Boot, and that there is no apparent clean abort from this possible. The raw-mode terminal screen contains the text:
┌────────────────────────┤ Configuring Secure Boot ├────────────────────────┐
│ │
│ Your system has UEFI Secure Boot enabled. │
│ │
│ UEFI Secure Boot requires additional configuration to work with │
│ third-party drivers. │
│ │
│ The system will assist you in configuring UEFI Secure Boot. To permit │
│ the use of third-party drivers, a new Machine-Owner Key (MOK) has been │
│ generated. This key now needs to be enrolled in your system's firmware. │
│ │
│ To ensure that this change is being made by you as an authorized user, │
│ and not by an attacker, you must choose a password now and then confirm │
│ the change after reboot using the same password, in both the "Enroll │
│ MOK" and "Change Secure Boot state" menus that will be presented to you │
│ when this system reboots. │
│ │
│ If you proceed but do not confirm the password upon reboot, Ubuntu will │
│ still be able to boot on your system but any hardware that requires │
│ third-party drivers to work correctly may not be usable. │
│ │
│ <Ok> │
│ │
└───────────────────────────────────────────────────────────────────────────┘
This leads to a second screen asking for a new password. There is a <Cancel> option, but selecting it merely loops back to the first screen.
Hitting C-c (Control-c) has no effect.
There are validity rules for the password, but they are presented only after you've entered an invalid password, twice, the second time to confirm.
After this installation proceeds, then terminates with a warning:
DKMS: install completed.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
W: Operation was interrupted before it could finish
I suspect that's just because you're trying to install an old version (as I wrote sysdig used to have a kernel extension but now should use eBPF functionality provided in stock kernels). I can't easily verify (no ubuntu at hand), but presumably if you install the vendor supplied, up-to-date version (first google hit I found: https://www.linuxcapable.com/how-to-install-sysdig-on-ubuntu...) it will work without UEFI changes.
On Windows, Process Monitor does the same thing --- with the additional feature that, since configuration is often stored in the registry there, it will also show which registry keys it touches.
I do this much more frequently than I would like. But occasionally you have issues where the program lists a directory then only acts on specific names. So the strace output won't tell you what the expected name is.
I use quick and dirty greps all the time, even when there is a "better" option available. It just works and is very intuitive in interactive contexts. Probably GP works in a similar way.
In lieu of strace, IDK how fast `ls -al /proc/${pid}/fd/*` can be polled and logged in search of which config file it is; almost like `lsof -n`:
pid=$$ # this $SHELL's pid
help set; # help help
(set -B; cat /proc/"${pid}"/{cmdline,environ}) \
| tr '\0' $'\n'
(while 1; do echo \#$pid; ls -al /proc/${pid}/fd/*; done) | tee "file_handle_to_file_path.${pid}.log"
# Ctrl-C
It's good Configuration Management practice to wrap managed config with a preamble/postamble indicating at least that some code somewhere wrote that and may overwrite whatever a user might manually change it to (prior to the configuration management system copying over or re-templating the config file content, thus overwriting any manual modifications)
## < managed by config_mgmt_system_xyz (in path.to.module.callable) >
## </ managed>
I don't think I've ever encountered a widely used program that doesn't have that information in its manpage. The manpage is the canonical source the article calls for.
With the exception of New Age software like Go, that for some reason doesn't use manpages, I'm not convinced that the problem implied here actually exists.
Doesn't the man page often list several potential locations where it looks for the file and the ordering behavior, leaving you to investigate and find which is actually loaded? It's still annoying.
But that is exactly the information you need in order to determine which config file is being read in all situations. A single location doesn't cut it if the reality is more complicated.
If the goal is to find out which particular file is being used at this particular time, on this particular system, the maximum verbosity log might have that information, and otherwise strace can help, as described in the other comments.
I really do not care about all the possible paths a config file might be read from. I want a flag that I can pass to a binary to see which paths it will actually read, and ideally, I'd like those paths to be logged to STDOUT on startup.
Yes, I want to know which files in which order.cI don't want a static list of all possible locations, I want it to deduce what files would take effect in a given environment and with the given config files - a program must already know this when it is ran - so let's just be open and print said paths.
Absolutely. For example, if you pass "-v" to ssh, it prints paths to configs that it tried to open, paths to identity files that it tried to open, to known_hosts files, etc. This info is not needed so very often, but when it is, it's incredibly useful.
Yes. Exactly. Which is why the program should tell you which one takes precedence.
The program already knows what config files it's loading and in what order, so why is it left up to the human to do all that work again? There should be a flag like --manifest or something that would spit out what actually got loaded from where.
Out of interest, I checked out man man on Linux and found the search logic documented in man 5 manpath. In addition to system directories specified in /etc/manpath.config, the man command checks directories in the user's PATH and maps them to related man directories: pathdir, pathdir/../man, pathdir/man, pathdir/../share/man, and pathdir/share/man are all considered. The MANPATH environment variable overrides all other configuration and should not be necessary.
So, following current practices, installing binaries in ~/.local/bin and man pages in ~/.local/share/man should work without any additional configuration.
> Ideally I'd like to avoid scanning all the way through the manual page or other documentation for the program to find out, because that's slow and annoying.
It's indeed "configured using symlinks" but not in the sense that you're free to use symlinks the way they're supposed to work but in the sense that if you don't want to use systemctl to place them, you can place them "by hand" if you know all the magic ways in which systemd interprets them and will remove them as it sees fit when you invoke a management command.
I have no idea what you mean by that. A systemd target is a folder in /etc/systemd/system/ which contains a bunch of symlinks to unit config files. When the target is requested, systemd makes sure that all these units are running. This is how symlinks have been used for configurations since forever (for example by apache on Debian).
ls: cannot access '/etc/systemd/system/mydaemon.service': No such file or directory
It has no business removing that symlink. It was placed there by me, and it should be treated no differently than if it was a regular file.
I have no problem with systemctl enable/disable creating and removing links in the appropriate target.d directories, but this symlink had nothing to do with that mechanism. Leave my stuff alone.
This doesn't work because unit files have to be in one of the load paths. Have you tried adding /path/to/ to $SYSTEMD_UNIT_PATH? This is documented in the Unit File Load Path section of the manpage [1].
> This is how symlinks have been used for configurations since forever (for example by apache on Debian).
This is not how symlinks have been used for configuration "since forever." You didn't have to add the link target to some "load path" variable when the link itself was in the correct directory.
A system-wide key-value store, that's distinct from file storage, seems like a pretty great idea. The Windows Registry is icky, we all know that, it somehow appeared out of nowhere in Windows 95 already crusty as hell. But if it were designed by people with taste and discretion, would it be good?
When you give people the freedom to use the KV store as they wish, you inevitably run into a meta-problem whereby the way they write their data structures in to the KV store is varied across applications. I don't know if you could standardise this structure beyond enforcing applications to use an OS-assigned private store, but I don't see how that would work with portable executables etc.
I suppose you'd need to be very opinionated for it to work, but people will still bodge it to make it work with their needs that your system doesn't natively satisfy (people putting JSON strings in the values, for example).
The filesystem is manageable by users (with few exceptions). You can't easily discover which part of a registry hive is associated with an application.
It's precisely as discoverable as the filesystem. It's a tree like the filesystem. With names given to nodes by the application, just like the filesystem.
The title read without context sounds like it's suggesting they should report which IP address/country/jurisdiction they're in. To any random user on the internet. Which would be insane. But it's really only saying "which system path they're in, to e.g. a sys admin".
I am familiar with how software is configured, and the implied context is not clear from only the headline, hence the headline needs fixing. "Everything" is massive overstatement: bank websites use configuration files, so if hackers try to log in to the bank, the website should report where the config files are located? Nope. Headline way too broad.
Configuration files are like parents’ advice - carefully thought out, well-intentioned, but often overwritten by the ‘environment variables’ of real life experiences!
I have this tool, which is used in a few different environments, so I gave it a “xxx config” command which tells which config files were read, which environment variables were used, which environment variables from the tool’s XXX_ namespace are unknown, what configuration values are effective etc.
Programs reconfigurable at runtime are by far rarity, as that's generally pretty hard thing to do. But yes, just have a config file, ENVs are nice as a backup for small apps, but never as only way
It's definitely nice when it does but it's also a whole lot of complexity to add to the code. You have to re-create all structures relying on configs, reconnect if needed, clear all previous ones, but only after any ongoing processing of requests received before the reload signal finishes.
Some go half-way, like HAProxy where it does spawn a new process but that process just binds with SO_REUSEPORT and signal old one to close the socket and finish all remaining connections. So you effectively get hitless reload, it's just that old process is the one finishing in-flight requests
Yeah, it def adds complexity but if you want it used in real world applications, it’s a must-have. With only env vars, it’s an impossibility to ever have (without supporting infrastructure and processes), which was the point I was trying to make.
Hardly. "Real world applications" need to tolerate single node failure anyway so reload functionality doesn't add all that much. Some system daemons, sure, them not working for a second can be disruptive so it is worthy endeavor there, but the amount of added complexity is nontrivial and the added complexity scales with apps' own complexity.
It's a bit different for desktop apps, as restarting app every time you change some config options would be far bigger PITA than some server app.
Unless you’re working for a tech company, it’s running on a single server (and it’s worth pointing out that most companies aren’t tech companies). That’s what I meant by “real world”, AKA, outside the tech bubble.
I've had rare cases of needing nested data structures in the environment. My solution was to specify them as JSON strings and load the value in code. It works.
Yes. Env vars beat constants but have stunted our thinking because we know deep down it's not a great idea to have too too many.
Config files are better, but can lead to the mess described here and typically only give us crude axes for overriding.
It's time for us to embrace & tame the complexity. If you use a feature flag tool already, you know what it's like to target different values based on rules.
If I want to experiment with changing the http timeout for specific calls from a single service in a single region I should be able to roll that out.
It's time to expand our brains and bring this feature flag style thinking to the rest of our configuration.
> If I want to experiment with changing the http timeout for specific calls from a single service in a single region I should be able to roll that out.
...why ? how often you do it ? What actual app feature it fills ? Why would you throw permanent code at something that you need to do once or twice?
That kind of thing is FAR better served as customizable hooks (loadable .so/.dll, maybe even attached Lua interpreter) rather than bazillion random flags for everything and checks cluttering code.
For anything more than trivial configurations it can be nice to commit them to source control, giving some reversion capabilities, and easy replication on other projects.
- If your configuration has more than 5-10 options then env vars become a mess while a configuration file is more maintainable
- Nested configuration / scoping is a mature advantage of configuration files
- You can reload configuration files whereas you can't reload environment variables during runtime
- A configuration file is a transparent record of your configuration and easier to backup and restore than env variables. Env variables can be loaded from many different locations depending on your os.
- In configuration files you can have typed values that are parsed automatically with env variables you need to parse them yourself. This is just a difference not that bad for env variables per se.
The main issue is to use configuration files residing somewhere in the filesystem. This looks like a global variable in a codebase (something we generally try to avoid).
Instead, the configuration file should be explicitly provided as a command line argument. Systemd sandboxing can also be useful to ensure the program only uses the expected set of files.
For instance, on my NixOS machine, the Nginx configuration is not in `/etc/nginx` but explicitly provided and can then be known with ps:
$ ps aux | grep nginx
nginx: master process /nix/store/9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0/bin/nginx -c /nix/store/9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf
It's not a bad idea but it's not applicable to every piece of software. I don't think that passing a config file for every git command would be convenient.
You can change the commandline string at runtime. You could inject a fake "-c correct/path" even if it's not there. (That's useful for other things too, like injecting the git revision of the app into the commandline)
> This looks like a global variable in a codebase (something we generally try to avoid).
Aren't they more like global constants than variables? Loaded at startup, and never change during that run of the program. (With the exception of only explicitly being re-read on SIGUSR1 for daemon-like programs.)
And global consts, or #defines, or whatever, are things we generally don't try to avoid?
$ redis-cli INFO
...
executable:/Users/antirez/hack/redis/src/./redis-server
config_file:/etc/redis/redis.conf
...
In the case of Redis, it is possible to do much more. With the CONFIG command you can check specific configuration parameters at runtime, and even change most of them.
There are commands to get reports about latency and memory usage, performing a self-analysis of the instance condition. You can see counters for commands called, and even obtian a live stream of the commands the instance is executing.
This should all be obvious features to make system software approachable, observable, and to make people able to act easily when shit happens, without having to guess where the configuration file is from the system startup scripts or alike. UX is not only about GUIs.
As a system administrator I want 2 things from config files:
* loading a directory of config files, so various Configuration Management tools have easier time deploying config (especially for multi-use apps like say syslog daemon, where multiple disparate apps might want to put a part of the config in)
* a function to dry-run config validation, so CM can alert about that instead of restarting an app to a wrong config
, so that custom configurations in that directory win over the distribution-provided defaults set up after this Include directive inside /etc/ssh/sshd_config.
Also: processes / running programs should provide a straightforward way to show their configuration.
While "sshd -T" shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the/a currently running sshd process.
This has footgun potential, as someone could think their sshd is secured and hardened by checking with "sshd -T", while the actually running sshd process still uses old configurations.
Apache also makes it needlessly hard to find out its effective (and active / currently running) configuration. I don't know of any better way then awkwardly using mod_info ( https://httpd.apache.org/docs/2.4/mod/mod_info.html ) to get such insights.
Not sure I get the reasoning here. That requires you to actually go and proofread the configuration of each and every running process, at which point it's easier to just verify the effective configuration on disk and restart the service.
SSH is a special case because it leaves active sessions alive when it restarts, but that doesn't generalize to other daemons. Just restart if in doubt.
It's a little bit away from the domain of config files, but I think CSS is the most graceful example of handling this as a generalised problem. Style configurations cascade, and the last one read is the one that gets applied. There are overrides to that flow, but you have to very overtly add a piece of syntax to the overriding line like `!important`. Two overlapping overrides revert back to the "last read" approach. The approach is so fundamental to CSS that the entire language is named after it, so there's no ambiguity or forwards compatibility concerns in this regard whatsoever.
> Specificity is the algorithm used by browsers to determine the CSS declaration that is the most relevant to an element, which in turn, determines the property value to apply to the element. The specificity algorithm calculates the weight of a CSS selector to determine which rule from competing CSS declarations gets applied to an element.
The rules of CSS specificity are a bit more involved than '!important' or not, and I frequently see beginners trip there, but apart from that I agree that the CSS rule set might be worth incorporating.
Fun trick to use if you use a css post processing (less, sass, etc.) is "&&&", it will resolve to a very specific rule which overwrites the other rules
I’m sure you know this, but for readers who don’t, it’s much more complex than “last defined wins with a few override options”. Based on “specificity”, rules based on the selector used.
Some examples:
1. #mydiv overrides .mydiv - IDs are more specific than classes
2. .outerdiv .innerdiv overrides .innerdiv - nested selectors are more specific
3. .mydiv overrides div - classes are more specific than elements
And a lot of other rules.
I can’t imagine I’d want this level of complexity in config files.
Rather specificity is a triplet of (ids sub-selectors, class sub-selectors, element sub-selectors). When attributes conflict, the one whose rule has the more specific selector wins.
The specificities of the rules you wrote out are (1,0,0), (0,1,0), (0,2,0), (0,1,0), (0,1,0), and (0,0,1).
In-line attributes (@style) beat out out-out-of line ones except if they’re !important. Specificity also disambiguates between !important attributes.
> I can’t imagine I’d want this level of complexity in config files.
Lots of people don’t want it in their CSS either. It’s generally considered good practice to make everything the same specificity and avoid !important.
Sure, haven't seen that in a while in well structured code bases.
> make everything the same specificity
How would you achieve same specificity on everything? Only use either ids or classes? No type selectors / pseudo-classes / pseudo-elements / multiple selectors? I can't imagine how that would be feasible and am not aware of anyone recommending this.
There are a large number of companies that require ids as the basis for every selector, by policy. This may not be interesting to you, but I think this is what they meant. The fact they may or may not be familiar to you is incidental.
I'm familiar with policies that generally strive for specificity of (0, 1, 0) like BEM or (1, 0, 0), but I'd be very interested in any policy that rules out pseudo-classes like `:hover`, as prohibiting them would leave you to handle any interactivity you'd get for free from CSS via JavaScript.
With :is and :where it is trivial to trick selectors
:is(selector_1, :where(selector_2))
Has always the specificity of selector_2, and if you choose an impossible selector for selector_1 (the easiest approach is to give multiple ids or use non existent classes/elements) it only select by selector_2
> Two overlapping overrides revert back to the "last read" approach.
IIRC, there's an important (heh) exception to that: user stylesheets. Without the "!important" override on the user stylesheet, the site stylesheet wins; with the "!important" override on the user stylesheet, the user stylesheet wins, even if the site stylesheet also had an "!important" for that rule.
The domain of style and configuration are very, very different. I'm not sure what works okay for displaying a website in different layouts on vastly different unknown devices share a lot of the same concerns of configuring programs on a machine
> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
> ...
And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.
systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules :
--
Understanding variable precedence
Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables):
1 command line values (for example, -u my_user, these are not variables)
2 role defaults (defined in role/defaults/main.yml) 1
3 inventory file or script group vars 2
4 inventory group_vars/all 3
5 playbook group_vars/all 3
6 inventory group_vars/* 3
7 playbook group_vars/* 3
8 inventory file or script host vars 2
9 inventory host_vars/* 3
10 playbook host_vars/* 3
11 host facts / cached set_facts 4
12 play vars
13 play vars_prompt
14 play vars_files
15 role vars (defined in role/vars/main.yml)
16 block vars (only for tasks in block)
17 task vars (only for the task)
18 include_vars
19 set_facts / registered vars
20 role (and include_role) params
21 include params
22 extra vars (for example, -e "user=my_user")(always win precedence)
Chef is worse. It's discouraged to use `override` in Chef cookbooks but nothing says that you can't, and introducing spooky action at a distance is just part of your life now.
IIRC, Ansible had this exact variable precedence long before they were acquired by RH.
I get that hating RH is in vogue currently, but let's keep it grounded in reality please.
EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
As an Ansible person, I couldn’t agree more. Being able to manage variables at basically any level of precedence is extremely useful.
I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
I love how there are 21 places to look if you're trying to figure out why a CLI parameter is being ignored. Does it at least print a warning or something?
According to the comment above, `cmd -u my_special_user` will only use the value "my_special_user" if none of the other 21 places have another configuration for the user, right?
".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server.
/usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.
/run is useful for configuration generation. Sure, you could maybe poke an API and have it do the right thing, but if "everything is a file" then why not put your configuration in a file? That way it's easier to see what's actually happening.
Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.
>Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.
>Read the manpage for all the hilarity.
If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr.
Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr.
Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy.
The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings.
BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.
>And this is just an excerpt which doesn't cover the details.
The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.
So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}.
If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).
We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.
There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.
You mean that the top-level Linux directory are hilariously badly named?
If so, I don't think anybody will disagree.
But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.
If would be nice to solve the issue, but it isn't anybody's priority.
I think there is a gap here between those who routinely (exclusively?) used UNIX machines along with 50, 100, 1000 other people all logged in at once (generally pre-Linux era) and those who only ever got to see and enjoy a time when everyone could have a UNIX workstation on their desktop (Linux, Mac, or other x86 flavour, take your pick).
Back in the shared era, the machines were better-suited to individual customization, and were more easily understood by system administrators. Now, they're optimized too... waste more computational resources when idle?
Yeah, I really feel like we're in a "horseless carriage" or "radio with pictures" era for computing. We have incredibly powerful machines available. But the dominant paradigm for managing that power is to slice it up into things that simulate much less powerful processors in software environments designed for a shared 1970s university department minicomputer.
In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.
I'm not sure I understand. When the "old paradigm" was new, computing power was much more limited. They couldn't afford to "burn" any computing power. But it's only stuck around, because now we have more computing power? That doesn't make sense
You might read some Kuhn. Paradigms change when the old one becomes unsupportable. But the vast increase in computing power per dollar means we can waste a lot of it without it being economically uncomfortable.
He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.
In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html
Isn’t that just a backronym? I read recently (here, I think) that on some original specific UNIX system, the user directories were there originally, and then other stuff that belonged elsewhere went in there because the other physical volume (where things like /bin were) ran out of space. Later another dish was added and profiles were moved there, to make /home.
If you want to use /usr for yourself, be prepared to throw the FHS and compability with almost everything away and rebuild your whole userland with something like NixOS/Guix.
> Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.
The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:
- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.
- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.
You don't have to be an expert to read multiple files. If you do need expert assistance with something like that please contact me and we can work something out.
Rebasing patches is pretty hard I'd say. Much harder than just explaining someone the logic behind /lib, /etc and /run. I mean, this directories are seperate for a reason. Knowing the idea behind the directory split (a.k.a. Linux Filesystem Hierarchy Standard) is more essential than knowing how to use the patch command line tool. There are more and more people who don't want to use the command line. And that's completely fine.
> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.
> If you find it hilarious it's because you don't understand the reason.
I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.
When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.
Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"
But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.
> it still imposes a large burden on the person trying to figure out "why is X doing Y?"
Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.
Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.
> Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.
But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.
"Make it as easy as possible, but not simpler". I think this applies perfectly here. Systemd has very complicated requirements due to its very unique position in a Linux distribution. And as a (power) end-user of it, you basically need you basically needs to touch only /etc
> Systemd has very complicated requirements due to its very unique position in a Linux distribution.
But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?
The combination of distributions existing and believing they “own” parts of the system, and users needing to override settings means it is as simple as possible, and no simpler.
A distribution 'owning' a directory means that the package manager is allowed to change files in there, and allowed to assume nothing major was changed from what it installed.
A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.
Systemd didn't create this problem. But it solves the issue of distributions shipping config files as part of packages and then on every package upgrade having to reconcile between the distribution's config and your modifications. Now with the drop in system you don't need to do that.
I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.
Users are only supposed to know about `/etc`. The rest of the directories are useful for distribution maintainers, people who package software for a distribution (but may not be maintainers), and some developers.
That's because some features are targeted at different audiences. End users want simplicity and ease if use. Admin also want those, but they are secondary to determinism, consistency, and ease of tracking changes.
I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.
Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.
In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.
That last bit is especially what interests me about systemd. From the way you describe it, it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users. But that they did so grudgingly and partially, again favoring their interests.
Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.
But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.
> it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users
I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).
The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.
When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.
Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).
If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).
If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).
Thanks for the details. I will check out the link.
I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.
It’s still hilarious. The fact that there’s a need for this hilarity doesn’t negate that. What a mess humanity have created… (Let’s write a suckless OS!)
Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user's needs.
Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.
Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.
However, I do not think there is any standard API to follow, I used "info cfg" but many alternatives were possible. So sysadmins have to explore to find out which is the introspection command to give. Not ideal, but I don't think there is a solution.
Another good place for that is the --version output, if your --version is one of those verbose multi-line ones (instead of a basic single-line output).
While at it I’ve made it a practice to expose an internal endpoint for every service to output its config. It has saved me on numerous occasions to catch config related bugs.
Could you explain a little more? I am not sure what exactly you mean by "exposing an internal endpoint" - do you mean that as a secured API endpoint which you can call? Isn't that hard to secure?
Depending on your environment, endpoints might not default to exposed. As an example where I work, we have to explicitly enumerate endpoints to expose (potentially including wildcards, like admin/*), and anything else is only accessible with direct access to the pod.
I have been leaning towards a different approach. Rather than reading multiple config sources at runtime, read them at build time, and emit a single config file with everything fully resolved. Put that on the filesystem, and have the app read it. You have an easy canonical reference for the configuration, with no need to expose anything. The lifecycle of the config doesn't have to match that off the app binary, so it could (and probably should) have a separate build and release process.
Agreed, logging in config retrieval code should log all this stuff, requests, where they are filled from, that sort of thing.
The problems with this approach when it's enabled is that log aggregators (e.g splunk) can end up including stuff like passwords which shouldn't be visible to the group able to view those logs, so some care is needed.
I'd rather have some config reserialization capability with provenance (perhaps as an opt-in). A good config reserialization can make itself useful in so many situations, if it has the right features. Like pointing out key-value pairs that don't have any effect, or simply writing out some documentation.
Logging has its merits, like discoverability when you stare at it without suspecting configuration to be the culprit, but I believe that there can and should be more.
I would prefer a convention… but first wins is backwards IMO. It really sucks when a bad convention gets adopted, like this first wins, or spaces instead of tabs. ;)
If there are multiple files in a wildcard pattern, what order are they resolved? I know for classpath loading, for example, it’s in directory order, which is file system dependent, and often based on the order the files were written to the directory and/or the size of the directory inode. Lots of fun figuring out why the wrong version of a class is consistently loaded on one or two servers out of twenty. (Always check your classpaths for duplicates kids.)
I've always seen this solved by sorting (lexicographical order of bytes, originally). First two characters were digits, for all files within a wildcard directory.
> If there are multiple files in a wildcard pattern, what order are they resolved
If you don't care (eg [0]) then do nothing and let the user bang it's head and reinvent the wheel.
If you care - just sort it and mark it as 'platform dependent' or somehow. Nobody really cares for the exact rules and a simple \d\d\-\w+\.conf pattern for the filenames is more than enough in 99% of cases.
Oh yes! Good thing if it only happens in the build server and not on deployments, but that can be "fun" enough. Bonus points if you have a culture of repeat builds being "special".
Because "first" and "last" don't convey priority. You have to know how the configuration parser works with defaults and overrides. Does the parser stop reading after the first match? Do later configuration directives override previous ones or are they appended to them, and if they are appended, in which order are they evaluated?
First and last are only ambiguous if we make the order ambiguous though. There's no reason for the order to be ambiguous, and if it is then obviously we can't use first/last. I don't think that first/last are either ambiguous or implementation dependent concepts. The other questions are simply other questions, and aren't directly related to whether the ordering is ambiguous or not.
There are plenty of applications that read a set list of configuration files and use either the first or last setting found. I agree with the responder that the last setting usually makes more sense.
First wins makes some sense from a security pov. You want to load the more "trusted" configuration files first because they must be able to further restrict the less trusted files, but don't want to let the less trusted files override the settings set by the trusted files.
Rejecting multiple definitions is better, unless you care about DoS (I would be pissed if I got locked out of ssh access due to an option being defined twice, for instance).
Amazon had a hierarchical configuration system for loading config that could override other config (or add to it, depending on the implementation). The Java framework would provide an endpoint to show you the final resolved config along with which file and line that value came from. Very helpful when trying to figure out why config was different than expected.
At least the inspector in Safari (and maybe other WebKit browsers) does something similar for CSS.
I’m all for it. It’s a pain when writing config systems (no longer just key/value, but key+value+meta), but very helpful. It can be a pain for things like JSON where libraries don’t give you that type of diagnostic information easily, however.
I had the distinct pleasure (?) of owning the C++, Python, and Java implementations of that config for a while and ... yeah, it had some positives, but the fact that it had two orthogonal hierarchies of configuration in addition to implementation-specific feature gaps and nuances made it something of a nightmare in practice.
Not relevant to tarsnap (and in general something I would be very careful about adding since people often don't realize what's in their environment) but I would say that command line options take priority over the environment and the environment takes priority over all configuration files.
Nice example, which opened a small rabbit hole for me: what does "first occurrence wins" mean for options that are meant to be used multiple times, such as "Port" in sshd_config. The man page only says "Multiple options of this type are permitted.", but now I wonder what happens when I put my own Port into /etc/ssh/sshd_config.d/whatever.conf but leave "Port 22" in sshd_config? Or is the (new since bookworm) default sshd_config commenting it out for this reason?
More generally, for multi-options do we maybe want "first file wins"?
Reading in general to more specific configuration directories in order and using the last value read makes the most sense to me. This is usually how I write my own applications.
Anyway, I thoroughly document the order and precedence, but the suggestion from Chris' blog to report more detail about the process is a good one.
Ant also had first writer wins. I tried to explain this so, so many times to so many people, and only rarely did it stick. On that project I spent way too much time fixing build errors, and ended up rearranging things to make the behavior more obvious rather than less, because without it people would fool themselves into thinking they had it right, because sometimes it would appear to work.
And the thing with developers is that they make lousy scientists. If they want something to work, they’ll take any information that supports that theory and run straight to the repository with it.
The trick is not to fool yourself, and you’re the easiest person to fool.
It's crazy that a year from now, no one is ever going to be thinking about this garbage ever again. By then, enough programmers will have discovered how insanely good GPT4 is at doing software configuration, and in the absence of the right answer, reading these verbose docs to finally just tell you what to do.
It would also help if an active process said if it was running using the current config.
For example, if a change has been introduced to a config file, but the process has not been restarted to incorporate those changes, this should be conveniently mentioned.
This is a big ridiculous, but the fact is that the entire order of processing there is a result of practical need evolved over likely thousands of organizations.
There are unit testing concerns, dev/test/stage/prod concerns, CLI args, OS vars, config files, IDE vs non-ide concerns, and more. And even some references to databases (if you squint and see JNDI as a database).
So yeah, finding out where property value X was set can be a bit maddening oftentimes.
"Configuration" is an afterthought in most languages and frameworks in initial iterations. So bespoke / one-off solutions will explode early on. As the original complaint details, "configuration" isn't just a simple property file. It gets complex pretty quickly, and a good configuration subsystem would be nice:
- supports most of the concerns above
- provides a means for logging the final config value set and WHERE IT SOURCED FROM
But... oh wait, config values aren't just key-value. Wait until you get to trees of values or other more complicated config. Then value overlaying gets pretty hard, and resolution code gets complicated too.
Think of infrastructure as code frameworks. The configuration is insanely deep and complicated. Many complicated systems have "configuration languages", and they are probably Turing Complete.
And once your configuration system is complicated enough ... your configuration system kind of implicitly needs ITS OWN configuration system.
So maybe the first config layer is actually a bootstrap for loading all the configuration info.
And.... then you have config values you need to secure. Hooo boy, now you have all the complexity of high security systems interfaces: authentication and authorization, decryption and hashing.
And invariably there are binary blobs encoded in text friendliness.
It gets crazy. Fast.
BUT THE AMOUNT OF SIMILARITY IS NUTS. NUTS. This is begging for a common approach or standard or library.
>>>Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach.
A bunch of applications at my job use a config-loading library that will fail on startup if the same option is set to two different value (on different lines). It's certainly frustrating at times...but it's also prevented a huge amount of ambiguity.
That would actually be my preference TBH. Failing quickly at startup is usually far better than unpredictable behaviour later on, particularly if it's a long-running service.
Though I've always thought that the behaviour of tools when launched interactively by the user, vs launched via an automated background process should probably be different - it may well be that if a user/administrator isn't aware it failed on machine startup the consequences of it not being running at all are worse than an ambiguous configuration setting.
There should be an option to report where any currently active non-default configuration settings actually came from, individually. (including if that's shell environment vars, etc).
I forget what tool I use regularly that actually does this. Is it git?
I maintained a CLI tool for a long time. I came to the conclusion that good config is a bit harder and more subtle than most people think. I agree with the premise of the article, but there are more layers to it, literally.
Depending on how much config you have, sooner rather than later you’ll have multiple sources: system defaults, user config files, sometimes profiles, env vars, and flags. Configuration may be sourced from a combination of these, so overrides need to be handled and naming needs to be consistent. Another source of mess is deeply nested config, such as for a user-defined project. Those are very awkward to add flags for, in which case it’s better to enforce a file (like kubernetes config).
To me, tools like git strikes a good balance. And by that I mean that project-local config is in a project root directory that’s not necessarily checked in to VCS, but picks up the config even if you’re in a subdir. It rarely causes any unwanted surprises, but at the cost of being somewhat flag heavy, which is then alleviated with user-defined aliases.
I'd add: everything should use configuration files that are stored on the machine. I know services that use configuration files and you are expected to upload them into the service using an IDE and a plugin to then store the config in it's database.
not supporting configuration files on disk allows a service to work without needing read or worse write access to the disk. storing configuration in the database is a benefit. (potentially even for security)
Lately I've written all my tools to not only announce where they're reading their configs from (which isn't precisely what OP is asking for but is close), but also to have an option to generate the default config that will be presumed if a config file is not found. "mytool --gen-config /tmp/newconfig.txt" should produce a fully-formed config with all defaults explicitly set which can be moved into /etc/mytool.conf or wherever it's supposed to go, all ready for me to tweak without having to look up settings names.
In my day job I do a lot of troubleshooting. That's most of my job... helping other IT folks figure out what the heck is happening with something.
I'd say that in 70% of things I work on, part of that time is figuring out what config files are used because the app/service/process/whatever is doing something the owner/tech/whatever doesn't expect, yet is rational, and they "didn't tell it to do that".
It'd be great if everything reported as the article mentions -- amazing, in fact -- but I feel like it's a pie in the sky wish. Like asking that all programs always exit cleanly or something. I feel like there'll always be inevitable edge cases -- libraries that themselves pick up config files or registry values or environment variables that aren't expected, who knows -- that'll need to be discovered. If things are already not going right with a program, I'm less likely to trust what it says it's doing and more likely to just watch it and see what it's actually reading and doing.
Yesterday I was wondering why the self signed root certificate and the certificates I added to my debian install didn't work with httpie (ssl warning) but did work with curl (and the browsers I added the RootCA to).
I found the explanation: https://github.com/httpie/httpie/issues/480 httpie doesn't look into /usr/local/share/ca-certificates but apparently it's not httpie's fault, it's the python requests library that doesn't check that folder.
I don't know what to think of it and who is supposed to fix it. But I am back to curl with my tail between the legs (because I oversold httpie to my coworkers) for now. There's only one place where the bucket stops with curl.
I read the thread after seeing your comment, and it strikes me that besides the first comment being a little brusque, sigmavirus24 comes across as both calm and helpful - doubly so given that the original issue appears to have been PEBKAC related
> I think I forgot to run "pip3 uninstall" and ran only "pip uninstall" as I had to use python3 to get working ssl in the first place.
The issue is still open, so it has nothing to do with PEBKAC. "A little brusque" is definitely underselling it. That was a fine display of internet assholery.
For once I would say it isn't Python's fault either. There are at least 10 different locations that certificates can be stored at on various Unix variants and Linux distros. Go checks them all, which is probably the only sane solution, but seriously come on Unix devs... Is it that hard to pick a standard location?
My favourite is a very bad pattern common in Docker images. Just configure it with env vars, and then a script plucks them into a config file somewhere.
And then document it misleadingly badly. These options are mandatory! (lists 5 of the 10 actual mandatory options).
Then a script branches on a env var value, which isn't one that's documented, and then fails opaquely if it took the wrong branch because you didn't know you needed to set that env var.
Best ones are the ones that consume your env vars and set _other_ env vars, it's great!
1) Document all the config options, and ensure you highlight any interactions between settings, and explain them. I'm biased because I used to work on this project at RH, but I really did like the documentation for Strimzi, here's how their env vars are documented [0].
To highlight a few examples of how I think they're good:
> STRIMZI_ZOOKEEPER_ADMIN_SESSION_TIMEOUT_MS
Optional, default 10000 ms. The session timeout for the Cluster Operator’s ZooKeeper admin client, in milliseconds. Increase the value if ZooKeeper requests from the Cluster Operator are regularly failing due to timeout issues.
We know if it's required, what it defaults, and what I love to see, why we might want to use it.
> STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
...
This [provided prop] is used when a KafkaMirrorMaker.spec.version property is specified but not the KafkaMirrorMaker.spec.image
I like this, explaining when this env var will or will not be used.
> STRIMZI_IMAGE_PULL_POLICY
Optional. The ImagePullPolicy that is applied to containers in all pods managed by the Cluster Operator. The valid values are Always, IfNotPresent, and Never. If not specified, the Kubernetes defaults are used. Changing the policy will result in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters.
Firstly, always great to enumerate all accepted values for enum-like props. But what I really like here is that the consequences of altering this value are explained.
> STRIMZI_LEADER_ELECTION_IDENTITY
Required when leader election is enabled. Configures the identity of a given Cluster Operator instance used during the leader election. The identity must be unique for each operator instance. You can use the downward API to configure it to the name of the pod where the Cluster Operator is deployed. (Code snippet omitted)
Interactions between config options highlighted - if you set STRIMZI_LEADER_ELECTION_ENABLED to true, this option is now required.
We're told that this must be unique. And a suggestion on one straightforward way to do this, with example code.
One more thing to call out as good:
> The environment variables are specified for the container image of the Cluster Operator in its Deployment configuration file. (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml)
Being told _where_ in the source code the env var is being consumed is great.
Now compare the Confluent docs for their Kafka Connect image. [1] A colleague of mine was using this image, and was connecting to Confluent Cloud, so needs to use an API key and secret. There's no mention of doing that at all. But you can. Just merely set the CONNECT_SASL_JAAS_CONFIG option, and make sure you also set
CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM and CONNECT_SASL_MECHANISM.
And very importantly, don't forget CONNECT_SECURITY_PROTOCOL, as not only will you have odd connection failures (usually manifests as the client dying at the ApiVersions negotiation stage of the Kafka protocol's handshake), but because a script run in the image uses the value of that undocumented setting to execute a prerequisite in different ways [2], and you'll get weird behaviour that obscures the real issue.
2) Support more than one way of configuring things - maybe I want to mount in a config file, maybe I want to provide a ConfigMap, maybe I want to do it via env vars. Well... [3]
If any of the frequent offenders are open source, I wonder if ot would be worth your while to submit (or get someone else to) a change to add this functionality.
I think the best way to achieve this is by providing an OS API that results in the files always being created in the same place. Applications/libraries could still choose their own filenames and syntax, just the location would be OS controlled. I think there is room for a new desktop/laptop OS to emerge and one good idea from mobile OS design I would like to see is having everything be an API call that allows the OS to impose standardisation, permissions and user preferences rather than the free-for-all desktop OSes have (though I propose letting the user run non-compliant applications, and not porting the iOS app store only model into the OS).
The problem with the Windows registry, at least back in the day, was that it was a single file that could be corrupted and it would wreck your whole system.
I think having a standard utility API for *nix configs makes a lot of sense. I'm surprised it doesn't exist.
I tried to find one, and there are some libraries for reading and parsing in every language, but nothing that seems to cover everything.
> The problem with the Windows registry, at least back in the day, was that it was a single file that could be corrupted and it would wreck your whole system.
Exactly. But Registry has been using NTFS and it's capabilities for rollbacks and recovery. Therefore, the problem is mostly solved. There are occasions [0] of bugs causing corruptions though but they are very rare.
I just implemented a "--config" command in my latest code that reports exactly what was used for configuration, files and environment variables, and where the files are actually located.
486 comments
[ 3.1 ms ] story [ 273 ms ] threadI had to do this to figure out where Cold Steel was reading save files from years back: https://vaughanhilts.me/blog/2018/02/16/playing-trails-of-co...
https://8thlight.com/insights/dtrace-even-better-than-strace...
Feel free to reference my feedback entry FB12061147 if you're reporting the same.
It’ll show all file events. No need to disable SIP. SIP is doing a lot of good work for users and unless you’re doing kernel work or low level coding I’d keep it enabled. Obv. There are other cases but for the general public keep it on.
https://mohit.io/blog/fs_usage-trace-file-system-calls-on-ma...
"The best way to hurt somebody who's lost everything is to give him back something that is broken."
For us, this thing is MacOS. I miss dtrace every damn day.
DTrace allowed you to ask the damn os what it was doing, since the man pages are random and do not match the command line help, new daemons constantly appear with docs like this:
NAME rapportd – Rapport Daemon.
SYNOPSIS Daemon that enables Phone Call Handoff and other communication features between Apple devices.
Dogshit.So while this can be a useful hack, it doesn't always work.
Though DTrace is "only" available on Windows, Mac OS X, Solaris, illumos, and FreeBSD.
Oracle has relicensed DTrace to be Linux-friendly and even made kernel patches, but it'll probably never end up in the mainline kernel.
Is it stable and production ready for heavy duty workloads?
[1]It used to have its own kernel extension but is eEBF based these days.
Hitting C-c (Control-c) has no effect.
There are validity rules for the password, but they are presented only after you've entered an invalid password, twice, the second time to confirm.
After this installation proceeds, then terminates with a warning:
Where the "W:" above is colored red.Distro was Ubuntu 20.04.
These days Proton makes a lot of this unnecessary. I hope Trails games will continue to at least function on Linux. :)
https://www.brendangregg.com/blog/2014-07-25/opensnoop-for-l...
https://github.com/brendangregg/perf-tools/blob/master/opens...
Strace: https://en.wikipedia.org/wiki/Strace
In lieu of strace, IDK how fast `ls -al /proc/${pid}/fd/*` can be polled and logged in search of which config file it is; almost like `lsof -n`:
It's good Configuration Management practice to wrap managed config with a preamble/postamble indicating at least that some code somewhere wrote that and may overwrite whatever a user might manually change it to (prior to the configuration management system copying over or re-templating the config file content, thus overwriting any manual modifications)With the exception of New Age software like Go, that for some reason doesn't use manpages, I'm not convinced that the problem implied here actually exists.
If the goal is to find out which particular file is being used at this particular time, on this particular system, the maximum verbosity log might have that information, and otherwise strace can help, as described in the other comments.
The program already knows what config files it's loading and in what order, so why is it left up to the human to do all that work again? There should be a flag like --manifest or something that would spit out what actually got loaded from where.
Primarily because man pages need to go in the MANPATH to be useful but these tools tend to dwell somewhere in ~/.
So, following current practices, installing binaries in ~/.local/bin and man pages in ~/.local/share/man should work without any additional configuration.
> Ideally I'd like to avoid scanning all the way through the manual page or other documentation for the program to find out, because that's slow and annoying.
systemctl daemon-reload
systemctl disable mydaemon.service
ls /etc/systemd/system/mydaemon.service
ls: cannot access '/etc/systemd/system/mydaemon.service': No such file or directory
It has no business removing that symlink. It was placed there by me, and it should be treated no differently than if it was a regular file.
I have no problem with systemctl enable/disable creating and removing links in the appropriate target.d directories, but this symlink had nothing to do with that mechanism. Leave my stuff alone.
[1] https://www.freedesktop.org/software/systemd/man/systemd.uni...
> This is how symlinks have been used for configurations since forever (for example by apache on Debian).
This is not how symlinks have been used for configuration "since forever." You didn't have to add the link target to some "load path" variable when the link itself was in the correct directory.
P.S. Debian's a2ensite and a2dissite also assume that your configs are in the sites-available folder, not elsewhere in the file system.
I think it's a reasonable complaint.
I suppose you'd need to be very opinionated for it to work, but people will still bodge it to make it work with their needs that your system doesn't natively satisfy (people putting JSON strings in the values, for example).
I don’t think anyone ever used that function.
Some go half-way, like HAProxy where it does spawn a new process but that process just binds with SO_REUSEPORT and signal old one to close the socket and finish all remaining connections. So you effectively get hitless reload, it's just that old process is the one finishing in-flight requests
It's a bit different for desktop apps, as restarting app every time you change some config options would be far bigger PITA than some server app.
I'm thinking of things like config for nginx, Apache, Kubernetes, etc.
Config files are better, but can lead to the mess described here and typically only give us crude axes for overriding.
It's time for us to embrace & tame the complexity. If you use a feature flag tool already, you know what it's like to target different values based on rules.
If I want to experiment with changing the http timeout for specific calls from a single service in a single region I should be able to roll that out.
It's time to expand our brains and bring this feature flag style thinking to the rest of our configuration.
...why ? how often you do it ? What actual app feature it fills ? Why would you throw permanent code at something that you need to do once or twice?
That kind of thing is FAR better served as customizable hooks (loadable .so/.dll, maybe even attached Lua interpreter) rather than bazillion random flags for everything and checks cluttering code.
httpClient.get("my/url", timeout: config.get("http.timeout"))
Where config is a library smart enough to do some rule evaluation over a live updated backing store to get the right value.
I'm not sure how .so/dll or lua is going to help here.
like if you need LLVM and didnt manage to find it under
"app_folder/llvm"
then let's go find it somewhere on the disk(s) and ask user if he wants to use it
For instance, on my NixOS machine, the Nginx configuration is not in `/etc/nginx` but explicitly provided and can then be known with ps:
$ ps aux | grep nginx
nginx: master process /nix/store/9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0/bin/nginx -c /nix/store/9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf
Aren't they more like global constants than variables? Loaded at startup, and never change during that run of the program. (With the exception of only explicitly being re-read on SIGUSR1 for daemon-like programs.)
And global consts, or #defines, or whatever, are things we generally don't try to avoid?
There are commands to get reports about latency and memory usage, performing a self-analysis of the instance condition. You can see counters for commands called, and even obtian a live stream of the commands the instance is executing.
This should all be obvious features to make system software approachable, observable, and to make people able to act easily when shit happens, without having to guess where the configuration file is from the system startup scripts or alike. UX is not only about GUIs.
* loading a directory of config files, so various Configuration Management tools have easier time deploying config (especially for multi-use apps like say syslog daemon, where multiple disparate apps might want to put a part of the config in)
* a function to dry-run config validation, so CM can alert about that instead of restarting an app to a wrong config
Finding where is one has never been a problem.
Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach.
Buried in https://man.freebsd.org/cgi/man.cgi?sshd_config(5) and awkwardly expressed:
"Unless noted otherwise, for each keyword, the first obtained value will be used."
That's why e.g. Debian's /etc/ssh/sshd_config ( https://salsa.debian.org/ssh-team/openssh/-/blob/debian/1%25... ) starts with
, so that custom configurations in that directory win over the distribution-provided defaults set up after this Include directive inside /etc/ssh/sshd_config.While "sshd -T" shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the/a currently running sshd process.
This has footgun potential, as someone could think their sshd is secured and hardened by checking with "sshd -T", while the actually running sshd process still uses old configurations.
Apache also makes it needlessly hard to find out its effective (and active / currently running) configuration. I don't know of any better way then awkwardly using mod_info ( https://httpd.apache.org/docs/2.4/mod/mod_info.html ) to get such insights.
SSH is a special case because it leaves active sessions alive when it restarts, but that doesn't generalize to other daemons. Just restart if in doubt.
https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity...
Some examples:
1. #mydiv overrides .mydiv - IDs are more specific than classes
2. .outerdiv .innerdiv overrides .innerdiv - nested selectors are more specific
3. .mydiv overrides div - classes are more specific than elements
And a lot of other rules.
I can’t imagine I’d want this level of complexity in config files.
Rather specificity is a triplet of (ids sub-selectors, class sub-selectors, element sub-selectors). When attributes conflict, the one whose rule has the more specific selector wins.
The specificities of the rules you wrote out are (1,0,0), (0,1,0), (0,2,0), (0,1,0), (0,1,0), and (0,0,1).
In-line attributes (@style) beat out out-out-of line ones except if they’re !important. Specificity also disambiguates between !important attributes.
Lots of people don’t want it in their CSS either. It’s generally considered good practice to make everything the same specificity and avoid !important.
Sure, haven't seen that in a while in well structured code bases.
> make everything the same specificity
How would you achieve same specificity on everything? Only use either ids or classes? No type selectors / pseudo-classes / pseudo-elements / multiple selectors? I can't imagine how that would be feasible and am not aware of anyone recommending this.
:is(selector_1, :where(selector_2))
Has always the specificity of selector_2, and if you choose an impossible selector for selector_1 (the easiest approach is to give multiple ids or use non existent classes/elements) it only select by selector_2
Specificity is at the heart of CSS, you're not supposed to design rules with equal specificity
IIRC, there's an important (heh) exception to that: user stylesheets. Without the "!important" override on the user stylesheet, the site stylesheet wins; with the "!important" override on the user stylesheet, the user stylesheet wins, even if the site stylesheet also had an "!important" for that rule.
Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.
E.g. from systemd.unit manpage ( https://www.freedesktop.org/software/systemd/man/systemd.uni... ):
>...
> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
> ...
And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.
that is a RedHat-isim
I get that hating RH is in vogue currently, but let's keep it grounded in reality please.
EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server.
/usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.
Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.
>Read the manpage for all the hilarity.
If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr.
Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr.
Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy.
The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings.
BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.
>And this is just an excerpt which doesn't cover the details.
The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.
So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).
We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.
There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.
https://access.redhat.com/documentation/en-us/red_hat_enterp...
There's also a section for /usr.
• Filesystem Hierarchy Standard <https://www.pathname.com/fhs/>
• For Debian (and derivatives): <https://www.debian.org/doc/debian-policy/ch-opersys.html#fil...>
<https://manpages.debian.org/stable/manpages/hier.7.en.html>
If so, I don't think anybody will disagree.
But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.
If would be nice to solve the issue, but it isn't anybody's priority.
But yeah, ease of use is nobody's priority
In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.
Well, most of us. A good check on this is a podcast where Dr Kevin Bowers is interviewed: https://www.youtube.com/watch?v=GlzWM5GKQT8
He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.
In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html
> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.
The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:
- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.
- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.
> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.
I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.
When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.
Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"
But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.
Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.
Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.
Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.
But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.
But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?
A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.
I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.
I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.
Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.
In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.
Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.
But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.
I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).
The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.
When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.
Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).
If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).
If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).
I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.
Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.
Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.
$ oq info cfg
Looking at the following paths (the last wins)
/opt/openquake/oq-engine/openquake/engine/openquake.cfg
/opt/openquake/venv/openquake.cfg
However, I do not think there is any standard API to follow, I used "info cfg" but many alternatives were possible. So sysadmins have to explore to find out which is the introspection command to give. Not ideal, but I don't think there is a solution.
https://docs.spring.io/spring-boot/docs/current/reference/ht...
I have been leaning towards a different approach. Rather than reading multiple config sources at runtime, read them at build time, and emit a single config file with everything fully resolved. Put that on the filesystem, and have the app read it. You have an easy canonical reference for the configuration, with no need to expose anything. The lifecycle of the config doesn't have to match that off the app binary, so it could (and probably should) have a separate build and release process.
The problems with this approach when it's enabled is that log aggregators (e.g splunk) can end up including stuff like passwords which shouldn't be visible to the group able to view those logs, so some care is needed.
Logging has its merits, like discoverability when you stare at it without suspecting configuration to be the culprit, but I believe that there can and should be more.
hopefully!
I recently discovered extra-enforcer-rules which can help with that, as long as your classpath comes from Maven dependencies.
If you don't care (eg [0]) then do nothing and let the user bang it's head and reinvent the wheel.
If you care - just sort it and mark it as 'platform dependent' or somehow. Nobody really cares for the exact rules and a simple \d\d\-\w+\.conf pattern for the filenames is more than enough in 99% of cases.
[0] https://www.zabbix.com/documentation/6.0/en/manual/appendix/...
This would be a problem for first wins too.
There are plenty of applications that read a set list of configuration files and use either the first or last setting found. I agree with the responder that the last setting usually makes more sense.
Rejecting multiple definitions is better, unless you care about DoS (I would be pissed if I got locked out of ssh access due to an option being defined twice, for instance).
At least the inspector in Safari (and maybe other WebKit browsers) does something similar for CSS.
I’m all for it. It’s a pain when writing config systems (no longer just key/value, but key+value+meta), but very helpful. It can be a pain for things like JSON where libraries don’t give you that type of diagnostic information easily, however.
And every configuration that overrides the built-in defaults are reported one by one.
Also, I started to detail how configuration file works in README.md file, under its section now.
Lastly, I decided to fix all of my tools to work on same specs from now on, regardless of the programming language they're written in:
1. Use TOML, and TOML only.
2. Config file overrides the defaults, Command line flags overrides the configuration file.
Don't forget "user configuration file overrides system configuration file (but is overridden by command line)".
In tarsnap the priority (lowest wins) is:
0: Command line
1: ~/.tarsnaprc
2: /etc/tarsnap.conf (may be in a different system etc depending on platform standards)
3: tarsnap defaults.
More generally, for multi-options do we maybe want "first file wins"?
Anyway, I thoroughly document the order and precedence, but the suggestion from Chris' blog to report more detail about the process is a good one.
And the thing with developers is that they make lousy scientists. If they want something to work, they’ll take any information that supports that theory and run straight to the repository with it.
The trick is not to fool yourself, and you’re the easiest person to fool.
For example, if a change has been introduced to a config file, but the process has not been restarted to incorporate those changes, this should be conveniently mentioned.
https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/refer...
This is a big ridiculous, but the fact is that the entire order of processing there is a result of practical need evolved over likely thousands of organizations.
There are unit testing concerns, dev/test/stage/prod concerns, CLI args, OS vars, config files, IDE vs non-ide concerns, and more. And even some references to databases (if you squint and see JNDI as a database).
So yeah, finding out where property value X was set can be a bit maddening oftentimes.
"Configuration" is an afterthought in most languages and frameworks in initial iterations. So bespoke / one-off solutions will explode early on. As the original complaint details, "configuration" isn't just a simple property file. It gets complex pretty quickly, and a good configuration subsystem would be nice:
- supports most of the concerns above
- provides a means for logging the final config value set and WHERE IT SOURCED FROM
But... oh wait, config values aren't just key-value. Wait until you get to trees of values or other more complicated config. Then value overlaying gets pretty hard, and resolution code gets complicated too.
Think of infrastructure as code frameworks. The configuration is insanely deep and complicated. Many complicated systems have "configuration languages", and they are probably Turing Complete.
And once your configuration system is complicated enough ... your configuration system kind of implicitly needs ITS OWN configuration system.
So maybe the first config layer is actually a bootstrap for loading all the configuration info.
And.... then you have config values you need to secure. Hooo boy, now you have all the complexity of high security systems interfaces: authentication and authorization, decryption and hashing.
And invariably there are binary blobs encoded in text friendliness.
It gets crazy. Fast.
BUT THE AMOUNT OF SIMILARITY IS NUTS. NUTS. This is begging for a common approach or standard or library.
A bunch of applications at my job use a config-loading library that will fail on startup if the same option is set to two different value (on different lines). It's certainly frustrating at times...but it's also prevented a huge amount of ambiguity.
I forget what tool I use regularly that actually does this. Is it git?
Depending on how much config you have, sooner rather than later you’ll have multiple sources: system defaults, user config files, sometimes profiles, env vars, and flags. Configuration may be sourced from a combination of these, so overrides need to be handled and naming needs to be consistent. Another source of mess is deeply nested config, such as for a user-defined project. Those are very awkward to add flags for, in which case it’s better to enforce a file (like kubernetes config).
To me, tools like git strikes a good balance. And by that I mean that project-local config is in a project root directory that’s not necessarily checked in to VCS, but picks up the config even if you’re in a subdir. It rarely causes any unwanted surprises, but at the cost of being somewhat flag heavy, which is then alleviated with user-defined aliases.
Especially those out-of-context .yaml files with lots of DSLs embedded.
I'd say that in 70% of things I work on, part of that time is figuring out what config files are used because the app/service/process/whatever is doing something the owner/tech/whatever doesn't expect, yet is rational, and they "didn't tell it to do that".
It'd be great if everything reported as the article mentions -- amazing, in fact -- but I feel like it's a pie in the sky wish. Like asking that all programs always exit cleanly or something. I feel like there'll always be inevitable edge cases -- libraries that themselves pick up config files or registry values or environment variables that aren't expected, who knows -- that'll need to be discovered. If things are already not going right with a program, I'm less likely to trust what it says it's doing and more likely to just watch it and see what it's actually reading and doing.
I found the explanation: https://github.com/httpie/httpie/issues/480 httpie doesn't look into /usr/local/share/ca-certificates but apparently it's not httpie's fault, it's the python requests library that doesn't check that folder.
I don't know what to think of it and who is supposed to fix it. But I am back to curl with my tail between the legs (because I oversold httpie to my coworkers) for now. There's only one place where the bucket stops with curl.
> I think I forgot to run "pip3 uninstall" and ran only "pip uninstall" as I had to use python3 to get working ssl in the first place.
You can override the REQUESTS_CA_BUNDLE env var to point to your own certs.
https://stackoverflow.com/a/37447847
And then document it misleadingly badly. These options are mandatory! (lists 5 of the 10 actual mandatory options).
Then a script branches on a env var value, which isn't one that's documented, and then fails opaquely if it took the wrong branch because you didn't know you needed to set that env var.
Best ones are the ones that consume your env vars and set _other_ env vars, it's great!
To highlight a few examples of how I think they're good:
> STRIMZI_ZOOKEEPER_ADMIN_SESSION_TIMEOUT_MS Optional, default 10000 ms. The session timeout for the Cluster Operator’s ZooKeeper admin client, in milliseconds. Increase the value if ZooKeeper requests from the Cluster Operator are regularly failing due to timeout issues.
We know if it's required, what it defaults, and what I love to see, why we might want to use it.
> STRIMZI_KAFKA_MIRROR_MAKER_IMAGES ... This [provided prop] is used when a KafkaMirrorMaker.spec.version property is specified but not the KafkaMirrorMaker.spec.image
I like this, explaining when this env var will or will not be used.
> STRIMZI_IMAGE_PULL_POLICY Optional. The ImagePullPolicy that is applied to containers in all pods managed by the Cluster Operator. The valid values are Always, IfNotPresent, and Never. If not specified, the Kubernetes defaults are used. Changing the policy will result in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters.
Firstly, always great to enumerate all accepted values for enum-like props. But what I really like here is that the consequences of altering this value are explained.
> STRIMZI_LEADER_ELECTION_IDENTITY Required when leader election is enabled. Configures the identity of a given Cluster Operator instance used during the leader election. The identity must be unique for each operator instance. You can use the downward API to configure it to the name of the pod where the Cluster Operator is deployed. (Code snippet omitted)
Interactions between config options highlighted - if you set STRIMZI_LEADER_ELECTION_ENABLED to true, this option is now required.
We're told that this must be unique. And a suggestion on one straightforward way to do this, with example code.
One more thing to call out as good:
> The environment variables are specified for the container image of the Cluster Operator in its Deployment configuration file. (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml)
Being told _where_ in the source code the env var is being consumed is great.
Now compare the Confluent docs for their Kafka Connect image. [1] A colleague of mine was using this image, and was connecting to Confluent Cloud, so needs to use an API key and secret. There's no mention of doing that at all. But you can. Just merely set the CONNECT_SASL_JAAS_CONFIG option, and make sure you also set CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM and CONNECT_SASL_MECHANISM.
And very importantly, don't forget CONNECT_SECURITY_PROTOCOL, as not only will you have odd connection failures (usually manifests as the client dying at the ApiVersions negotiation stage of the Kafka protocol's handshake), but because a script run in the image uses the value of that undocumented setting to execute a prerequisite in different ways [2], and you'll get weird behaviour that obscures the real issue.
2) Support more than one way of configuring things - maybe I want to mount in a config file, maybe I want to provide a ConfigMap, maybe I want to do it via env vars. Well... [3]
[0]: https://strimzi.io/docs/operators/latest/deploying.html#ref-...
[1]: https://docs.confluent.io/platform/current/installation/dock...
[2]: mch82 ↗ Thanks. This is great!
if you know the PID aka Process ID for the application yr running try lsof -p PID_NUM_HERE
it will give you a list of every open file your application is using along with it's full path
finally, if that gave you too many lines try lsof -p PID_NUM_HERE | grep -E 'txt|properties|config'
I know you wanted the files listed it in a help file however this will work 100% of the time
Best, Stephen
https://macos-defaults.com/#%F0%9F%99%8B-what-s-a-defaults-c...
I think having a standard utility API for *nix configs makes a lot of sense. I'm surprised it doesn't exist.
I tried to find one, and there are some libraries for reading and parsing in every language, but nothing that seems to cover everything.
This bash script seems to be a fairly "built in" way to parse them: https://unix.stackexchange.com/questions/441076/which-is-the...
Exactly. But Registry has been using NTFS and it's capabilities for rollbacks and recovery. Therefore, the problem is mostly solved. There are occasions [0] of bugs causing corruptions though but they are very rare.
0. https://www.bleepingcomputer.com/news/microsoft/windows-10-n...
It allows you to trace system calls, and, crucially, file opens.
Makes life a lot easier imho.