67 comments

[ 3.3 ms ] story [ 118 ms ] thread
One of the most useful group-related tricks (introduced in BSD Unix, I believe) is the setgid bit on a directory. If this is set, new files and subdirectories created within the directory will have the same group as the directory, rather than the group of the process that created them.
I use this as a basic indicator of someone's Unix competency. Not as a straight binary thing, but if I'm talking to someone (interviewing or maybe debugging something) I've found that if you know about setgid/setuid you have probably been interested enough to know what you are doing.
I've seen interviewers ask for the sticky bit and the difference between hard and soft links as similar shibboleths.
I've seen them ask how to restore execution permissions after `chmod a-x /bin/chmod'. I usually answer to those smartasses that I my favourite way is to run the very non-executable /bin/chmod without any file juggling. I'm yet to meet anybody who wasn't surprised that it's possible.
explain?
Manually call e.g. /lib/ld-linux.so.2 (man ld.so for more info).

    shabble@host:~$ cp /bin/chmod /tmp/chmoo
    shabble@host:~$ chmod 600 /tmp/chmoo
    shabble@host:~$ ll /tmp/ch*
    -rw------- 1 shabble shabble 59K Nov 21 17:15 /tmp/chmoo
    shabble@host:~$ /tmp/chmoo
    -bash: /tmp/chmoo: Permission denied
    shabble@host:~$ /lib64/ld-linux-x86-64.so.2 /tmp/chmoo
    /tmp/chmoo: missing operand
    Try '/tmp/chmoo --help' for more information.
Well, that's a new one on me. I'd just have 'cat'ted the chmod executable onto some other executable, which keeps the permissions.
I'm a little surprised some smartypants hasn't come along and added a check to ld to prevent this from working, probably for "security" reasons.

I've known about this trick for a long time, but I've always thought its days were numbered.

Another way to solve the problem:

  scp unfuckedhost:/bin/chmod . 
  ./chmod (stuff)
This is my go-to method for doing that. Years ago, in the 90s, I used that trick (with ftp) to overwrite a custom login script used to "prevent" users from logging in. You'd login and it would spit out like "You are not allowed to login!" and then immediately log you out. Well, ftp (this is before sftp or scp) let me copy over that. This was actually one of the computers used by the newspaper in the next town over. I'm still shocked actually that the worst I ever got from the sysadmin there was a stern phone call.
I've worked around this without knowing about setgid ...

It's funny how easily I suffer tiny papercuts that way, it should be obvious "there's a better way".

(comment deleted)
I finally created an account specifically to upvote this. I have created a shared folder in my laptop, running Ubuntu, with symbolic links from 2 users home directories (to avoid having data (mostly photos) stored twice) and manually fixing the group of new files has been a chore!

Thanks!!

  [dozzie@zorn /srv]$ ls -ld foo
  drwxrwxrwx  2 root  wheel  uarch 2 Nov 21 12:57 foo/
  [dozzie@zorn /srv]$ touch foo/nabla
  [dozzie@zorn /srv]$ ls -l foo/nabla
  -rw-r--r--  1 dozzie  wheel  uarch 0 Nov 21 12:58 foo/nabla
  [dozzie@zorn /srv]$ id
  uid=1001(dozzie) gid=1001(dozzie) groups=1001(dozzie)
  [dozzie@zorn /srv]$ uname -s -r
  FreeBSD 11.0-RELEASE-p1
This is called "BSD semantics". You can get the same behaviour with appropriate mount option for Ext2/3/4 and XFS.
What filesystem was that? And why is this behavior filesystem-specific?
Not what filesystem, it's what OS. FreeBSD. And ZFS, but UFS works the same.
I could be wrong but I thought the file system's "drivers" did the handling of file system permissions rather than the OS / kernel? Or at least this was my experience when playing around with FUSE on hobby projects.
So why did you mentation "Ext2/3/4 and XFS" if filesystem doesn't matter?
I don't see what you don't understand. I demonstrated that on FreeBSD created file inherits the group of its parent directory even without SGID, i.e. that FreeBSD exhibits BSD semantics. I also added that if you can have BSD semantics on Ext2/3/4 and XFS if you mount them under Linux with appropriate options.
(comment deleted)
No, you didn't say anything about Linux.
You’re getting downvoted a bunch here, but I believe your basic point is valid. To take it back to the GP: SGID bit and other such OS and/or filesystem specific semantics are not the best measurement of someone’s overall UNIX understanding.

Now, IMO asking someone if they understand what the difference is between chmod <octal notation> VS chmod <friendly notation> is and why it exists is a bit more telling, because this touches down on what interface most unix filesystems try to present and also how they’re implemented to a degree.

The specifics on any given system are just a manpage away but the general understanding only really comes from having absorbed some actual unix interactions.

And certainly knowing more than just Linux should get one bonus points...

While it’s nice to have it as default behavior when it makes sense, it’s also nice to have setgid as an option rather than the rule.
Nice or not, you can't talk about directory's SGID behaviour as it was unversal truth, because it's Linux-specific, and even then it's something that can be changed.
Except it does work FreeBSD:

    [lau@primus ~/test]$ mkdir setgid nosetgid
    
    [root@primus /home/lau/test]# chmod g+s setgid
    
    [root@primus /home/lau/test]# chgrp lau setgid
    
    [lau@primus ~/test]$ ls -l
    total 1
    drwxr-xr-x  2 lau  users  4 21 Nov 14:52 nosetgid
    drwxr-sr-x  2 lau  lau    4 21 Nov 14:51 setgid
    
    [lau@primus ~/test]$ touch {nosetgid,setgid}/test
    
    [lau@primus ~/test]$ ls -l *
    nosetgid:
    total 1
    -rw-r--r--  1 lau   users  0 21 Nov 14:52 test
    
    setgid:
    total 1
    -rw-r--r--  1 lau   lau    0 21 Nov 14:51 test
Looking at your example, I'd guess you didn't set the setgid bit as your `ls` doesn't show the group column to be `S` (instead stating `X` which is the default a directory would be set as).

OS build and file system:

    [lau@primus ~/test]$ uname -a
    FreeBSD primus 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
    
    [lau@primus ~/test]$ mount | grep " / "
    zroot2 on / (zfs, local, nfsv4acls)
> Except it does work FreeBSD:

Congratulations, you have an incomplete example.

And what exactly did you want to prove? That an OS-specific behaviour can be exhibited by more than one OS?

You said, and I quote, "it's Linux-specific" and I was demonstrating that it's not Linux-specific.

I also question your point that it's "OS-specific" because I'm pretty sure I had to handle the permission logic myself in the hobby file systems I've written (albeit they were compiled against FUSE rather than kernel objects). However I don't have any examples to hand which I can provide to that end. Hopefully a more experienced file system developer can jump in here as to whether this behavior is controlled by the OS or by the file system. However the fact that you can alter it's behavior via mount options would also suggest I'm right.

> You said, and I quote, "it's Linux-specific" and I was demonstrating that it's not Linux-specific.

You were attempting at demonstrating that the OS specific behaviour can also be exhibited elsewhere (also, you failed at that attempt; you haven't shown what group your shell belongs to). If you were to demonstrate that it's universal (as opposed to be OS specific), you would need to invoke some standard that describes the behaviour, which you didn't.

> I'm pretty sure I had to handle the permission logic myself in the hobby file systems I've written

Hint: what's the ownership of a file created on FAT mounted under Linux? It should suggest you what part of the kernel needs to handle the permissions.

And by the way, filesystems are typically implemented in kernel, so saying that it's not kernel that handles them is somewhat misguided. It's just that Linux implementations of filesystems usually set the group of the file to the EGID of the process that created the file.

> If you were to demonstrate that it's universal

I really wasn't. I was just making a point that it's not just Linux-specific like you claimed. In fact if you re-read the thread you'd realise that nobody argued this behaviour was universal. Given the breadth of POSIX-like systems out there, I doubt anyone would dare make such a bold statement. So you're arguing against point that was never raised.

> And by the way, filesystems are typically implemented in kernel, so saying that it's not kernel that handles them is somewhat misguided.

Well yes file systems generally are, except when they are not. For example ntfs-3g (which is also compiled against FUSE). Plus the file system code is modular - not in the monolithic vs microkernel sense but rather in terms of the body of code is a self-contained chunk that can be optionally compiled in or not. This also means that some file systems are written with some degree of portability in mind (easier said than done I know!!!)

What the argument is about is whether the permission logic described (specifically the setgid bit on folders) is handled by the file system code or externally to the file system code in other parts of the kernel (thus the behavior is defined by the OS rather than the file system).

You're making the case that the behavior is independent to the file system implementation (thus the logic is handled outside the file system code) because it is OS-specific. I'm stating that when I've been writing file systems I've had to handle that logic inside the file system code itself thus the logic is not OS-specific but rather file system-specific.

Now there will be standard way of implementing the logic because people will have certain expectations about how the file system should behave. Much like most $SHELL's broadly follow a common POSIX standard. So you would get the illusion of the discussed behavior being OS-specific even though the actual business logic is implemented at a different stage in the stack (ie the file system code). There might also be instances where one OS might patch the file system code to change its behavior to match the expectations of that OS - but I'm just speculating here. All I know is that I've had to write the logic myself - the kernel did not handle that for me.

I'm obviously not boasting to be a low level kernel hacker so I'd happily bow to a more reasoned argument but the examples you've given have just been guesswork about kernel operations based on some flawed experimentation with user space tools (I say 'flawed' because you didn't even bother to set the setgid bit in your original example when you suggested FreeBSD didn't support said behavior).

> Hint: what's the ownership of a file created on FAT mounted under Linux? It should suggest you what part of the kernel needs to handle the permissions.

That doesn't prove either of our points. Nor anything aside that Linux can mount FAT file systems.

>> If you were to demonstrate that it's universal

> I really wasn't. I was just making a point that it's not just Linux-specific like you claimed.

I never claimed it's Linux only, only that there are other systems with different semantics.

>> And by the way, filesystems are typically implemented in kernel, so saying that it's not kernel that handles them is somewhat misguided.

> Well yes file systems generally are, except when they are not. For example ntfs-3g [...]

...except that FUSE is in-kernel mechanism to delegate VFS callbacks to userspace program. It's hard to argue in such a case that the work is not the kernel's responsibility.

> What the argument is about is whether the permission logic described (specifically the setgid bit on folders) is handled by the file system code or externally to the file system code in other parts of the kernel

Oh. This explains quite a bit, because I thought the argument was about whether the OS' usual semantics (group being inherited from directory or from EGID) are like that or not, not where in kernel this behaviour is coded.

> You're making the case that the behavior is independent to the file system implementation [...]

No, I'm making the case that Linux usually works one way, and BSD usually works the other way. I never claimed that anything was coded in any specific kernel subsystem, so don't add the statement yourself.

We are going round in circles so I'll have one last attempt to explain my point :)

> I never claimed it's Linux only, only that there are other systems with different semantics.

Maybe you shouldn't have said "Linux-specific" if you didn't mean it was specific to Linux.

Also I demonstrated that FreeBSD + ZFS behaves the same as Linux on my system. I cannot comment about why your example didn't work the same. Perhaps you were using UFS instead of ZFS? Or non-default file system mount options / or other tweaks? Aside running ZFS as root and having compression enabled, my root zpool is pretty vanilla. But either way, it goes back to my point that this is a file system specific behavior not an OS-specific one.

> ...except that FUSE is in-kernel mechanism to delegate VFS callbacks to userspace program. It's hard to argue in such a case that the work is not the kernel's responsibility.

Your counterargument is just reiterating my point. File systems compiled against FUSE do not run in kernel space. The kernel syscalls obviously need to hook to your file system still (which is where the FUSE wrapper comes into play), but the FUSE API does all the user space / kernel space memory swapping so your actual FUSE file system doesn't run in the kernel.

This will obviously cause some extra overhead but with the exception of ntfs-3g, most file systems compiled against FUSE tend to be less performance-orientated anyway (eg they might be network bound). But as an aside note, this is also why I don't recommend people using ntfs-3g on low-footprint devices like the Raspberry Pi.

> Oh. This explains quite a bit, because I thought the argument was about whether the OS' usual semantics (group being inherited from directory or from EGID) are like that or not, not where in kernel this behaviour is coded.

> No, I'm making the case that Linux usually works one way, and BSD usually works the other way. I never claimed that anything was coded in any specific kernel subsystem, so don't add the statement yourself.

I know, I was the one that brought code into the debate. I did so very deliberately because software is code and you were making a point about the behavior of software. I'll I'm saying is that I don't agree with you that this behavior is OS-specific because the behavior is defined by the file system. Ie one file system could handle this behavior differently to another file system on the same OS by default. This is why I make a point about "where in the kernel the behavior is coded". It's not just an academic point, it's directly refuting the entire argument you've been having.

Anyhow, since your profile lists you as a developer as well as systems administrator - and you clearly do have a good understanding of the underlying architecture of BSD and Linux - I suggest you play around with a hobby file system yourself. It's good fun and FUSE is dead easy to learn as there will be bindings for most popular languages. I did mine in Go using an API that closely mirrors FUSE's C libraries but I've seen that most higher level languages (eg like Javascript) have FUSE bindings too - if that's your thing instead. Or obviously there's Rust and C/C++ if you prefer to work more directly with FUSE. You don't even need to work with hardware since you could just wrap FUSE around MySQL queries (as one of my latest experiments does) or even contents described in an ASCII file (as my first project did). Because Linux / BSD / whatever doesn't handle the actually writing of the file system (and why would it?), it means it's up to the file system to implement whatever set of behaviors surrounding permissions it wants.

> Also I demonstrated that FreeBSD + ZFS behaves the same as Linux on my system.

No, you didn't. You attempted that, but you failed miserably by not including the most important piece of information: primary group of your shell.

> Perhaps you were using UFS instead of ZFS? Or non-default file system mount options / or other tweaks?

No, it was ZFS, as I stated clearly elsewhere. No options from me whatsoever. Now try the experiment with creating files again, but this time check your process' EGID. It may turn out that your interpretation of your own output was wrong.

> I'll I'm saying is that I don't agree with you that this behavior is OS-specific because the behavior is defined by the file system.

Being defined by a filesystem handling code and being the usual behaviour under particular OS are not the same thing.

Your claim boils down to disagreeing that it's the most prevalent behaviour for the OS, and you justify your position with the behaviour being coded in particular place, no matter that most of the other places for a given OS work in the same way. This is silly stance.

> I suggest you play around with a hobby file system yourself.

And this exercise would give me what, assuming that I haven't implemented a filesystem already?

> No, you didn't. You attempted that, but you failed miserably by not including the most important piece of information: primary group of your shell.

I did check that but must have missed it off the post. Sorry about that.

> Being defined by a filesystem handling code and being the usual behaviour under particular OS are not the same thing.

That was my whole bloody point! Glad you finally agree with me. It's the file systems that define that behavior so it's file system-specific not OS-specific (nor Linux-specific) like you previously claimed.

> Your claim boils down to disagreeing that it's the most prevalent behaviour for the OS, and you justify your position with the behaviour being coded in particular place, no matter that most of the other places for a given OS work in the same way. This is silly stance.

No, my point was that it's a file system-specific behavior not an OS-specific behavior. A point you then tried to pass off as your own (above) despite me categorically stating - numerous times throughout the discourse - exactly what my position is. Honestly I think this argument for you is less about facts and more about gradually moving the goal posts so you don't appear wrong.

> And this exercise would give me what

A greater understanding of a topic you've been happy to engage in arguments about yet reluctant to listen to other peoples feedback on. You're clearly an intelligent guy so if you're not going to take the experience of your peers into account (and I cannot blame you for wanting to trust your own experiences) then why not research this topic properly instead?

> assuming that I haven't implemented a filesystem already?

It's a bit late in the discussion to be playing the bluff card. :P

Dude, my whole point revolves around the fact that the most prevalent behaviour for a given OS is OS-specific, which is a fucking tautology. What kind of feedback you expect me to accept from somebody who tries to make me believe the tautology is false?

Not to mention that the somebody tries to imply that I don't have enough knowledge about topic I'm quite sure I have, without ever pointing any error on my side, beside the tautology thing above.

Congratulations, you're passive-aggressive here without any reason to be.

Explains the downvote flood pretty well.

Not sure why this is being downvoted. Historically SGID did do something different - and may still in some of the commercial unixes.
From the HN guidelines:

Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

SunOS 4 definitely had this behavior, and I think it's even older than that — it probably goes back to 4.1bsd.
I absolutely love Julia Evan's writing. I find her articles / zines to cover interesting and useful technical topics while remaining _extremely_ accessible to me (especially when I was a student)! I really admire her ability to present technical topics in plain language.
I fully agree. Even on topics I feel I know well, she's 1) shown me something I didn't know, and 2) shown me how a "newbie" could see the topic, making it easier for me to help teach/mentor someone else.
It's pretty interesting how one can make a subject look accessible. Often the main factor for ignorance is ceremony.
Also "newgrp" can be used for join "new" groups without re-login.
And here I've been using `su - $ME` all this time. Thanks!
Worth noting that it's not functionally all that different in that newgrp spawns a new shell under the current one; aside from the - creating a login shell anyway - but you can do that with newgrp too
Some highlighted words do not appear using Firefox 57 on my Ubuntu 16.04.3, like the word "julia" at the third line. It's OK with Chromium.

EDIT: Seems a Firefox related bug, i noticed this strange behavior with other websites inc. Stack Overflow since then. Will investigate asap.

i am also using firefox 57 on ubuntu 16.04.3 and it looks fine to me. it must be something with your specific setup.
Ohh, so THAT's why I have to log out and back in to have my group changes take effect. That, along with having to start a new shell to pick up new env vars from ~/.profile, are my two biggest annoyances with the Linux process model. Reminds me a bit of the Windows 98 days when you had to restart to change your IP address. I really wish someone would sit down and figure out how to propagate group and environment changes to already running processes and implement them.
Take a look at the command “newgrp”
I think the op is looking for something like

addgroup && newgrp

You can load your .profile into your current shell using either '. ~/.profile' or 'source ~/.profile'.
(comment deleted)
For what it's worth, you have to logout/login on Windows for group changes to take effect. This makes me suspect something similar is at play in the Windows world.
The book she mentions, "The Linux Programming Interface," sounds pretty useful, but it's seven years old at this point. Does anyone know how much has changed or if there's a new version coming any time soon? Seems like it's worth $70 but the age has me concerned. I'm sure the basics, like the things this article is about, haven't changed, but I bet all the stuff around cgroups would be useful for how setgid works with a process.
Useful is an understatement of The Linux Programming Interface (TLPI), yes seven years old but the only major thing that has probably changed is the number of syscalls has risen, other than that, still solid for understanding the OS primitives and interfaces that the Kernel uses to interact with user land. On the BSD side "The Design and Implementation of the FreeBSD Operating System (2nd Edition)" is also a great book to further one's understanding of operating system interfaces on the BSD side.
Cool, thank you for the feedback!
(comment deleted)
The big thing I've found missing is discussion of containerization primitives, e.g. cgroups. The other Linux namespaces are discussed where appropriate, but in 2017 it would be nice to have a section on all these together.
"dr--r--r-- 1 bork awesome 6872 Sep 24 11:09 file.txt"

This doesn't make sense.

$ mkdir file.txt

$ ls -l

drwxr-xr-x 2 USER GROUP 4096 Nov 21 11:49 file.txt

$ chmod -xw file.txt

$ for i in {1..180}; do touch file.txt/long_name_$i; done

$ ls -l

dr--r--r-- 2 USER GROUP 12288 Nov 21 11:50 file.txt

File size is a bit off, but that's based on sector size and may be separately configurable - that's a bit beyond my knowledge.

but she says right above it:

"So, for example, if a process is owned by the julia user and julia is in the awesome group, then the process would be allowed to read this file."

I'm not sure what you aren't understanding. Directories are just a type of file, so that's not wrong, the dir/file is owned by the `awesome` group, and the group permissions are `r--` so the group can indeed read it.
Or one could apply Occam's razor and admit that "d" was a typo.
Of course, that's a much more likely scenario. I was attempting to show that the line was not completely nonsensical is all.
It's kind of useless, but not illegal. You won't be able to stat/open any of the files in the directory, although anybody can read their names. Putting a .txt on a directory name is also perfectly legal and still in bad form.

I agree the most likely explanation is accidentally putting the d in there in the editing process.

Regarding setuid, this is why when you run programs like ping(8) it doesn't require root access to open a raw socket. ping's setuid is set so upon execution it executes as root since ping is owned by root and then calls setuid(getuid()) to run as the intended user.
In some distributions, ping is now no longer setuid, but instead setcap with CAP_NET_RAW to narrow down the privileges gained.