This is one of the reasons why the kubernetes network model[0] is kinda neat. Every service running in each pod can bind to port 80 without any conflicts because every pod gets a dedicated ip address.
No, not at all. This problem was solved decades ago via a myriad of different workarounds.
You can simply turn off privileged ports if you feel like it. Or do things like setuid such as how Apache starts as root but switches to userspace immediately.
It's not ideal, but to say the privileged port thing is an issue is bizarre to me. It's the least interesting item the author brings up in the article, and certainly was not the primary driver behind multi-tenancy virtualization.
Better yet is using reverse proxies - service providers have been doing this for ages. A single shared HA cluster of haproxy, that maintains records for each individual application/tenant that lives wherever it likes. This is the model I prefer, since it allows me (as a sysadmin) to protect the developers from themselves by being able to easily filter at layer 7 in front of their application as needed. Also lets you direct traffic easily, and is generally the model used by any container orchestration service.
Very easy to expose all that via various APIs and tooling so users can self-service.
In Linux the thing the author wants is /proc/sys/net/ipv4/ip_unprivileged_port_start which defaults to 1024 but can be set to anything you like. Such as 0.
Edit: I didn't realize how new that was. Kernels 4.11+ only. I think some people were using this on custom patched kernels though because I've been seeing it around. Was committed in January.
How does that solve the problem? How to 100 users bind their locally-installed web server to port 80? How do you make it so that only my user can bind to only my IP?
No, the things that would have obviated the author's specific example were (a finer-grained version of) ip_unprivileged_port_start and SRV records back in the 90s, when the customs actually developed.
That's the whole point about path dependence. We went down this massively more complex and expensive path because we didn't do a few teeny weeny little things to network interfaces and permissions back in the 90s.
> Step two: extend user and group permissions and ownership into the realm of network resources, allowing UID/GID read/write/bind (bind in place of execute) permission masks to be set for devices and IPs. By default a bind() call that does not specify an address (0.0.0.0 or ::0) will listen for packets or connections on all interfaces that the current user has the appropriate permission to bind and outgoing connections will default to the first interface the user owns.
I think that this can be achieved with network namespaces and some iptables magic, entirely in userspace (and without the whole burden of containers).
Anyway, IMO the point of containers is not network virtualization, but rather isolation of dependencies. It's much easier to just pack everything into a container than to invent proper non-root package manager (like Nix).
The point of directories is to provide proper isolation of dependencies. But that followed the same path.
What leads to... It's better to add non-root mounting into the author's list. I don't think there's even anything that needs to change, permissions are already there.
Well, someone has to administrate the nginx routing config, or there needs to be a mechanism for each webapp sitting behind the nginx to register itself as a forwarding target.
What I want to know is why can't we get all the major web browsers to look for (and honor) SRV records, so that www.mydomain.com can transparently have the browser connect to port 30338 (or whatever port my app is listening on). Then we dispense with the need for proxies altogether.
No problem at all, given that the underlying operating system would already require that somebody got privileged access.
The point of the article as I understand it, isn't so much about the possibility of shared web hosting - it's obviously possible. It's more about how early design decisions shape our paradigms and the way we understand computing and the necessary resources.
This wouldn't change anything because you cannot run multiple service instances on same port so users still would have to specify port in URL. SRV record in DNS could help if anything supported it. But it is not needed now because with IPv6 we have enough addresses to assign each process unique IP.
> Reproductive organs are what evolutionary biologists would call a highly conserved system, meaning they don't tend to change often.
No. Reproductive genes are well-known for what evolutionary biologists (such as myself) call positive selection, meaning they tend to change more often than you would expect.
Well, the author was talking about organs, not genes. Generally speaking the organ arrangement described in the article (external testes, internal ovaries) is pretty well conserved among placental land mammals even if genes related to other aspects of reproduction are not.
The author is wrong in saying that the testes are not protected, though. They're quite well-protected, with a leg on each side. Simple proof of this is that it's clear when these are damaged due to the pain, but most men will only experience this pain a couple of times in their life.
The testes are even more protected in quadrupeds, far away from any frontal fighting or sparring. And if you're a quadruped that is being chased by something that can damage your testes from behind, you have far bigger problems than damage to your testes.
There are plenty of trade-offs but network resource ownership?
For a datacenter isn't it more efficient to pack workloads across dense machines than to over-provision idle ones?
While a factor, maybe, I don't think privileged ports are causing excess CO2 production.
Although I've been interested in low-power edge networking with uni-kernels, would be an interesting project to work on (thinking micro-dc's with homogenous ARM-based boards at the edge and having a dynamic system to boot up uni-kernels on demand closest to the requesting user... a system that mostly stays off unless it's needed).
X.25 isn't the whole networking stack. Elements of OSI were available, and are still available, but most people use something else because it's easier: good enough, soon enough.
I'm not sure that the OSI people even realized that they were in competition with anything else. I think they basically neglected the low end of the market.
At the time there were multiple LAN protocols that were mostly used for file sharing, Netware, Appletalk, NetBEUI, etc. You had NFS for TCP/IP but it wasn't really used for anything other than UNIX workstations.
Probably it would have required somebody to write a cut-down OSI stack for MS-DOS that could be linked to a particular killer application (whatever that was, maybe FTAM).
This would still have had to compete with the way that TCP/IP was able to swallow up the other LAN protocols, we wouldn't have had to go through the IPv4 to IPv6 migration though.
The "privileged port" model was basically dead at the time of the Morris worm 30 years ago. It should have been sorted out in that time but there was never the momentum.
Everything in UNIX is a file, apart from the things that aren't. If ports were files, you'd be able to chown them or put them in a group for delegation purposes.
Isn't this exactly what shared hosting is? I bet companies like dreamhost regularly put 50 clients on a box. And Webfaction[0] will even give you pretty open shell access to a shared environment.
They get around the port issue by using a nginx frontend that uses the Host to route each request to the correct apache instance each user is running out of their home directory. All these apache instances are bound to higher unprivileged ports.
Imagine a world where you don't need a router! A world where the browser would look up a http/s service record for the host and connect directly to the unprivileged port.
> How did we end up with the nested complexity explosion of OS->VM->containers->... instead of the simplicity of multi-user operating systems?
> [...], it pushes about 5-10 megabits of traffic most of the time but the software it runs only needs about 10mb of RAM and less than a megabyte (!!!) of disk space. It also utilizes less than 1% of available CPU.
> The virtual machine it runs on, however, occupies about 8gb of disk and at least 768mb of RAM. All that space is to store an entire CentOS Linux base installation, [...]
The exo-kernel and uni-kernel people have the opposite idea: go all the way to virtualization, and eliminate the traditional operating system.
You can still have all the conveniences of the programming models you like, but eg file-systems and network protocols will be implemented in user level libraries.
I can easily see a world where history repeats itself here. VMs get thinner, hypervisors get fatter to make up the difference, all the other layers of abstraction get squished out of the system, and what we're left with is just another process model. It's a model built on different assumptions than the one we're used to, with stronger guarantees about isolation than we might be comfortable with at first, but one that has an appeal even for desktop computing.
Later, once everyone has a hypervisor in their pocket, we tell our children about how cloud infrastructures used to be so expensive that only big corporations could maintain them, and they came in server racks the size of refrigerators.
Exokernels are, as I understand them, even lighter weight than VMs. Instead of user-level libraries implementing file systems and network stacks for emulated hardware, or dedicated hardware on a machine with multiple instances of those devices, the kernel's protection mechanisms are redesigned to work at a lower level: disk blocks as managed by inodes, raw packets as identified by filters, time slices as scheduled by user space.
Combined with a sandboxed-by-default capability security model, I think that's the best approach. You no longer need virtualization or users or containers- you just assign hardware and software resources to applications, following the principle of least power. You not only still have the same convenient programming models, but the same (or arguably better) monitoring and management tools.
Yes, and the only thing the kernel (or hypervisor) would do is provide for security and isolation, but not provide any abstraction whatsoever. Not even cross-platform hardware compatibility: all of that is for the user-level libraries.
Somewhat related, but I always thought it would make sense for browsers to use DNS based service discovery to find the correct HTTP port, and then fall back to 80/443. That alone would make it easier for users in a multi-tenant environment to self host, IMO, and eliminate some of the need for SNI.
Unfortunately, HTTP is just too widespread of a protocol - you would end up having to listen for legacy clients on 80/443 forever, making it a nonstarter.
The sooner we get started, the sooner we'll be able to actually take advantage of it.
It's true that you have to wait a long time, but that's no reason to avoid it. SNI wasn't widely supported initially, for example, but these days you can safely require it unless you have weird requirements.
There probably isn't enough need for it to get the major vendors to start adopting it, though.
Browsers won't lead the way. Fetching SRV records will increase network latency, and that's one of the main points of competition among browsers today.
My DNS knowledge is a bit rusty. I remember there are some issues with requiring different types of record on the same request (most servers block it), and that timeout worked differently for different types of record.
I did never try resolving HTTP services anyway, so I may be completely wrong. I've had problems with TXT and MX.
The reality is that this is not that infeasible a change as everyone wants us to think that it is. It is simply a change to the DNS resolution logic for HTTP/HTTPS. The major browser vendors could make the change in relatively short order. The major hurdle is that it would increase the number of DNS queries to resolve a website.
XMPP is an example of a protocol that currently has this correct, in that it uses SRV records to control ports for a given host.
There is no such thing as a "host-name resolution request". A DNS query specifies a domain name and a record type, and gets as the result a list of all records of that type under that name. Record types would be A (IPv4 address), AAAA (IPv6 address), MX (mail exchanger name), SRV (server name and port for a particular service), TXT (free-form text), and many others, most of them nowadays unused.
You can specify "ANY" as the type in the query and get all results. Try running "host -a ycombinator.com". It doesn't recursively resolve CNAMEs like an A query does, though. Also, at least Cloudflare refuses those requests, to reduce DNS reflection attacks.
ANY gives you all records for the given name in the cache of the name server that you are querying, which usually will be incomplete in the case of recursive resolvers.
The "It will increase the number of DNS queries" is another of the many attempts at a counterargument that is not really true if one analyses it, that has been coming up for two decades now.
In reality, the number of back-end DNS queries to perform an A or an AAAA lookup can vary enormously, and can number in the tens or hundreds of queries already. It's not generally true that the number of back-end lookups will increase, because the number of back-end lookups varies wildly by time (depending from what is cached from momement to moment) by domain name (depending from the amount of gluelessness) and by country. There is so much variation in there already that the variation caused by a two-step SRV lookup can quite often be lost in the noise.
Of course, SRV lookups do not even have to be two-step in the first place. A proxy or content DNS server is free to add the requisite A and AAAA resource record sets as additional section data in its response, meaning that the client obtains both pieces of information in a single transaction. And of course several real world DNS server softwares do exactly this.
Whenever you see these objections, always remember that there are protocols where SRV resource records have been in use for coming up to twenty years. And yet one never hears of actual problems with those protocols that mirror the supposed projected problems of using SRV lookups for HTTP. A lot of what one hears about why this would not work is simply bad analysis and excuse making.
The worst part perhaps is that the code to do this for one of the WWW browsers was actually written, the only actual technical niggles were solved 17 years ago, and -- perhaps most tellingly of all -- whilst people are still telling us how this cannot be done all these years after the code was written, there are parts of the world that are quietly and happily doing it right now. The FreeBSD packaging system uses SRV resource records and HTTP/HTTPS, for example. It's also an example of how real world DNS servers can and do send all of the data in one transaction using the additional section. They've even made it in-bailiwick.
Well, I don't think that anyone claims that it doesn't work, and I don't think some packaging system is a good comparison. I mean, email effectively has been using SRV records (even if they are called MX records in that case ...) for three decades, so obviously it works. But it commonly requires an additional round trip for (initial) resolution, which adds delay, which is kindof relevant for interactive use, not so much for package managers or SMTP.
Also, you as the site operator can't really do anything about how recursive resolvers handle additional section data (and there are millions of them deployed on crappy home routers with probably equally crappy DNS software), in contrast to out-of-zone glue, which is something completely under your control.
That being said, I don't think it makes sense to not standardize it for mandatory client-side use with HTTP, (a) because network latencies are not exactly increasing, even mobile networks are moving towards single-digit millisecond RTTs, and (b) browsers could just be backwards compatible, and issue SRV, A, and AAAA queries for the URI domain name at the same time, so you would only need one round trip to figure out the addresses of sites that don't want to use SRV, and you'd only get a negligible amount of traffic for one additional DNS request without any additional latency.
As a workaround, couldn't you just have, say, nginx listen on 80/443 and either redirect or reverse proxy to the correct endpoints based on the SRV records?
That way, compliant browsers can connect directly while SRV-ignoring browsers would just get proxied (or redirected), and once set up there would be no special actions needed by the users, save for telling the proxy about the keys and certs.
Umm, the real problem is that I can't discover what port you are on.
So, in the article's instance, the machine owner needs to set up a web server on default port 80 that redirects to the correct web server on the correct non-privileged port.
Sure you can, this is the entire point of service discovery. We've built so much crap to deal with the fact that web browsers don't resolve SRV records (and at least for HTTP/1.1 prohibited by RFC from doing so).
This article makes no sense. Virtualization's first use was to run a guest OS on a different host OS. Containerization's first use was to achieve reproducible configuration at a lower cost than virtualization. Neither were trying to solve a multi-tenant hosting problem where everybody is running the same system -- were it that simple!
The first use of virtualization that I can remember was VM/370. It did the things you said plus hosted systems for many users connecting via terminals. Supporting legacy systems and getting higher utilization with multiple workloads were also advertised benefits of VMware on x86. IBM and NUMA vendors touted isolation of multiple workloads or users with things like LPARS. The cloud providers now claim a lot of this stuff.
So, it seems like it was intended to do these things going back to the 70's. They also designed insecure, bloated solutions that INFOSEC founders like Paul Karger called them out for. Resulted in better designs like KVM/370, KeyKOS, VAX VMM, separation kernels, and recently mCertiKOS. Most peddling virtualization for multiple users still use bloated, untrustworthy components though.
I was a little bit annoyed (enough to comment) by the use of the wrong units. 'mb' is millibit, perhaps millibyte, but certainly not megabyte (MB or MiB).
Similarly:
> it pushes about 5-10 megabits of traffic most of the time
Bits are a unit of information, not flow. Probably the author meant Mib/s.
This annoyed me too. The author does not seem to be aware that case is highly important when dealing with SI/binary prefixes as well as unit abbreviations. If you don't even know (or care) that "b" is a bit and "B" is a byte, and you use them incorrectly, how am I supposed to trust your technical knowledge about the rest of the stuff you're talking about?
The author is the creator of ZeroTier. He is likely aware and didn't care because it's extremely pedantic and most people know what he means in informal settings, which this is. I expect he'd use the correct case in an RFC or something.
Getting abbreviations for bits/bytes correct is not "extremely pedantic", it's about communicating correctly. In a network context, which this is, "b" means bits, but he was using it to refer to bytes. Also I've never heard of ZeroTier, but even if I had, I probably would not have made the connection that he was the author of it, so being correct about these things is important for establishing credibility with new audiences.
IIRC, most if not all uses of 'mb' or 'gb' were about disk space or RAM, so not really a network context - this is about bloat, ports are just the whipping boy.
And I'm not against pedantry in the right context, but this is just a casual, relatively nontechnical rant. Pedantry is really not needed.
I agree about the milli prefix (I hate its misuse guts, too), but netadmins
write the unit designation as either "Mb/s" or "Mbit", so among them it's kind
of jargon to talk about "megabits" when one means "megabits per second".
Jargon has this weird property of often being incorrect terminology.
But while I would accept that from a network engineer, I don't know what
background the author has.
Simple enough to add to a configuration management workflow. I used to maintain a configurable reverse proxy, until I decided that was way too much work and to only put one web facing app per server.
edit: rc.local may not be the best place to put these rules, check this link:
Unprivileged users can't use iptables, so that doesn't change anything about the blog post. I'd suggest that if it's so easy, but still doesn't get done, that still also leaves the blog post intact. In a way it doesn't help when there's a thousand ways to do something, so none of them get done.
This seems to be a common misunderstanding of the blog post. Yes, there's a ton of ways for root to create privileged ports, and a ton of ways to delegate it in various ways. But all of them require actions by root, and sysadmining, and aren't as secure as a system designed to work this way from the beginning would be, so nobody uses them, so in terms of addressing his discussion points, they might as well not exist.
But surely, creating the user requires sysadmin privs, and as part of setting up the user the appropriate work could be done to allow whatever hole-punching is required. The reality though is that if Mary, Joe and Tim all want to listen in port 80, there's going to be a mess - hence using a common server (on a privileged port!) to demux incoming requests.
I don't understand the use case here. Are people really expecting to be able to configure stock Linux distribution images purely from unprivileged user accounts?
Because this iptables hack can be built right into a custom Linux distribution.
That's not enough, because this would allow only one unprivileged user to bind to the HTTP port (in this case - 8080). Another user trying to bind to the same port would get an error. So for this scenario the author is talking about a custom solution with a reverse proxy would probably work best - getting all requests directed at 80 and distributing them to as many servers as necessary.
My solution was to only put one HTTP server on a machine. I agree that a reverse proxy is the solution for those really wanting HTTP multi-tenancy on individual machines. But is this a really in-demand use case?
The biggest change which would have made this much easier in my mind is adding port numbers to DNS. Instead of having HTTP at port 80, have the port number for HTTP returned as part of the DNS query. You can allocate a block of ports to each tenant on a host and let them use them as they see fit, you don't need to use the low ports for anything.
Great articular...
But, forgetting the (apparently GBs) of size, isn't this really what docker and such like are trying to achieve? (maybe I'm missing something). Especially when it comes to only storing a single instance of the base image if all 50+ users are sharing the same image? So it will only store the differences in each container.
It sorts out your network issues, user can be root in the container and bind to whatever port they like and IP addresses are assigned to each container...
What other overheads are there? (this is a rhetorical question - you seem much more knowledgeable than myself)
:)
It was a fun read, and there is a clever thought there which I'll get to in a minute. But first ...
The author gets a lot wrong, it isn't uncommon because most software people see a "server" as a set of numbers "A" bytes of memory, "B" bytes of disk space, and "C" hertz of clock speed.
Computers aren't numbers of course, they are systems, and the systems are a collection of moving parts that are connected by linkages which constrain and amplify the various parts. In system design there is a concept of 'balance' which discusses how the linkages enable and constrain such that an understanding of the amount of "work" that can be done by the system is understood.
There is a great example in the article comparing a Raspberry Pi to a Sparcstation 10. Most of the software the author used "back in the day" is still around in one form or another, and a RasPi is cheap, so its easy to create modest recreation of the environment from that time and to build simulated users who can access the "server". Just two or three users and the Pi will "choke". Understanding how a system with so many "bigger" numbers is less capable of doing the same "work" as one with much smaller numbers is a useful exercise to run if you are into systems analysis. The system the Pi was designed to run, and the one for which it is fairly well balanced is a smartphone. Something the SPARC 10 would truly suck at.
The clever idea however is making networking resources just another resource like disks. Interestingly enough, with IPV6 that is much easier than it was before (to the point about how things were done before in networking, vs now). It is straight forward to create a model of network interfaces that is equivalent to serial lines. Have the kernel get set an IPV6 subnet with a block of 32K addresses and let your user program open /dev/net/<1> through <32766> because its IP each "net" device comes with its own set of port numbers etc. IPC is just networking and a cleverly coded kernel running on a machine with a 64 bit virtual address space would do direct data placement. No need to do even any page table mashing.
> Just two or three users and the Pi will "choke".
Do you really know this? What's the limiting factor?
I've never played with a Pi and I don't really know, but I don't see offhand why it would be so bad. Interrupt-handling and context-switching times should be much better. There's plenty of main memory. The file system should be faster, running on flash. I don't see where the bottleneck would be.
Probably only limiting factor in the hardware is that R-Pi does not have any meaningful high-speed low-latency IO.
But on the software side there is one huge problem: today's Linux kernel and userspace is incredibly bloated in comparison to early 90's SunOS. On the other hand, said SunOS probably did not support many things that everybody expects today, like scalability to large-ish SMP systems (ie. >2 CPUs), shared libraries, loadable kernel modules, threads...
What I meant by the IO is that the aforementioned sparcstation had ethernet card connected to SBus which was capable of directly producing interrupts and doing DMA, and all the modems were connected through some multiport serial card/concentrator which tried very hard to offload stuff from the CPU. Which is completely different situation than R-Pi with everything hanging off USB with it's "novel" approach to interrupts.
It was able to, but it was not especially efficient at doing so, to some extent because the hardware platforms themselves were not scalable and only appeared to be so because of brute force (even e10k is essentially shared bus SMP machine, sun4d-style).
In fact, the NUMA support in non specialized systems (one would want to say "non-IRIX") is surprisingly recent development. In Linux it is one of the things that happened in the 2.4/2.6 timeframe, which is the same timeframe when mainstream distributions started to be too bloated for lot of old or underpowered hardware.
Interestingly, the SparcStation10 would run Solaris 9 and premier support for the SS10 only ended in June 2016, even though the product EOLed in 1994 and had end of support in 1999.
I'm not saying that SunOS was somehow slow to implement these things, but that the SunOS that ran on the aforementioned SS10 shell server. Which seems about right given the fact that first Solaris with userlevel LWPs was 5.2 (dec 93) and pthread-compatibility layer was introduced in 5.5 (nov 95). And it seems quite plausible that they ran SunOS 4.x and not Solaris.
Yeah that bit doesn't make any sense. If anything the pi would support more users with similar workloads to those being run on the sparcstation in the same fashion (without containers).
> Do you really know this? What's the limiting factor?
Yes, I've got a bunch of RasPi's (originals, v2's, and V3s).
The big issue is I/O bandwidth. All of the meaningful I/O in a server setup goes through one USB 2.1 hub. The second issue is that all of the memory accesses have to traverse the (useless) GPU to get to the CPU.
As designed for a phone, having the CPU be a ride along to the GPU makes perfect sense. The "thing" that is important in a smartphone is the screen, its updates and I/O to and from it. But as a general purpose server that isn't balanced.
We put RasPi servers on the Internet and local network at Blekko for a variety of tasks, and I've used them at home as well for servers. Very successful having them be single purpose servers (a DNS cache, a video streamer, a data collector) and very unsuccessful when they are multitasking multiple 'kinds' of network clients. Tried various configs with EXIM and Dovecot as a mail server, it processes very slowly. (an SSH session on the machine was 10s of seconds between the key press and the appearance of the character on my end, took me back to the worst of times on a timeshared system :-).
But seriously, everyone should try this. Set up a minecraft server, set up a web server, set up what ever you want and watch it. It is easy to set up multiple "stressor" clients with Python. Trace the flow of data from client to network to app to disk to app to network to client etc.
Then do the same thing on a machine with the same numbers and twice the I/O bandwidth (Odroid has a couple of boards with similar specs but better I/O bandwidth as do a bunch of boards in the 96boards.org spec)
It will be time well spent if you're ever asked to get the most out of a system.
RPI's Ethernet adapter is internally connected through USB (you may also see another hub connected to the motherboard hub here).
pi@raspberrypi:~ $ lsusb
Bus 001 Device 005: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. LAN9500 Ethernet 10/100 Adapter / SMSC9512/9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
I see. Does anyone know what the rationale for that design decision was? I'm comparing to, for example, the Dragino, which is in the same general price range but does not have this, um, "feature".
Possibly because there was nowhere else to hang the port off with that Broadcom SoC. See also the myriad SoCs and devices that use a USB->SATA converter where I really just want SATA (e.g. a little known, obscure device called the Playstation 3).
But, the latest models are quite fast. I've run a variety of microbenchmarks and a RPi3B has twice the memory bandwidth of a Sun V240 (which is many times the machine that a SS10 was.) It has roughly the same single threaded CPU/FPU performance and twice as many cores a the V240. Storage wise the microsd is faster than the SCSI drives you'd find in an SS10. The SS10 ethernet was 10 Mbps. You could add an SBUS adapter to get up to 100 Mbps, but that's it.
Realistically, you would be able to support 10x whatever an SS10 would do. But if you hang your storage off that USB port, then you're likely to be pretty unhappy. But still would be more of a machine than an SS10.
In the ideal world? With a copy of the full schematic, a datasheet for the SoC, and source code to the kernel and drivers. Its harder with RasPi because various bits are obscured but many of the drivers are open enough to figure much of it out.
This work was already done in other "Unix" environments like Solaris and BSD (zones and jails)
My personal favorite is SmartOS (based on illumos) it runs zones on bare metal and even has support for running Linux containers (docker)! It does this by wrapping the Linux syscall APIs and translating them. Magic stuff!
Launch a Linux native container and run 'ps' and you will only see processes that you own!
A FreeBSD jail does not emulate or create a virtual machine - it's just a fancy chroot mechanism that produces only the unix processes that actually get run inside the jail.
A jailed httpd does not take up any more resources than the exact same httpd run on the base system.
This makes it extremely efficient and, in fact, allows you to create an even richer multi-user platform than the original one the op has nostalgia for: a multi user unix system where everyone gets to be root.
> (USB is slowly replacing the lighter plug, but most cars still have them.)
And yet, that's just trading for different path dependence annoyances. 5V, 500mA limit (or proprietary higher-voltage hackland), and that feeling of connecting your valuable phone to a low-bid switcher? Airports, cars, home outlets - I'll choose the appropriate third party device every time! I just wish that car power socket wasn't so dildonic so that more outlets would come already built in.
217 comments
[ 3.1 ms ] story [ 240 ms ] thread[0] https://kubernetes.io/docs/concepts/cluster-administration/n...
You can simply turn off privileged ports if you feel like it. Or do things like setuid such as how Apache starts as root but switches to userspace immediately.
It's not ideal, but to say the privileged port thing is an issue is bizarre to me. It's the least interesting item the author brings up in the article, and certainly was not the primary driver behind multi-tenancy virtualization.
Better yet is using reverse proxies - service providers have been doing this for ages. A single shared HA cluster of haproxy, that maintains records for each individual application/tenant that lives wherever it likes. This is the model I prefer, since it allows me (as a sysadmin) to protect the developers from themselves by being able to easily filter at layer 7 in front of their application as needed. Also lets you direct traffic easily, and is generally the model used by any container orchestration service.
Very easy to expose all that via various APIs and tooling so users can self-service.
Edit: I didn't realize how new that was. Kernels 4.11+ only. I think some people were using this on custom patched kernels though because I've been seeing it around. Was committed in January.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
> setcap 'cap_net_bind_service=+ep' /path/to/executable
huge blog post for what is literally a one line fix
Therefore, Linux simply doesn't allow it.
then all users benefits from it.
I think that this can be achieved with network namespaces and some iptables magic, entirely in userspace (and without the whole burden of containers).
Anyway, IMO the point of containers is not network virtualization, but rather isolation of dependencies. It's much easier to just pack everything into a container than to invent proper non-root package manager (like Nix).
What leads to... It's better to add non-root mounting into the author's list. I don't think there's even anything that needs to change, permissions are already there.
Really, what would the problem be with just deploying an nginx instance that listens on 80 and routes by domain?
I really like this idea, it's really elegant.
What I want to know is why can't we get all the major web browsers to look for (and honor) SRV records, so that www.mydomain.com can transparently have the browser connect to port 30338 (or whatever port my app is listening on). Then we dispense with the need for proxies altogether.
The point of the article as I understand it, isn't so much about the possibility of shared web hosting - it's obviously possible. It's more about how early design decisions shape our paradigms and the way we understand computing and the necessary resources.
Guix and Nix. I haven't used Nix, but Guix even allows ad-hoc containers running only specific programs and their dependencies:
https://www.gnu.org/software/guix/manual/html_node/Invoking-...
That doesn't help with the privileged port problem, but per-user services can be dealt with.
No. Reproductive genes are well-known for what evolutionary biologists (such as myself) call positive selection, meaning they tend to change more often than you would expect.
See, for example, https://www.nature.com/nature/journal/v403/n6767/full/403304...
The testes are even more protected in quadrupeds, far away from any frontal fighting or sparring. And if you're a quadruped that is being chased by something that can damage your testes from behind, you have far bigger problems than damage to your testes.
For a datacenter isn't it more efficient to pack workloads across dense machines than to over-provision idle ones?
While a factor, maybe, I don't think privileged ports are causing excess CO2 production.
Although I've been interested in low-power edge networking with uni-kernels, would be an interesting project to work on (thinking micro-dc's with homogenous ARM-based boards at the edge and having a dynamic system to boot up uni-kernels on demand closest to the requesting user... a system that mostly stays off unless it's needed).
In this case Ethernet + TCP/IP beat OSI.
Also, JavaScript beat any number of sensible scripting languages. Houses are frequently constructed such that they are not very serviceable.
TCP/IP didn't win by being first.
At the time there were multiple LAN protocols that were mostly used for file sharing, Netware, Appletalk, NetBEUI, etc. You had NFS for TCP/IP but it wasn't really used for anything other than UNIX workstations.
Probably it would have required somebody to write a cut-down OSI stack for MS-DOS that could be linked to a particular killer application (whatever that was, maybe FTAM).
This would still have had to compete with the way that TCP/IP was able to swallow up the other LAN protocols, we wouldn't have had to go through the IPv4 to IPv6 migration though.
Everything in UNIX is a file, apart from the things that aren't. If ports were files, you'd be able to chown them or put them in a group for delegation purposes.
[0] https://www.webfaction.com/
They get around the port issue by using a nginx frontend that uses the Host to route each request to the correct apache instance each user is running out of their home directory. All these apache instances are bound to higher unprivileged ports.
Hah, hundreds, if not thousands of accounts can be on a single shared server. 'Unlimited hosting for $5/month' requires you to pack em in.
> [...], it pushes about 5-10 megabits of traffic most of the time but the software it runs only needs about 10mb of RAM and less than a megabyte (!!!) of disk space. It also utilizes less than 1% of available CPU.
> The virtual machine it runs on, however, occupies about 8gb of disk and at least 768mb of RAM. All that space is to store an entire CentOS Linux base installation, [...]
The exo-kernel and uni-kernel people have the opposite idea: go all the way to virtualization, and eliminate the traditional operating system.
You can still have all the conveniences of the programming models you like, but eg file-systems and network protocols will be implemented in user level libraries.
Later, once everyone has a hypervisor in their pocket, we tell our children about how cloud infrastructures used to be so expensive that only big corporations could maintain them, and they came in server racks the size of refrigerators.
Combined with a sandboxed-by-default capability security model, I think that's the best approach. You no longer need virtualization or users or containers- you just assign hardware and software resources to applications, following the principle of least power. You not only still have the same convenient programming models, but the same (or arguably better) monitoring and management tools.
Instead of syscalls the libraries do the actual work of talking to the hardware.
Basically what some of us were doing when programming 8 and 16 bit computers 30 - 40 years ago.
https://tools.ietf.org/html/draft-andrews-http-srv-01 https://tools.ietf.org/html/draft-jennings-http-srv-05
Unfortunately, HTTP is just too widespread of a protocol - you would end up having to listen for legacy clients on 80/443 forever, making it a nonstarter.
It's true that you have to wait a long time, but that's no reason to avoid it. SNI wasn't widely supported initially, for example, but these days you can safely require it unless you have weird requirements.
There probably isn't enough need for it to get the major vendors to start adopting it, though.
I did never try resolving HTTP services anyway, so I may be completely wrong. I've had problems with TXT and MX.
XMPP is an example of a protocol that currently has this correct, in that it uses SRV records to control ports for a given host.
* http://jdebp.eu/Softwares/djbwares/qmail-patches.html#any-to...
* http://jdebp.eu./FGA/dns-srv-record-use-by-clients.html
In reality, the number of back-end DNS queries to perform an A or an AAAA lookup can vary enormously, and can number in the tens or hundreds of queries already. It's not generally true that the number of back-end lookups will increase, because the number of back-end lookups varies wildly by time (depending from what is cached from momement to moment) by domain name (depending from the amount of gluelessness) and by country. There is so much variation in there already that the variation caused by a two-step SRV lookup can quite often be lost in the noise.
Of course, SRV lookups do not even have to be two-step in the first place. A proxy or content DNS server is free to add the requisite A and AAAA resource record sets as additional section data in its response, meaning that the client obtains both pieces of information in a single transaction. And of course several real world DNS server softwares do exactly this.
* https://news.ycombinator.com/item?id=8850302
Whenever you see these objections, always remember that there are protocols where SRV resource records have been in use for coming up to twenty years. And yet one never hears of actual problems with those protocols that mirror the supposed projected problems of using SRV lookups for HTTP. A lot of what one hears about why this would not work is simply bad analysis and excuse making.
The worst part perhaps is that the code to do this for one of the WWW browsers was actually written, the only actual technical niggles were solved 17 years ago, and -- perhaps most tellingly of all -- whilst people are still telling us how this cannot be done all these years after the code was written, there are parts of the world that are quietly and happily doing it right now. The FreeBSD packaging system uses SRV resource records and HTTP/HTTPS, for example. It's also an example of how real world DNS servers can and do send all of the data in one transaction using the additional section. They've even made it in-bailiwick.
Also, you as the site operator can't really do anything about how recursive resolvers handle additional section data (and there are millions of them deployed on crappy home routers with probably equally crappy DNS software), in contrast to out-of-zone glue, which is something completely under your control.
That being said, I don't think it makes sense to not standardize it for mandatory client-side use with HTTP, (a) because network latencies are not exactly increasing, even mobile networks are moving towards single-digit millisecond RTTs, and (b) browsers could just be backwards compatible, and issue SRV, A, and AAAA queries for the URI domain name at the same time, so you would only need one round trip to figure out the addresses of sites that don't want to use SRV, and you'd only get a negligible amount of traffic for one additional DNS request without any additional latency.
No, it does not. I just explained that to you, with examples even.
* http://jdebp.eu./FGA/dns-srv-record-use-by-clients.html
That way, compliant browsers can connect directly while SRV-ignoring browsers would just get proxied (or redirected), and once set up there would be no special actions needed by the users, save for telling the proxy about the keys and certs.
So, in the article's instance, the machine owner needs to set up a web server on default port 80 that redirects to the correct web server on the correct non-privileged port.
"You must post with the original title, or we'll change it"
"The original title was bad, so we changed it"
At this point, why even give the option to give a title at all?
So, it seems like it was intended to do these things going back to the 70's. They also designed insecure, bloated solutions that INFOSEC founders like Paul Karger called them out for. Resulted in better designs like KVM/370, KeyKOS, VAX VMM, separation kernels, and recently mCertiKOS. Most peddling virtualization for multiple users still use bloated, untrustworthy components though.
Similarly:
> it pushes about 5-10 megabits of traffic most of the time
Bits are a unit of information, not flow. Probably the author meant Mib/s.
And I'm not against pedantry in the right context, but this is just a casual, relatively nontechnical rant. Pedantry is really not needed.
But while I would accept that from a network engineer, I don't know what background the author has.
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080
exit 0
Simple enough to add to a configuration management workflow. I used to maintain a configurable reverse proxy, until I decided that was way too much work and to only put one web facing app per server.
edit: rc.local may not be the best place to put these rules, check this link:
http://bencane.com/2011/12/30/when-its-ok-and-not-ok-to-use-...
This seems to be a common misunderstanding of the blog post. Yes, there's a ton of ways for root to create privileged ports, and a ton of ways to delegate it in various ways. But all of them require actions by root, and sysadmining, and aren't as secure as a system designed to work this way from the beginning would be, so nobody uses them, so in terms of addressing his discussion points, they might as well not exist.
Because this iptables hack can be built right into a custom Linux distribution.
SRV solves this problem nicely, unfortunately SRV support for HTTP 2.0 was rejected.
The author gets a lot wrong, it isn't uncommon because most software people see a "server" as a set of numbers "A" bytes of memory, "B" bytes of disk space, and "C" hertz of clock speed.
Computers aren't numbers of course, they are systems, and the systems are a collection of moving parts that are connected by linkages which constrain and amplify the various parts. In system design there is a concept of 'balance' which discusses how the linkages enable and constrain such that an understanding of the amount of "work" that can be done by the system is understood.
There is a great example in the article comparing a Raspberry Pi to a Sparcstation 10. Most of the software the author used "back in the day" is still around in one form or another, and a RasPi is cheap, so its easy to create modest recreation of the environment from that time and to build simulated users who can access the "server". Just two or three users and the Pi will "choke". Understanding how a system with so many "bigger" numbers is less capable of doing the same "work" as one with much smaller numbers is a useful exercise to run if you are into systems analysis. The system the Pi was designed to run, and the one for which it is fairly well balanced is a smartphone. Something the SPARC 10 would truly suck at.
The clever idea however is making networking resources just another resource like disks. Interestingly enough, with IPV6 that is much easier than it was before (to the point about how things were done before in networking, vs now). It is straight forward to create a model of network interfaces that is equivalent to serial lines. Have the kernel get set an IPV6 subnet with a block of 32K addresses and let your user program open /dev/net/<1> through <32766> because its IP each "net" device comes with its own set of port numbers etc. IPC is just networking and a cleverly coded kernel running on a machine with a 64 bit virtual address space would do direct data placement. No need to do even any page table mashing.
Do you really know this? What's the limiting factor?
I've never played with a Pi and I don't really know, but I don't see offhand why it would be so bad. Interrupt-handling and context-switching times should be much better. There's plenty of main memory. The file system should be faster, running on flash. I don't see where the bottleneck would be.
But on the software side there is one huge problem: today's Linux kernel and userspace is incredibly bloated in comparison to early 90's SunOS. On the other hand, said SunOS probably did not support many things that everybody expects today, like scalability to large-ish SMP systems (ie. >2 CPUs), shared libraries, loadable kernel modules, threads...
Anyway, how much I/O do you need to support people running terminal apps over Ethernet?
But then, it had to, because SPARC CPUs have been no match to X86s for almost as long.
In fact, the NUMA support in non specialized systems (one would want to say "non-IRIX") is surprisingly recent development. In Linux it is one of the things that happened in the 2.4/2.6 timeframe, which is the same timeframe when mainstream distributions started to be too bloated for lot of old or underpowered hardware.
It definitely supported M:1 and M:N threading models while Linux was still designing NPTL and LinuxThreads models.
The big issue is I/O bandwidth. All of the meaningful I/O in a server setup goes through one USB 2.1 hub. The second issue is that all of the memory accesses have to traverse the (useless) GPU to get to the CPU.
As designed for a phone, having the CPU be a ride along to the GPU makes perfect sense. The "thing" that is important in a smartphone is the screen, its updates and I/O to and from it. But as a general purpose server that isn't balanced.
We put RasPi servers on the Internet and local network at Blekko for a variety of tasks, and I've used them at home as well for servers. Very successful having them be single purpose servers (a DNS cache, a video streamer, a data collector) and very unsuccessful when they are multitasking multiple 'kinds' of network clients. Tried various configs with EXIM and Dovecot as a mail server, it processes very slowly. (an SSH session on the machine was 10s of seconds between the key press and the appearance of the character on my end, took me back to the worst of times on a timeshared system :-).
But seriously, everyone should try this. Set up a minecraft server, set up a web server, set up what ever you want and watch it. It is easy to set up multiple "stressor" clients with Python. Trace the flow of data from client to network to app to disk to app to network to client etc.
Then do the same thing on a machine with the same numbers and twice the I/O bandwidth (Odroid has a couple of boards with similar specs but better I/O bandwidth as do a bunch of boards in the 96boards.org spec)
It will be time well spent if you're ever asked to get the most out of a system.
Even Ethernet and 802.11 wifi packets?
https://raspberrypi.stackexchange.com/questions/46076/soc-cp...
Ah. Dragino uses an Atheros SoC. Not the first time I've seen a Broadcom chip have an...interesting feature.
https://www.element14.com/community/servlet/JiveServlet/show...
But, the latest models are quite fast. I've run a variety of microbenchmarks and a RPi3B has twice the memory bandwidth of a Sun V240 (which is many times the machine that a SS10 was.) It has roughly the same single threaded CPU/FPU performance and twice as many cores a the V240. Storage wise the microsd is faster than the SCSI drives you'd find in an SS10. The SS10 ethernet was 10 Mbps. You could add an SBUS adapter to get up to 100 Mbps, but that's it.
Realistically, you would be able to support 10x whatever an SS10 would do. But if you hang your storage off that USB port, then you're likely to be pretty unhappy. But still would be more of a machine than an SS10.
How would one go about doing that?
My personal favorite is SmartOS (based on illumos) it runs zones on bare metal and even has support for running Linux containers (docker)! It does this by wrapping the Linux syscall APIs and translating them. Magic stuff!
Launch a Linux native container and run 'ps' and you will only see processes that you own!
reference: https://wiki.smartos.org/display/DOC/Home
A FreeBSD jail does not emulate or create a virtual machine - it's just a fancy chroot mechanism that produces only the unix processes that actually get run inside the jail.
A jailed httpd does not take up any more resources than the exact same httpd run on the base system.
This makes it extremely efficient and, in fact, allows you to create an even richer multi-user platform than the original one the op has nostalgia for: a multi user unix system where everyone gets to be root.
And yet, that's just trading for different path dependence annoyances. 5V, 500mA limit (or proprietary higher-voltage hackland), and that feeling of connecting your valuable phone to a low-bid switcher? Airports, cars, home outlets - I'll choose the appropriate third party device every time! I just wish that car power socket wasn't so dildonic so that more outlets would come already built in.