The Microsoft Windows ecosystem has always had a laughable security posture. Many large organisations design the IT where the Domain Controller and AD is at the heart and center of everything. One ring to rule them all. I find this ridiculous. Welp, all these crises and vulberabilities have surely helped to keep the lights on for my cybersecurity consultancy.
I'll say: Microsoft's solution, for all it's warts and problems, is (currently) the best integrated solution for organisational use. Jamf (for use in macOS/iOS) and a plethora of options for Android (like BlackBerry's ESP and Samsung's Knox), while decent, doesn't have that integrated feel like Microsoft's. RedHat's and SuSE's options are good on Linux, but it still feels clunky, even considering the clunkiness of Microsoft's tools.
The best contender is ironically Google and Chromebooks, but I feel that they still need to improve their solution though.
If you can recommend a core banking ops solution and a loan origination software for banks under $1B in assets that doesn't require Windows I will gladly go around to all my clients and sell it. Something comparable to what Jack Henry, Fiserv, and Finastra are hawking.
Small businesses can get away with doing everything on websites and a gmail account. Large businesses can 'just' make their own stuff. Middle market businesses that aren't selling software are all trapped on Windows by the software they can buy.
If you're running a single forest with/or a single AD/domain I'd argue that Microsoft has done nothing wrong. Separation of concerns has been a thing for decades now. And no, I'm not suggesting Microsoft's stuff is above reproach.
For what it's worth, Server Core installs don't have Print Spooler installed, and that's what Microsoft currently recommends as a domain controller. But since it doesn't have a desktop UI, it's a bit less liked by smaller entities. And of course, not all Windows Server roles support being run that way.
It's safe to disable Print Spooler on servers that aren't print servers, so I expect that to be a standing recommendation in the future. The other thing though is this vulnerability works against desktop PCs, which need the Print Spooler to print... However, you can set a group policy to disable remote printing to a desktop PC, which should block the flaw. Desktop PCs probably shouldn't be acting as print servers, so this is mostly a good idea to implement too.
So the available mitigations/attack surface reductions available here are plentiful... and honestly probably should just be best practice now.
I used to do consulting work, and there's many small businesses that have "the server" that does everything for them. Microsoft used to even sell "small business server" for this role. I think nowadays they're pushing those customers to the cloud instead, but it takes some companies a long time to upgrade.
Microsoft literally sells an entire line of Server products ("Server Essentials") that are designed to be a combined AD/Printer/File Share/DNS/DHCP/etc server.
Selling a product to a small business them jumping all over that small business for using it that way seems pretty wrong to me. Larger businesses can and likely should do better (e.g. isolate printers on their own VLAN?).
Ultimately blaming end users is rarely constructive. Frankly the Windows Printer Spooler is using 1990s security thinking in 2021, and that's on Microsoft.
> Microsoft literally sells an entire line of Server products ("Server Essentials") that are designed to be a combined AD/Printer/File Share/DNS/DHCP/etc server.
WOW... Since when small business needs AD ?? :) DHCP, yes but DNS ? "Small business" so no DNS internaly needed so obviously Internet facing... And that do as file share and printer server ?? "Only in Microsoft" :>
Becouse MS OSes do not have real ability to install/uninstall basic system components like eg. printer spooler...
And how many times now MS spooler was RCE or admin rights source ??
I hope this issue is the motivator Microsoft needs to finally do something about their awful printing stack. Aside from taking it out of SYSTEM scope(!), the whole thing needs and has needed substantial modernization for a long time now (and hiding parts of it using the new Settings app is a very weak abstraction).
Microsoft like a lot of big techs needs to start rewarding people for maintaining/updating legacy stuff, rather than promoting/rewarding people for new shinies while the technical debt grows out of control. The printer stack which has barely received an update since Windows XP is a perfect example.
I mean, they seriously done many things in Windows 10 (and analogous servers) to improve security without too much breakage, like moving font processing from kernel-level to user-level (n.b. in the process, they have thrown Adobe code). Print spooler is tricky though: even if the official design of the API theoretically allows it to run user-level, ask users of HP products (including business ones) how many times their computer crash due to weirdness in HP drivers. (Why does the HP printing drivers run in kernel-level mode? I don't know!) I'm pretty sure that nothing will happen when the spooler is moved to user-side processing, right? (In all seriousness, the drivers will break and probably crash the system. These products are officially out-of-support, so short of Microsoft paying HP to revamp the drivers nothing will happen.)
In all probability, probably even Microsoft is contemplating such move but breaking a lot of stuff is reputationally damaging to them, even if it's not really their fault in this case. Removing SafeROM is easier because it's just grumbling gamers who can probably patch up Windows 7, but imagine the support calls when the print spooler, a critical component, is moved. Even when a rational mind would put the blame on HP (et al., it's not just HP designing "weird" drivers), the problem will appear due to a Microsoft change and therefore Microsoft gets the blame.
It's getting to the point where a sane mitigation would be to spin up a Windows 10 instance in Hyper-V, running the print spooler and driver in that, and using USB/ethernet passthrough!
Some might react and say "just move to Linux (even only the servers!)" and while I'll say NeXT/Apple's design of CUPS is great, it's harder to get a good Linux printer driver that's designed for business use (HPLIP is... painful, and I'm pretty sure that's out of most support contracts).
De-install HPLIP and use "IPP Everywhere" if you can. Since I've done that I've had reliable printing that doesn't break on package updates. Given that I use Arch, that was an occasional surprise finale to an update session. It usually started with my SO having an urgent print job to do.
Also, put your preferred paper size in /etc/papersize if you are not letter size inclined.
xerox color machines with fiery controllers aren't that well working with ipp, the last time I worked with them (3 years ago) it crashed pretty often, which meant a full reboot was needed which takes like 5+ minutes on these which was not possible. lpd is still super simple and works super well on these machines, even when using non standard ports
Why even running a Windows 10 instance in that case? Esp. if it's for backward compat with old crap; run a Win 2k, it will be lighter. And only marginally less maintained, which does not even matter if you isolate like that anyway.
It wasn't hugely advertised, but Win 7 had something called "XP Mode" where it would literally start windows XP in a VM completely in the background, and any program you wanted to start would actually run inside that VM, with windows drawing the application window like a native Win 7 app. It was magical in how well that worked. And yeah, MS just provided a special XP image for this, you didn't need a separate licence for it.
> Printer drivers haven't run in the kernel since NT 4.0.
I wish that's true, and the APIs are there (and should be used!), but except for incompetence I don't know how some drivers still have that kernel-side code (unless DRM? And if it is, why?)
I even thought that the 64-bit and Vista driver reset have banished kernel-level drivers, but in the past 6 months I still see printer drivers with kernel-level components. I just scratch my head why?
(Some of them are for PCI (yes, not a typo) connector, which is reasonable but weird for a printer. Others are running to printer port and USB but the kernel-level components are still there, active according to Device Manager).
Oh I can tell you exactly why, particularly because I've worked with a bunch of ex HP printer devs.
"Hey we should upgrade to the latest standards!"
"Why?"
"Because they are safer!"
"Yeah, but it might break things"
"But it's safer!"
"Yeah, but the current shit is working. Don't mess with a working thing".
It's practically a religion there. I interned there years ago, it was a momentous task getting internal people to move from IE 7 to firefox, chrome, or IE8. What I'd done was improve one of their internal tools with an updated javascript library. However, it ran like ass on IE 7 and perfectly fine on pretty much everything else.
Windows does all kind of random things at SYSTEM scope and maybe for most of it, it is a terrible idea; yet they think that somehow VBS will save the world, and do not give a shit about the obvious existing shortcomings of their stack... At one point a virus could exploit a bug in the MS Defender AV engine and ironically execute under SYSTEM by just being scanned (!!) and they reacted by introducing MP_FORCE_USE_SANDBOX, however it seems that instead of quickly extending the usage of that sandbox to every user they... now just spawn the worker process (if the env var is there) but it does not do anything anymore :/
I get that engineers are attracted to working on new shinny things, but at one point not maintaining the old stuff (or even just properly removing!) under the excuse that the new unfinished one is more important is just lack of vaguely competent project management, and overall lack of professionalism. Even maybe of individual contributors: if you do not push strongly to get the shit cleaned-up in priority, you are as guilty as the people directing you to work on other (likely soon to be deprecated and unmaintained too) things.
Not wanting to defend MS employees here, but maybe they are paid based on the new shiny things they produce, not based on fixing old stuff. KPI forever
I found this funny because a large portion of my job has morphed into designing reports for different departments to measure performance. It boils down to a lot of binary yes/no formulas in excel and measuring those yes/no as a performance indicator.
KPI forever indeed. I never expected as a sysadmin I’d be creating excel reports to measure the performance of others.
Lol this is getting ridiculous. I don't know why they stopped working on their Defender architecture, the sandbox seemed a good idea. Maybe they have been told to work on Win11 VBS processor requirement wankery instead, leaving that class of problems unfixed for years.
I think they might be hoping that printing, and print spooling, might be going away.
In the future, applications might be able to communicate directly with a network printer via TCP/HTTP. The printer itself does spooling, queueing, management, etc. The OS doesn't need to be involved at all. "Legacy Printer" support can be disabled by default.
If you need to change ink cartridges you either do it on the screen of the printer, or install the manufacturers tool which also just communicates over TCP/HTTP and doesn't use any special interfaces.
>The printer stack which has barely received an update since Windows XP is a perfect example.
The big change was the shift to XPS based drivers in Vista (which was back ported to XP). The trouble is the printer manufacturers have largely ignored this and continued to produce GDI based drivers. I sometimes wonder if Windows has too much backwards compatibility.
Allegedly Vista was going to 100% force XPS drivers, but because they dropped backward compatibility in the network and sound stacks they got a lot of push back that they were crossing too many backwards compatible lines and the printer companies were the ones with the most skin in the game (and cash warchests from ink cartridge sales; and yet the least incentives to update old drivers with better software) so they backed off.
Appearances are they tried to break printer driver compatibility again in Windows 8 (which had some smart ideas on how to modernize the print stack but never got off the ground with them because no drivers adapted to the new stack; and then Windows 10 dropped most of what made the new stack special including not just moving everything into user space but moving everything into app sandboxes; no more annoying printer "background app" notifications and ads would have been amazing) and got worse pushback from the entrenched interests.
You would think Microsoft would have enough might to push back on that. Which printer manufacturer is going to sell a printer without drivers for the dominant operating system.
Unfortunately, it would be the consumers that suffer if Microsoft did push back. Since somewhere in the 90s consumer printer drivers have been as much about DRM for ink cartridges as anything else. Microsoft isn't allowed the keys to the DRM so they can't make generic drivers for old hardware. The printer companies would love an excuse to force everyone off old hardware to new hardware with fancier DRM.
Also, in terms of naming and shaming specific manufacturers, HP/HPE alone has already been known to hold entire industries hostage to old (unsupported) versions of Windows or the various whims of their AIX/HP-UX Unix-based OS efforts rather than support up-to-date drivers on Windows for printers and plotters (and medical equipment and other electronics). It really is easy to imagine them telling the same bad infosec advice to consumers if Microsoft were to break compatibility in their printer drivers. No one wants the chaos of supporting their family members trying to stick on an ancient unsupported Windows XP version simply because their reliable old inkjet printer told them to. Sure everyone thinks they want the Year of Linux on the Desktop, but no one actually wants to help a friend dual boot into HP-UX, and debug cups dump output for them, just to print something twice a year.
Have any printers had the DRM in the driver rather than the printer firmware?
My experience working with printer manufacturers is most of them would love to not have the responsibility of writing printer drivers. If they could offload the work to MS they would be over the moon.
I think we'd all benefit if Microsoft had more direct control over legacy print drivers. Printer manufacturers seem some of the worst offenders in bad driver software in general.
I also assume that if Microsoft had the ability to create more generic printer drivers they would. Given the huge amount of older legacy hardware that falls under any of the other Generic Class Drivers it kind of sticks out as a sore thumb how few printers have generic drivers. (IIRC, there's a Generic PostScript driver that works with early [Apple led era] laser printers back when PostScript was new and considered sufficient as the only language between the computer and the printer, and nothing recent.)
My understanding is that most inkjet printers especially speak OEM proprietary languages that's not just PostScript or PDF (or Microsoft's mostly failed dream of an open XML format with fewer patent hangups than PS/PDF they tried in XPS). You would assume some of those proprietary languages include commands at least ancillary to the cartridge DRM (initiating checks, validating DRM status codes and checksums, things like that) if not directly involved. DRM is the main reason to blame these proprietary languages for remaining proprietary and not well standardized across/between manufacturers (and thus don't leave us with enough generic OS printer drivers today), though it's probably not the sole reason for the proprietary languages. I'm sure the manufacturers also have other reasons for keeping their printers using proprietary languages such as not wanting to license PS/PDF or leak too many low level details of their hardware designs (beyond just possibly leaking DRM information) or patented "secret sauce" of their own languages they believe to be a competitive asset.
Microsoft's XPS push was one attempt to get them off proprietary languages and allow them to offload far more driver work to Windows directly. Seems a shame none of the printer manufacturers seemed to take them up on that.
To quote from that: Bill Gates' directive that Microsoft emphasize security features in its product is a major strategy shift, but some experts wonder if there's anything behind the words.
Now, 19 1/2 years later, Bill's directive will finally be taken seriously? Maybe once we hit 1,000,000 "critical" security flaws in Microsoft products? How many more before we hit that number? Does anyone even bother to count?
Can you people stop submitting this fraud reporter's content on here? The guy doxxed security researchers working under pseudonyms when they criticized his bad articles. This guy should be yeeted into the sun, not voted atop HN for every piece of blog spam.
I don't know if the current article is good, but the site has certainly had some good articles. If we were to judge every site by the worst thing that had appeared on it (or, for that matter, every person by the worst thing they've done), there wouldn't be much left. That's a strategy for a much less interesting world, so it violates the prime directive of HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
So when I try the registry key workaround, it doesn't work, because my registry doesn't have a Printers section. Which is logical, because the last time I owned a printer was long before I bought this machine. Does that mean I don't have to worry, or that I'm wide open to attack? All the discussion so far seems silent on what happens if you don't have a printer. (This seems common to a lot of discussion about 'security vulnerability in Foo' - are you exposed if you don't have and have never used Foo? It doesn't say.)
No. Should I? Is it the case that my machine is exposed without them, and creating them is necessary to solve the problem? Or that it's fine the way it is now, and creating them would cause the very problem I would be hoping to solve?
By way of verification, not only did I double check the path in regedit, but I also tried copy pasting the command line from the Microsoft page, and that gave path not found.
>Does that mean I don't have to worry, or that I'm wide open to attack?
If it's windows client the printing feature is most certainly enabled, so yes you're open to attack. You can think of the registry as a global config file. If a given section is missing, the program will just do whatever the default is. The same applies for windows. A section (path) missing doesn't mean the feature isn't enabled.
For those of you with Windows networks under your belt, I stayed up late writing a PowerShell script to tackle the install with some reporting capabilities.
As of this morning the darn hotfix isn't available from standard Windows Update app, and the out-of-band files are unique depending on Windows release, so this factors that in.
65 comments
[ 3.1 ms ] story [ 124 ms ] threadThe best contender is ironically Google and Chromebooks, but I feel that they still need to improve their solution though.
If you can recommend a core banking ops solution and a loan origination software for banks under $1B in assets that doesn't require Windows I will gladly go around to all my clients and sell it. Something comparable to what Jack Henry, Fiserv, and Finastra are hawking.
Small businesses can get away with doing everything on websites and a gmail account. Large businesses can 'just' make their own stuff. Middle market businesses that aren't selling software are all trapped on Windows by the software they can buy.
I've heard anecdotal claims that Amazon has a large internal AD.
It's safe to disable Print Spooler on servers that aren't print servers, so I expect that to be a standing recommendation in the future. The other thing though is this vulnerability works against desktop PCs, which need the Print Spooler to print... However, you can set a group policy to disable remote printing to a desktop PC, which should block the flaw. Desktop PCs probably shouldn't be acting as print servers, so this is mostly a good idea to implement too.
So the available mitigations/attack surface reductions available here are plentiful... and honestly probably should just be best practice now.
Microsoft literally sells an entire line of Server products ("Server Essentials") that are designed to be a combined AD/Printer/File Share/DNS/DHCP/etc server.
Selling a product to a small business them jumping all over that small business for using it that way seems pretty wrong to me. Larger businesses can and likely should do better (e.g. isolate printers on their own VLAN?).
Ultimately blaming end users is rarely constructive. Frankly the Windows Printer Spooler is using 1990s security thinking in 2021, and that's on Microsoft.
WOW... Since when small business needs AD ?? :) DHCP, yes but DNS ? "Small business" so no DNS internaly needed so obviously Internet facing... And that do as file share and printer server ?? "Only in Microsoft" :>
Becouse MS OSes do not have real ability to install/uninstall basic system components like eg. printer spooler...
And how many times now MS spooler was RCE or admin rights source ??
Microsoft like a lot of big techs needs to start rewarding people for maintaining/updating legacy stuff, rather than promoting/rewarding people for new shinies while the technical debt grows out of control. The printer stack which has barely received an update since Windows XP is a perfect example.
In all probability, probably even Microsoft is contemplating such move but breaking a lot of stuff is reputationally damaging to them, even if it's not really their fault in this case. Removing SafeROM is easier because it's just grumbling gamers who can probably patch up Windows 7, but imagine the support calls when the print spooler, a critical component, is moved. Even when a rational mind would put the blame on HP (et al., it's not just HP designing "weird" drivers), the problem will appear due to a Microsoft change and therefore Microsoft gets the blame.
Also, put your preferred paper size in /etc/papersize if you are not letter size inclined.
... OK I've just tested that against my home HP MFP and 9100/tcp shows in the results.
If lpd does it for you on Xerox printers, then fine: use that. I was discussing HPLIP ie getting rid of a Hewlett Packard thing!
Printer drivers haven't run in the kernel since NT 4.0.
I wish that's true, and the APIs are there (and should be used!), but except for incompetence I don't know how some drivers still have that kernel-side code (unless DRM? And if it is, why?)
I even thought that the 64-bit and Vista driver reset have banished kernel-level drivers, but in the past 6 months I still see printer drivers with kernel-level components. I just scratch my head why?
(Some of them are for PCI (yes, not a typo) connector, which is reasonable but weird for a printer. Others are running to printer port and USB but the kernel-level components are still there, active according to Device Manager).
"Hey we should upgrade to the latest standards!"
"Why?"
"Because they are safer!"
"Yeah, but it might break things"
"But it's safer!"
"Yeah, but the current shit is working. Don't mess with a working thing".
It's practically a religion there. I interned there years ago, it was a momentous task getting internal people to move from IE 7 to firefox, chrome, or IE8. What I'd done was improve one of their internal tools with an updated javascript library. However, it ran like ass on IE 7 and perfectly fine on pretty much everything else.
Windows does all kind of random things at SYSTEM scope and maybe for most of it, it is a terrible idea; yet they think that somehow VBS will save the world, and do not give a shit about the obvious existing shortcomings of their stack... At one point a virus could exploit a bug in the MS Defender AV engine and ironically execute under SYSTEM by just being scanned (!!) and they reacted by introducing MP_FORCE_USE_SANDBOX, however it seems that instead of quickly extending the usage of that sandbox to every user they... now just spawn the worker process (if the env var is there) but it does not do anything anymore :/
I get that engineers are attracted to working on new shinny things, but at one point not maintaining the old stuff (or even just properly removing!) under the excuse that the new unfinished one is more important is just lack of vaguely competent project management, and overall lack of professionalism. Even maybe of individual contributors: if you do not push strongly to get the shit cleaned-up in priority, you are as guilty as the people directing you to work on other (likely soon to be deprecated and unmaintained too) things.
KPI forever indeed. I never expected as a sysadmin I’d be creating excel reports to measure the performance of others.
https://twitter.com/taviso/status/1412809764363411457
In the future, applications might be able to communicate directly with a network printer via TCP/HTTP. The printer itself does spooling, queueing, management, etc. The OS doesn't need to be involved at all. "Legacy Printer" support can be disabled by default.
If you need to change ink cartridges you either do it on the screen of the printer, or install the manufacturers tool which also just communicates over TCP/HTTP and doesn't use any special interfaces.
https://www.microsoft.com/en-ww/microsoft-365/windows/univer...
The big change was the shift to XPS based drivers in Vista (which was back ported to XP). The trouble is the printer manufacturers have largely ignored this and continued to produce GDI based drivers. I sometimes wonder if Windows has too much backwards compatibility.
Appearances are they tried to break printer driver compatibility again in Windows 8 (which had some smart ideas on how to modernize the print stack but never got off the ground with them because no drivers adapted to the new stack; and then Windows 10 dropped most of what made the new stack special including not just moving everything into user space but moving everything into app sandboxes; no more annoying printer "background app" notifications and ads would have been amazing) and got worse pushback from the entrenched interests.
Also, in terms of naming and shaming specific manufacturers, HP/HPE alone has already been known to hold entire industries hostage to old (unsupported) versions of Windows or the various whims of their AIX/HP-UX Unix-based OS efforts rather than support up-to-date drivers on Windows for printers and plotters (and medical equipment and other electronics). It really is easy to imagine them telling the same bad infosec advice to consumers if Microsoft were to break compatibility in their printer drivers. No one wants the chaos of supporting their family members trying to stick on an ancient unsupported Windows XP version simply because their reliable old inkjet printer told them to. Sure everyone thinks they want the Year of Linux on the Desktop, but no one actually wants to help a friend dual boot into HP-UX, and debug cups dump output for them, just to print something twice a year.
My experience working with printer manufacturers is most of them would love to not have the responsibility of writing printer drivers. If they could offload the work to MS they would be over the moon.
I also assume that if Microsoft had the ability to create more generic printer drivers they would. Given the huge amount of older legacy hardware that falls under any of the other Generic Class Drivers it kind of sticks out as a sore thumb how few printers have generic drivers. (IIRC, there's a Generic PostScript driver that works with early [Apple led era] laser printers back when PostScript was new and considered sufficient as the only language between the computer and the printer, and nothing recent.)
My understanding is that most inkjet printers especially speak OEM proprietary languages that's not just PostScript or PDF (or Microsoft's mostly failed dream of an open XML format with fewer patent hangups than PS/PDF they tried in XPS). You would assume some of those proprietary languages include commands at least ancillary to the cartridge DRM (initiating checks, validating DRM status codes and checksums, things like that) if not directly involved. DRM is the main reason to blame these proprietary languages for remaining proprietary and not well standardized across/between manufacturers (and thus don't leave us with enough generic OS printer drivers today), though it's probably not the sole reason for the proprietary languages. I'm sure the manufacturers also have other reasons for keeping their printers using proprietary languages such as not wanting to license PS/PDF or leak too many low level details of their hardware designs (beyond just possibly leaking DRM information) or patented "secret sauce" of their own languages they believe to be a competitive asset.
Microsoft's XPS push was one attempt to get them off proprietary languages and allow them to offload far more driver work to Windows directly. Seems a shame none of the printer manufacturers seemed to take them up on that.
Yes, finally, this will be the defining event that finally gets Microsoft to give a flying fuck about security or about code quality. /s
Anyone remember this stuff circa January 2002?
Gates Finally Discovers Security https://www.wired.com/2002/01/gates-finally-discovers-securi...
To quote from that: Bill Gates' directive that Microsoft emphasize security features in its product is a major strategy shift, but some experts wonder if there's anything behind the words.
Now, 19 1/2 years later, Bill's directive will finally be taken seriously? Maybe once we hit 1,000,000 "critical" security flaws in Microsoft products? How many more before we hit that number? Does anyone even bother to count?
https://twitter.com/malwarejake/status/1127388503166345217
https://twitter.com/ArcayrIRL/status/1121412963351420928
Or, put differently, I don't think his offenses are bad enough to 'cancel' him.
I don't know if the current article is good, but the site has certainly had some good articles. If we were to judge every site by the worst thing that had appeared on it (or, for that matter, every person by the worst thing they've done), there wouldn't be much left. That's a strategy for a much less interesting world, so it violates the prime directive of HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
If it's windows client the printing feature is most certainly enabled, so yes you're open to attack. You can think of the registry as a global config file. If a given section is missing, the program will just do whatever the default is. The same applies for windows. A section (path) missing doesn't mean the feature isn't enabled.
https://www.reddit.com/r/sysadmin/comments/oflbny/windows_pr...
Furthermore, the MS patch doesn't work properly:
https://www.theregister.com/2021/07/07/printnightmare_fix_fa...
As of this morning the darn hotfix isn't available from standard Windows Update app, and the out-of-band files are unique depending on Windows release, so this factors that in.
https://github.com/djcabrera/printNightmareUtility/blob/main...
Hope it helps.