41 comments

[ 3.1 ms ] story [ 97.1 ms ] thread
> That's one more reason not to rely on automated testing only.

I took an alternative conclusion - don't use shadow DOM or other bleeding edge tech when something much simpler will do. What ever happened to using document.getElementById? You can even skip much of this if you finish your work on the server before sending that text/html response...

I sometimes wonder if there's a huge thing I'm missing, but then I remember our customers love our product. Not a single one of them can tell that we don't use react or WASM. In fact, we have yet to receive a single complaint regarding our multipart form posts resulting in actual browser navigation. For whatever reason, the users don't seem to give a shit about this. It only ever seems to come up in the most ridiculous nerd conversations.

Shadow DOM is not "bleeding edge". It's been in all major browsers for years.
shadow don't has always been on the platform the only new big but is an API exposing it. before you just never got to change certain components like select dropdowns
Shadow DOM is frankly more bleeding edge than most new things, in the sense that you’ll cut yourself badly if you try using it much. There are just so many subtle and annoying problems with it because it broke the DOM model that has been around from the start, and now a bunch of stuff is just inaccessible from the DOM. The vast majority of places I see people thinking “Shadow DOM might be suitable here”, it’s a terrible idea. Good places to use it are actually really niche.

Declarative Shadow DOM, more recently added and not yet universally supported, fixes one of the two biggest problems with it (that you have to load and run scripts before it can work), but the other one (that it’s strong insulating encapsulation, which is just occasionally what you want but normally not) is by design and is not going to change.

In the forms they’ve ended up in, Shadow DOM was a mistake (yes, they should have just left browser widgets like <select> and <audio controls> special), and Web Components were almost certainly a mistake.

I've been using shadow DOM almost exclusively for 7 years in very high profile important applications with no incident due to its use. If your code is properly encapsulated you won't have a problem. There shouldn't be any reason for other production components to pierce the shadow DOM boundary.
Since you talk from this angle, here’s an example from a different perspective of how this is still a problem (though I maintain it’s almost always a terrible idea for development/maintenance reasons where people think to use it, too—few components actually want what Shadow DOM gets them):

I’m a user. I want to write user scripts to tweak things to my liking and to support my specific use cases. If you use shadow roots (especially closed ones), you may fundamentally prevent me from doing so, removing the agency of my user agent. I don’t like that.

It's fine to use whatever tools you consider most appropriate for your situation and use case.

The tools you mention have helped enable a ton of new forms of development and interactions on the web.

Consider: I was recently baking a cake and didn't have to use a sous vide for anything, I don't understand why people keep hyping those up so much. That's similar to how your comment reads to me.

> Consider: I was recently baking a cake and didn't have to use a sous vide for anything, I don't understand why people keep hyping those up so much. That's similar to how your comment reads to me.

This analogy seems to work pretty well. On the other side of the fence, I see people talking about how good their sous vide cakes are and how they’d never work at a bakery that didn’t exclusively use a sous vide.

That’s not to say that the cooking method is never viable, just that it’s touted as the only thing necessary. Meanwhile online baking recipes list “make sure you own a sous vide” as the first step.

Where do you see people talking about how good their sous vide cakes are and how they’d never work at a bakery that didn’t exclusively use a sous vide? Who here "touted [it] as the only thing necessary"? In fact, what cakes have you seen that are cooked via sous vide?

Are you sure you understood the analogy? The components of it were:

1. Person says they built a basic website without React, doesn't know why people use React (it's because the use case for React typically isn't "build a basic website given unlimited time and effort")

2. Person says they made a cake without sous vide, doesn't know why people use sous vide (it's because the use case for sous vide typically isn't "bake a cake")

The person you are responding to is embracing the analogy and using it to demonstrate how--if you also agree that the analogy works well--ridiculous it thereby would be how online tutorials for websites (not cakes: that's a the point) make it sound like the way to do anything and the very first technique you should learn in a tutorial is React (not sous vide: that's the point).
But they don't, which is why the analogy works well: just like websites for baking largely don't say "the only way to bake cakes is sous vide", basic website builder sites largely don't say "the only way to build basic websites is React". That's the point.
> basic website builder sites

Assuming you interpreted this from "online baking recipes", I was intending to refer to tutorials that explain something like "how to disable a button" and step 1 is "npm install $POPULAR_LIBRARY".

Tutorials online largely don't say that react is the only way to build a basic website, just like they largely don't say that the only way to bake a cake is sous vide.

That's the point: tutorial sites largely don't do those things.

As for the separate discussion of whether they largely suggest React as a first option for building a given web app, there's no problem with that: it's entirely possible that the best path is to try React first, then look at other, lower-level options only if necessary.

The problem is only if they largely say that's the only option, which they don't, so what's the problem?

> It's fine to use whatever tools you consider most appropriate for your situation and use case.

1000% agreed.

The part where this breaks down for me is the observation that many developers don't seem to have conducted the barest of due diligence regarding what tools and options are even available.

I can accept that you went with ReactJS, but not if you are operating in a vacuum where access to MDN is restricted and you didn't even consider the merits of a "no-build" web ecosystem.

I think it's common for someone who prefers X to believe that another who hasn't chosen X has made an uninformed choice, or not done their due diligence (and of course the person believing this, considers themselves informed, etc). This belief is 1 step away from believing that everyone who didn't choose X is wrong. It's actually pretty likely they did sufficient due diligence, and picked the better option for them.

Personally, I believe if they found something that works, more power to them. In all likelihood, the "no-build" option is likely more time and more work than using frameworks, for the sort of non-basic web apps someone might choose them for.

Doing a comprehensive due diligence of all the possible ways to build something is, imo, the sort of analysis paralysis that stops hackers from just going out and building. Yes, you could become an expert in the full stack, such that you could build an app omitting any or all of them. Or you can build a thing. You're going to throw the first one away anyways.

> In fact, we have yet to receive a single complaint regarding our multipart form posts resulting in actual browser navigation.

Just out of curiosity, how do you handle errors, specifically timeouts here? I'm guessing the HTTP server serves a custom error page, does it have a reference to the previous page with all the data they tried to submit and so on? Probably how I would have handled it.

Timeouts are a bit trickier, as the user would just see the browser error page and dassit.

Nice thing with decoupling server responses from what the browser actually renders is that you can handle things like that exactly like how you (as a developer) want to, without breaking the current page.

But, as always, it depends. Seems you folks are giving your users a lot of value from whatever you're doing regardless, so seems what you're doing is working :) Congrats.

Such error handling is easier, because it’s a normal part of the flow, rather than a separate path that’s not tested. If submitting fails to load the page, you get the normal browser error page and reload, submitting it again. And that’s what you want it to be, unless you can actively do something else in the meantime (which is still a good deal more complex and not typically supported by web apps). By contrast, my experience says that scripting-based submission is very flimsy in the presence of unreliable network. It’s crazy just how many things enter broken, unrecoverable states because it didn’t occur to the authors that maybe that fetch could fail, and maybe you might want to try again. Or worse, and practically never handled properly, when network conditions lead to a stuck request, where the client is waiting for something that will never come. In the browser, you just press F5/Ctrl+R/⌘R when you recognise this situation, but when scripting-powered the only way of fixing this may be briefly toggling the network adapter in order to kill the request (because you can’t just reload the page because you’d lose in-page form state, another of these things that the browser gives you for free but scripting plays havoc with, though Firefox is best at compensating), and hoping that leads the page to let you submit again.

Seriously, scripting-based submission, navigation, &c. is reimplementing a lot of what the browser give you for free, and it can’t be done perfectly (e.g. tab loading indicator) and what is reimplemented is absolutely never as flawless as possible. Yes, you get some interesting advantages, but you lose a lot too, more than developers tend to admit.

>In the browser, you just press F5/Ctrl+R/⌘R

Many users have no idea what does that mean and will just panic seeing the default error page. They will not read the message there and either will abandon the form, reach customer support or will restart their flow. You are absolutely right about the complexity of the form submission when network problems are taken into account, but default browser behavior isn't the best solution.

I suspect (though perhaps I’m naively optimistic) you may be doing the affected users an injustice. Most people will either practically never experience these error pages, or be familiar with them and their causes because they know their internet connection is dodgy at times. And the “Try Again” button’s pretty clear.
You will be indeed very optimistic to assume that generic „Try again“ is clear to everyone. In some cases the effects of it are unclear even to power users and they won’t click it. Let’s say I have got timeout page after clicking the final button in the checkout process: how do I know that it won’t place the order twice if I submit again? How do I know that it will submit it again correctly? After all, those timeouts happen often due to server-side delays.

Today most people do not encounter those pages, because they interact with SPAs and more likely to encounter different error behavior. Good internet connection won’t prevent anything. Imagine a child accidentally unplugging the wifi router and grandma seeing the timeout page, because her browser doesn’t know that wifi is down. What she would do? This is the kind of situation IT support is dealing with regularly.

> how do you handle errors, specifically timeouts here?

We don't.

We are using stateless HTTP semantics throughout. There is no web socket or other streaming abstraction. You can key in a form, come back tomorrow and submit it and the server won't care (assuming business logic checks out).

If the system is down at submission time and someone wants to retry after it comes up? No problem. The browser will give you an opportunity to resubmit the form data since the prior POST timed out. This has been a thing since ~forever.

The end user experience when the service is completely down is that the browser displays "connection to server timed out" with various initial troubleshooting steps that the vendor manages. Either the server is up and everything works, or it is down and nothing does. Whatever partially-available thing we might be going for doesn't make much sense in my head.

I recently launched an app in an extremely saturated space (AI characters, https://tryspellbound.com)

Most of my differentiation is in the UX: specifically things like animations that were carefully selected to make the UI feel snappy even though the AI isn't that quick.

It sounds silly, but there are literally 100 apps doing exactly what mine does. If my site felt like submitting a form, and the UI had native selects, and overall it felt like a site that leaned more on native web technology, the demographic that uses it would have shunned it for other options before giving it a chance.

Sometimes you're in a market where you get the luxury of ignoring looks and just delivering value in a way that doesn't allow people to complain, but other times you have to give the people what they want: and a lot of them want that "it feels like an app" experience.

How many distinct aspects of your product actually require asynchronous submission of form data?

A little bit of javascript/AJAX/etc is perfectly acceptable. We didn't get 100% of the way there with multipart/form because iOS/Safari exists and will crash hard if you try to post big forms.

You don't need ReactJS to submit a user input and avoid a page load. This is ~20 lines of vanilla javascript.

> iOS/Safari exists and will crash hard if you try to post big forms.

I find things like https://stackoverflow.com/questions/10195459/ios-safari-cras... and https://bugs.webkit.org/show_bug.cgi?id=84168, from 2012. Is that still an issue? I find it difficult to imagine you could have a bug that prevents you from submitting a form with even half a megabyte of data being allowed to last for long.

> I find things like https://stackoverflow.com/questions/10195459/ios-safari-cras... and https://bugs.webkit.org/show_bug.cgi?id=84168, from 2012. Is that still an issue?

Yes. It is 100% still an issue. That is the exact bug I was looking at. I can get Safari on my latest gen iPad to crash hard after only 3 submissions pretty much every time. We are sending large images in our forms but it's only like 1-5 megabytes.

Wow. That’s bonkers. That’s not even large images, that’s just regular photos. Upload a few photos the traditional way and your browser crashes? In 2024? Madness.

(I guess the size threshold has increased as device memory has increased.)

The whole experience has been like discovering a dead canary in a coal mine.

This tells me that very few developers are doing things "the old" way or otherwise questioning mainstream technology narratives.

I sometimes find myself wondering if multipart/form is going to be deprecated soon. If you look at the latest Azure Function isolated worker stack, this hasn't even been dealt with yet. I had to grab a 3rd party library to parse these submissions out of my request bodies.

If you have a fancy web socket use case, no problem - we got your back. Want to build a boring LOB business app? Better be ready to start shimming those HTTP post mechanisms.

I have certainly found myself surprised at times in the last decade or more when frameworks and similar have simply not supported multipart/form-data for a long time, given that it’s the only way you can use <input type=file> (since it needs to convey the file name as well, as a header on the part; application/x-www-form-urlencoded has no way of conveying that).
It's not just about web requests: Small things like the select that reflects various font choices correctly, consistently styling form elements as deeply as the site does, the live updating rich formatting inside the main input (shockingly annoying to deal with, even with React), the per-word fade-in effects for text with memoization and other state-dependent optimizations since Chrome chokes on too many spans, handling accessibility for all of these little pieces...

There'd be a lot of "just a little bit of vanilla Javascript and CSS" sprinkled all over that'd all need to have state managed in a manual way that wouldn't be a meaningful improvement over React.

Like I said, I'm working on something where seemingly minute UI/UX details matter to the target user. So I can't just say "CSS and Vanilla JS don't support that without pulling teeth, it's a small thing, so I'm going to ignore it". Sometimes that's going to be a valid strategy and sometimes it's not.

To this day a lot of frameworks and libraries are built with convenient CSS selector string APIs, thinly wrapping querySelector. These don't work well with shadow DOM.

Passing element references is more compatible, but more verbose. And in the case of test framework POMs, a poor choice - because it leads to stale element references in a reactive or virtual DOM.

Some libs like Playwright are re-living the jQuery timeline and building enhanced selector engines that pierce shadow DOM by default.

We're forced into a future where your toolbox either contains shadow DOM or querySelector.

Shadow DOM's adoption problem comes down to ergonomics. We prefer jQuery's vision of the web, grabbing elements with handy one-liners. Shadow DOM broke that. And the practical benefits of shadow DOM have never outweighed the convenience of selectors.

> Some libs like Playwright are re-living the jQuery timeline and building enhanced selector engines that pierce shadow DOM by default.

Our ability to use a testing framework like Playwright without having to resort to a bunch of weird adapter code is a big part of why we went with dumb-simple web technology. Their recording feature "just works" for us and we get perfect replay 100% of the time.

> enhanced selector engines that pierce shadow DOM by default

It’s worth noting, for those that don’t realise it, that from client-side scripting you can’t reliably pierce the Shadow DOM. Playwright must be doing this with privileged APIs.

Specifically, although element.attachShadow({ mode: "open" }) sets element.shadowRoot to the ShadowRoot that it returns, element.attachShadow({ mode: "closed" }) only returns the root, and you can throw that away or store it in a private property or closure variable or whatever, so that it’s impossible to access from outside. Yes, shadow roots give you an inviolable membrane (and one that I don’t think content blocker are piercing yet, because not many sites have been abusing it yet). The only way outside code will even be able to discern that there is a closed shadow root is by trying to create one and having that fail (which is a canary, you can only do it once).

I like the idea of web components, but feels like a mistake to have welded shadow dom to the implementation (or at least the default way they are implemented/described/advertised).

The vast majority of devs I've worked with who try and use them are basically using them like JS frameworks or HTML templates in normal apps.

Shadow DOM is just really a poor fit for this type of use IMHO and results in practical issues like the OP.

Shadow Dom is completely optional and you can leave it out if you don't need an encapsulated DOM at runtime. It's not at all welded to the implementation.
> Shadow Dom is completely optional and you can leave it out if you don't need an encapsulated DOM at runtime. It's not at all welded to the implementation.

That is partly my point then - every time I've run into this with devs, the defaults, docs, comments, etc assume Shadow DOM is needed/important and then things like the OP bite them.

Shadow DOM is amazing - it enables interoperable composition and has much safer encapsulation again random styles.
People over complicate this all the time. Things work as expected or they don't. Things cause interference/regression or they don't. Your tests are not Agile user stories so they should not read like Agile user stories. Your tests should determine if things are broken as directly and clearly as possible without doing more, and they must do so as fast as possible.

At the start, actions from good unit tests in the browser execute user simulated events against DOM nodes the user has access to. That is all, because that is the actual user experience If your testing approach can do this directly, as in without framework magic, it will work the same with your favorite framework and vanilla JS and it will execute much faster.

Then test assertion is as simple as validating changes to DOM nodes, or no changes in the case of negative tests. Such assertions can include the presence and/or visibility of a DOM node, value of text, the size and/or location of a DOM node, presentation, node order, and so forth. User facing events include absolutely everything a user could do with a mouse, touch, or keyboard with only exception to mouse cursor movement.

If that is no clear do nothing else with testing until the above is ironed out. Once the above is solidified more advanced testing can determine whether events/listeners are attached as expected or whether hidden data/metadata is available to the page as expected.

Also, the major common testing tools make use of Selenium styled remote access to the browser via means like CDP or W3C's WebDriver protocol to backdoor a browser. These are really complicated, way more complicated than you need if testing on a site you own. Testing is far more simple, and secure due to same-origin policy, if you conditionally embedded a test receiver script into the page and beacon out test assertion results via WebSocket.

This would be like testing bytecode produced by Python. Why would you look at an abstraction that is not meant to be looked at? Test the actual DOM.
shadow DOM doesn't play well with assistive technology, so you shouldn't really be using it in production right now anyways
I haven't had the same experience. Can you provide an example of this?