71 comments

[ 3.8 ms ] story [ 143 ms ] thread
That is a great read! I like the rather simple root-cause, the bad master data Mel identified and sorted out. Stuff like that is usually covered up by excess inventory, now with all the supply chain issues, those issues disrupt things.

And it shows the one point I always try to drive across: Logistics and Supply Chain is a people business. No amount of ML, algorithms and cooperation / collaboration tools can change that. If things get hairy, pick up the phone and talk to people. Even better, use 21st century tech to automate routine stuff and use the free time to talk and build relationships with people at your supplier or customers.

As someone working in the space, I'd like to respectfully disagree: All players on the market right now, us included, don't focus on replacing, but facilitating human communication by using algorithms. The supply chain is in bad need of a digital overhaul, because things are fundamentally intransparent. Even big companies have no idea where their suppliers came from, what expensive lunch invitation actually got a deal started. With a sudden, global market disruption, they start to notice entire factories stand still if a single, tier-3 supplier is unable to deliver in time. The solution is insights into supply chains, and visibility for suppliers, so people can make business more easily.
Sure, that idea of transparency is around since someone came up with the SCOR model. The reason why I, personally, don't like the whole algorithm and ML things in Supply Chain is that I saw it more often than not coming from people with limited operational experience jumping on the latest bandwagon / hype train. And I have yet to see supply chain modelling that resulted in overall lower lead times and inventory levels.

I am all in for using those tools so, but with discretion. E.g. if optimization software comes up with lower safety stock levels than what you have, use those. If the calculated levels are higher than actual ones, stick with the actual ones.

I highly doubt so, that companies that failed to control their supply chains properly pre-COVID disruption would be able to benefit from increased insight into sub-tier suppliers. Those that do benefit from that insight are those that were already better than most before and will be even better after those insights are gained. One of the reasons nobody seems to be able to catch up to the likes of Apple, Amazon or Proctor&Gamble.

I have always assumed that the big use of blockchain is for supply chain management. If "everyone" had access to the same chain, would that not bring "everyone" up to the level of at least knowing where the stock actually was?

Edit: ok let me show how little I understand or have thought about this.

Its possible for every primary producer to create a new "chain" for each of its products. For factories this is in the bounds of possible. At this point everyone who touches it could use a public / private key hand off to confirm receipt and then append to the chain. I suspect this will hit massive problems that I don't know about, but the principle makes sense. I do not know if the incentive align - but globally the idea of being able to track every single manufactured good across the world seems like it should be profitable.

The biggest issue with blockchain is the need to match the virtual blockchain to the physical goods. Once that link is lost, which happens every day, the blockchain is basically useless. Also, you uave to trust whomever that what the blockchain says is correct. Sure, you can audit these things on site, but then I don't see the benefit of a blockchain.

Also, supply chains are made up of often times competing players. The amount of insight one could gain on other companies from a comolete blockchain is, IMHO, a big hinderance as well.

I think my only answer is ... yes but ...

but the value to every other player in the supply chain is so high, it feels inevitable

Blockchain feels like the right answer to the problem, but it's just infeasible in practice. Who do you trust to handle the blockchain for the entire world? Having a chain that maps the entire global trade will quickly grow to unwieldy proportions - if you need a data center to even store the petabytes of the chain's data, the entire point is moot: You'd again have to trust large vendors. If you instead opted to have smaller chains, perhaps on the goods category, country, or even reseller level, the market would fragment. We'd soon end up with multiple chains claiming different things, rendering the whole endeavour obsolete again.

And we didn't even talk about meatspace issues yet: Things disappear, get stolen, lost, destroyed, dissolve, companies change ownership, move to other locations, bribe, steal, commit fraud, get banned from selling things somewhere... it's a huge mess! Now sure, all of that isn't directly related to a supply chain. But on a global scale, a supply chain is always just an incredibly limited view of a network of goods moving from A to B. To prevent reality in the blockchain drifting apart, you'd need a way to model all of this.

In the meantime, I'd much rather see simpler solutions: Nobody seems to have tried printing QR codes on physical products that contain PrivKey-signed manufacturing information, for example.

We got it so easy with software that my occasional build problems look like child's play. Actually manufacturing millions of those different chips is hidden from sight, but oh so complicated, it's essentially pinnacle of human civilisation.
After reading this story I wonder if we are already into deep, deep trouble. So many things depend on timely production and delivery of complex goods and very often that production depends on a stack of badly written and badly used software operated by underpaid and demotivated staff. How long until we literally cannot improvise a solution anymore? When has the last "process optimization" by some MBA consultant put up so many layers between motivated human beings that a problem is unfixable?

Could civilization actually end like this? Not in a big bang but slowly grinding to a halt because no one knows how stuff works besides some nonfunctional web forms?

Everything looks crazy when you look at the details but the fact that we could all stay home for months and the world largely kept chugging along with only minor pricing changes and delays should give you an indication of how resilient supply chains are. No food was missing, no essential goods, largely the world stopped and yet all the important goods kept flowing. It's incredible to me how food shortages didn't hit multiple countries for example, which would be an indication of problems.
Sorry but you are sorely mistaken. The disruption for well-off people in rich countries was fairly minimal, but in other parts of the world and even in the poor areas of rich countries people felt the effects. Food insecurity in some countries is higher than ever, not to mention effects like years.of education missed (e.g. Philippines had 18 months of home schooling).
People are suffering, but we were not close to the end of civilization.
i dont know what you mean by “we could all stay home”, most people never stopped working, amazon deliveries kept coming, butchers and meatpackers never slowed down, and transpacific cargo ships kept moving (long beach port snafu notwithstanding)

also, I did have to go without my preferred brands of couscous and fresh olives for some months, so as far as i’m concerned the end was nigh

> No food was missing

Speak for yourself. A branch of my extended family all starved to death, together with lots of people in their town, and I was talking to them every day, but still couldn't get them food in time.

It likely wouldn't have happened without the pandemic. Staple food prices have gone up a lot (like more than double in some regions), and availability gone way down because the one guy with the truck decides to not deliver to the more remote regions cos he's making more money now doing more local deliveries.

That sounds really awful, condolences. It's bizarre to think that high tech services (assuming you spoke to them over the internet) were available while primary needs were not being met.

Do you have thoughts on what could have prevented this at a system level?

No... There will always be people living on the margins of survivability. Either too poor to afford food, and at risk of starvation... Or in a place too hot for humans... Or too dry for humans... If nobody lived in these places, then as population grows, people would be displaced to these places.

People at those margins then die when conditions change. I wish those people weren't family. But if it weren't them, it would have been someone else.

I'm so sorry that happened. Your story really highlights the huge disconnect between how the rich and the poor felt the effects of the pandemic and subsequent supply chain disasters.

May I ask which country this happened in?

> without the pandemic

Without the response to the pandemic. The virus is not responsible for the idiotic shitshow humans put up to "combat" it (spoiler: it didn't work, but at least we can live with it, I guess).

> A branch of my extended family all starved to death, together with lots of people in their town

I'm sorry for your loss. I can't imagine a whole branch of the family tree dying.

Can you say where this happened?

People in factories, ships, trucks, farms etc didn’t stay home.
Many people in factories did stay home. When the pandemic started my office was in a factory and i was told I wasn't allowed in the office. To get some test equipment sitting on my desk I had to get permission a week ahead of time. Likewise a lot of factory managers didn't go to the factory.

The people working the line in the factory need to go to the factory. However my company was paranoid that something would happen to them. If enough people working the line get sick production stops and then all profit stops. By contrast if a lot of people not on the line are sick but production continues - over a year is balances out.

And yet many factories continue to operate.

Notice how we have a backlog of ships to unload and not empty ships? That’s because factories continued operations.

Exactly the point: factories operated, but the managers tried (to various level of trying, some not very good) to protect the factory.
> we could all stay home for months

Maybe Hacker News was able to stay home for months but none of my friends making minimum wage--sorry, essential workers--ever got a break. I think you're dramatically overestimating how many people actually got to stay home and for how long. And obviously there's big class divisions between who does and doesn't get to stay home as well

> Could civilization actually end like this? Not in a big bang but slowly grinding to a halt because no one knows how stuff works besides some nonfunctional web forms?

You just summarized the Idiocracy plot.

But you do have a point... just look how much money companies are willing to shell out for IBM mainframes (new hardware that has cycle-accurate emulation of 70s-era hardware) or for pensioners who lived and breathed COBOL. So many large-scale IT projects in government or corporations fail because of the boatload of undocumented assumptions and edge cases that have been built up over many decades, or don't even get started because of the effort that a migration brings.

Could civilization actually end like this? Not in a big bang but slowly grinding to a halt because no one knows how stuff works besides some nonfunctional web forms?

No.

I'll give you a clear counter-example: my iPhone 13 mini.

Go read an iFixit teardown of any advanced product like that. Many state-of-the art chips, many suppliers, a huge logistics effort.

And yet Verizon managed to send me one of the damn things just a few days after I ordered, shortly after the phones were introduced.

Sure, some people trying to fund their product using kickstarter have problems, but the big boys can build and deliver hundreds of millions of "supercomputers" per year.

Absolutely. As things get more complex, the foundations become unstable and fewer and fewer people know how the underlying stack works.

You'll be interested in this talk, "Preventing the Collapse of Civilization"

https://www.youtube.com/watch?v=ZSRHeXYDLko

Thank you for this link. I had never heard of the ancient mechanical clock mechanisms. "The arc of progress" really is a fiction. Human knowledge needs to be passed down and nurtured. Same with culture.
This exposes a bit of something that everyone in the electronics industry knows, but nobody speaks about openly — that the electronic component distributors are IT dinosaurs. Their computer systems are dated and their software not up to today's standards. Most of them haven't realized that it is their software that is their core value right now, not the parts they have sitting on shelves.

I think this market is up for disruption once somebody brave enough appears and enters the market, starting from the software side.

Like a state-of-the-art ERP system?
Given that a "successful" ERP rollout seems to entail paying an army of ABAP consultants for years, and a major revamp of company processes, maybe the ERP market is also ripe for disruption? ;-)
Good luck with that. Because any ERP roll-out will involve the rework of business processes, whether you use SAP with expensive ABAP consultants or something else. Reworking the business processes to match the ERP is, by the way, the way it should be. Trying to do it the other way round is usually a good route into disaster.
Bad handling of out of stock components aside, I'm routinely amazed at how effective DigiKey and Mouser are. I can order a cart of 40 different items, each costing $0.05, getting 1 or 2 of each, and somehow they manage to routinely deliver in 2 days exactly what I ordered. They must lose quite a lot of money on those R&D orders - there's just no way you can make money having someone hunt down a specific resistor, cut 3 of them off a reel and put them in a bag.
Ah, that depends on order volumes (overall, not average order size) and their warehouse set-up. And it depends a lot on how much you pay for shipment. That being said, there are automotive parts suppliers that ship, reliably, orders of 3 screws, two nuts and 4 washers. From the UK to the EU nonetheless.

I could imagine either dirt cheap labor in well laid out warehouses or some form of automated picking solution.

> I could imagine either dirt cheap labor in well laid out warehouses or some form of automated picking solution.

i'd love to see a proper dive/writeup from someone in-the-know. lacking that, here's a third, completely speculative theory:

from what i hear, these big distributors are excellent at providing quick turnaround time. suppose that in order to guarantee this for large customers -- who presumably provide the bulk of revenue and who most value this service -- DigiKey must continually over-provision its operations. and so whenever there isn't a number of outstanding rush orders, there's extra staff available, at no marginal cost to the business, to assist the little guy.

This. It is astonishing how efficient they are.

I sometimes feel bad having to order a 3 cent part at Farnell/Element14. Which is then packed and shipped from the UK to the Netherlands without any handling or shipping fee. Usually it is delivered within 1 business day.

That said, they are treating a 3 cent part equal to a 100.000 dollar piece of equipment, which probably makes financial sense for them. Its the consistency that I, as a professional customer am willing to pay for. Farnell has become my one-stop-shop when it comes to parts and tools, I may be able to get something cheaper elsewhere, but the convenience and speed make up for that.

> I think this market is up for disruption once somebody brave enough appears and enters the market, starting from the software side.

Starting from the software side doesn't help very much, when you need enormous amounts of capital to build up initial inventory.

I think the problem is that it is an actual hard problem which is not trivial to solve. There are just so many different parts and they all come in different versions (packaging, industrial - automotive - military, etc.) with productions sites all over the world. I am actually pretty impressed by sites like octopart, how well they organize all that data and I am not quite sure what they could do better. But of course it's true, electronic products have so many different components and even one small missing component can cause havoc.
Excellent story. Love the bit a out adding a raspberry pi to the order so it wouldn’t get lost. Clearly for someone who could afford or need 18000 units there would be no supply chain issue. So it sounds like there are potentially more issues if you are a very small operation.
The entire electronics supply chain is currently a shit show and this post explained it perfectly. Other anecdotes from our experiences:

- Chipmaker/Distributor quotes 36 weeks for the parts, we arrange all shipments of all components to the factory based on that date since that part had the longest lead time (~80 unique part numbers in product). 4 weeks later, parts arrive at factory!. We have to wait for all other components for another 32 weeks. Manufacturer does not understand why we're mad since they shipped early!

- Distributor accepts order for a 10ppm MEMS oscillator. After 6 months of silence, we contact the manufacturer directly and they confirm that 10ppm parts will not be manufactured for at least another year since they are prioritizing 50ppm parts.

- Microcontroller manufacturer quotes ~2 years(!) for a part. The brokers have it in stock, and quote approximately 20 times the usual pricing (with a date code of 2021 to rub it in!)

- FPGA manufacturer accepts order for 500pcs, with a lead time of 48 weeks (ok!). After 30 weeks, manufacturer increases pricing for 20% for all orders, including open, active orders. They offer us to cancel the order if we want!

- Analog IC manufacturer announces that they will increase pricing of all products, including all open and active orders, but they can not tell how much of an increase it will be until..... until they are ready to announce I suppose.

- Chip manufacturer sells a chip through its own web page only. Part is currently not defined in any distributors so you have to use their web site. Chip is currently not in stock. However, their order system does not accept back-orders(?!). You have to check back "often" to catch the exact moment when this chip becomes available.

... the list goes on and on ...

> - Chipmaker/Distributor quotes 36 weeks for the parts, we arrange all shipments of all components to the factory based on that date since that part had the longest lead time (~80 unique part numbers in product). 4 weeks later, parts arrive at factory!. We have to wait for all other components for another 32 weeks. Manufacturer does not understand why we're mad since they shipped early!

Ah, so true! It shows nicely why predictable but longer lead times are better than potentially shorter ones that are totally chaotic. Being to properly plan is such a huge factor.

I've cleared international freight through Customs for a living for 15.5 years now, it's been interesting watching the reduction in electrical components coming in. It used to be that half or more of the shipments I would process a day would be integrated circuits/memory modules/resistors/capacitors. Now I'll see memory modules once a week, integrated circuits might be 5% of what I see a day, capacitors and resistors I'll see a few times a week now.

The value of the shipments is way way down too, instead of seeing a few hundred thousand dollars of ICs in a shipment, tens of thousands of ICs, I'll see tens of thousands of dollars of ICS in a shipment with the count in the thousands because, presumably, that's all that can be had.

I also notice HDD, SSD, and flash cards/USB storage imports seem to be way, way, down - presumably because they contain ICs as well.

Are you located in the USA?

What you say is somewhat surprising to me. I would imagine the total flow of components would be similar, but would be concentrated to certain destinations/buyers.

If cross-border trade of chips are actually in the decline, that suggests the components are being manufactured in lower quantities (highly unlikely), or they are staying close to and being consumed by (or, maybe hoarded and stocked at) where they are being manufactured.

>Are you located in the USA?

Yes. I clear freight for IND, OAK, EWR, MEM and a few other ports.

> that suggests the components are being manufactured in lower quantities

Eh, to me it suggests people are taking delivery as soon as they can of whatever they can get and that factories are possibly giving more priority to companies in their own country over international customers.

Ugh, I wouldn't wanna be releasing a product relying on hardware under these circumstances. Sometimes the best reaction to a crisis (such as this one leading to delays) is quitting altogether. It doesn't mean it will not restart again at some point (nor that it will!), its taking your loss and putting your money and energy elsewhere. Furthermore, it does something positive: it reduces demand, increases general availability. Which in turn has the potential to reduce incentive to increase price.

Another example: with the high gas price in Europe I feel more incentive to use more clothing, do more sports (increases blood circulation / body temp), and less electricity/gas. I don't want to force it upon my young children though, but I can afford the higher gas price. If you can't, you have to adapt.

That last paragraph…TI sells a part for set top boxes (TPS65235). It’s been sold out for months, but it’s possible to receive a notification email when they suddenly decide to run their own productions for the part (their website says 50 weeks). I always receive the notifications in the middle of the night, so by the time I order in the morning, all of the inventory is sold out. One time I happened to be awake during a notification and was unable to purchase just six minutes after receiving the notification.
> I confirmed that in fact the parts are shipped to Digikey as blanks, and they have a machine that can program the parts on-site.

I have never seen this before... I would expect the parts to either be factory programmed (and marked), or programmed by the customer (For example, the oscillator starts at 1 Mhz, and then you do some odd handshake to change the rate.).

Are there any other parts programmed at distributors? If this is a rare thing, I can completely see how the stock control mechanism can't cope with one part number incoming to the warehouse, but many different ones exiting...

(comment deleted)
> I’m considering ordering 18,000 oscillators and reselling the excess as singles via Crowd Supply

I've often wondered why no one offers something like this as a service. People who want to order a small quantity of some item with a large minimum order size could register their interest somewhere, and others who are interested could add themselves to their order, until the necessary threshold was reached.

Presumably the problem is one of scale, as the service would need millions of registered users in order to have a chance that enough of them all wanted the same thing at the same time.

For that reason, though, I'm surprised that Amazon hasn't offered this yet. If there's a product that sells as a box of 100 identical items, and you only want 1 or 2, you should be able to click a button that says "Let me join a group to order this product", and Amazon could charge a fee to split open the box for you and re-ship the individual items to the various members of the group.

Wasn’t this what Massdrop was for? (Now apparently at drop.com)
> "Drop is designing products differently."

It looks like it's more about enthusiasts buying specially designed niche products, rather than casual consumers buying a small portion of a large batch of low-value items, but it's an interesting site nonetheless, thank you.

Digi-Key and Mouser already do this for many many parts. Both offer to sell you a short bit of tape-held parts re-reeled and ready for insertion into a pick and place machine. Granted that's just for electronics and it sounds like you want it for more general things.

Part of the problem for doing it with other things is likely regulations and labeling. Back near the start of COVID I heard stories about how there was lots of cheese in bulk packs which couldn't easily be repurposed into consumer-friendly sized units (most normal consumers don't want to buy in units of cubic feet of cheese in the USA). Part of the reason it couldn't easily be repurposed was simply that no one had ever created a consumer-sized package label for it. So not only did the cheese need to be handled again, packaged again, but it also had to be relabeled and approved by what ever agencies approve or rate cheeses.

In the case of sour cream, the issue was the line making 5 gallon buckets couldn't fill smaller containers, and even if they could there weren't enough smaller containers. Whole factories are optimized for a package format and can't switch.

This is a trade off - some factories are more flexible than others. However if you can fill both sizes of containers (in reality there are more than two sizes, but for simplicity I'll use two) that means half the time the jigs needed for the one size are not in use - but they still need to be cleaned and such. You can't even skip cleaning because harmful bacteria doesn't care that it wasn't use much - in fact you might need to clean more often if the constant movement isn't flushing insignificant amounts away.

Consumer rights laws in various locations might make this tricky; you have to handle repair/refund, fitness for purpose, etc. When things are sold for retail, the manufacturer will have a support route for this. You'll also potentially fall afoul of how regulations differ between business sales and consumer sales.
Sounds like FUD. There are plenty of electronics distributors, small and large, that you can order individual ICs from as a consumer. How are they all pulling it off if it's so tricky?

Some parts have MOQs, I guess distributors don't see the business value in buying reels to sell a chip or three if it's reasonable to expect that the rest of the reel will sit on shelf for months (or worse).

> I did a quick trawl through the patents filed by SiTimes, and I’m guessing the MEMS oscillator chip contains at least two separate oscillators. These oscillators are intentionally different, so that their frequency drift with temperature also have different, but predictable, curves. They can use the relative difference of the frequencies to very precisely measure the absolute temperature of the pair of oscillators by comparing the instantaneous difference between the two frequencies. In other words, they took the exact problem that plagues silicon designs, and turned it into a feature: they built a very precise temperature sensor out of two silicon oscillators.

I finally got around to reading the excellent Logitude by Dava Sobel (https://www.goodreads.com/book/show/4806.Longitude) and to a lay-person this sounds like a similar principle that Harrison applied to design the bimetallic strip used to preserve the accuracy of his clocks in the face of temperature changes (https://en.wikipedia.org/wiki/Bimetallic_strip). Very cool.

Thank you kind sir. I just woke up and the bimetallic strip has already made my day.
I've had a couple of opportunities to interact with Digikey support and can confirm.

They are incredibly, almost unbelievably helpful.

They've more than earned my business as my primary distributor.

Digikey has been reliably amazing for a very long time.

Calling them up in CONUS and getting a real human, normally with a strong “Bobby’s World” yooper accent, who speaks flawless and rapid NATO Phonetic Alphabet and rattles off your part numbers back to you to double check them … perfect.

Two things jump out at me: 1. Having to order parts yourself and send them to a factory is something I’ve done a few times and still baffles me. It was easier for me to buy some parts from the US, ship to me in the UK and then send to China than for the Chinese factory to do it. All totally legit and about a 95% time saving.

2. > I finally do the “unthinkable” in the web age – I pick up the phone, and dial in hoping to reach a real human being to plead my case. I dial the extension for the custom department sales rep. It drops straight to voice mail. I call back again, this time punching the number to draw a lottery ticket for a new sales rep.

He should have done this at the start. When it comes to supply chain problems it’s usually always a people problem. The first thing you do as soon as you get an undesirable result from email is pick up the phone and have a conversation.

It works pretty much every time.

This exactly. The difference between the faceless monoliths of the FAANG customer support world and those of the electronics distribution world is that in the latter case there is a sales rep behind the monolith who stands to get a substantial commission for making sure you can successfully get parts.

If there's anything I've gained from this hellish year+ of building hardware, it's a much more frank and open relationship with our various CMs and distribution reps. These problems are none of their faults, and in many cases they've gone to ridiculous lengths to keep us (a relatively small fish) afloat.

100% my default was to just pick up the phone and call a supplier - then when you’re in a bind you have a personal relationship that goes a long way to smoothing out issues (in both directions)
I thought I understood this until:

> the parts are shipped to Digikey as blanks, and they have a machine that can program the parts on-site

What is going on here? Is the thing that there is a picture of earlier in the blog post (the two chips bonded together with some other parts surrounding it) a piece that can do many different jobs and be sold as many different SKUs??

I believe they're referencing a fixed clock speed, so I'd assume that they have a SKU for standard clock speeds, which can be set as needed.
Its a programmable pll clock synthesizer(100ppm) pretending to be a crystal oscillator.
I like Bunnie a lot and I always enjoy reading posts from his website.

That said, while I generally support his hardware manufacturing ideas, it's exceptionally hard to compete in this space against the largest electronics companies and against Asian manufacturing. As someone who also used to be involved in North American electronics manufacturing, there are so many factors, from local environmental regulations to the large network effects in Shenzhen, that it's almost impossible unless you have something truly novel and brilliant marketing. Good luck to Bunnie, I hope it works out.

Finally, let me just comment on the magnificent customer service of Digikey. If I ever had to source anything electronic, I would use them again in a heartbeat, to the point where I would use them over cheaper alternatives, because they are so good.