I've been wondering if using such exhaustive datasets would allow to discover the existing supply chain routes (which is usually a closed guarded trade secret). For that I would model the truck, ship and freight train traffic as edges, and mines/factories/companies as nodes in a graph. The weight of each edge is proportional to the number of containers that flow through it per unit of time. The question is, would it be possible to statistically infer the interdependence among nodes?
I started it and so far I got nice visualizations of air and maritime traffic. I should resume it when I find some time.
In yet other graphs, Great War mobilisation was largely a function of the rail networks. To this day, it's possible to buy a "rail" ticket between finland and sweden that winds up "air gapped" via bus transfer.
It's possible to have different gauges at the same station, requiring just a change of train rather than an additional bus transfer. And some systems are variable gauge, allowing for the same train to operate on different systems.
Back in the day choosing your rail gauge had a large input from the military, since it was strategically disadvantageous to have the enemy supply lines easily extended into your heartland. This was before the days of air forces or even proper highway systems, so logistics depended largely on horse drawn wagon trains.
It's not so different from the Soviets choosing an ever so slightly different caliber for their mortars so it's not compatible with the ones from NATO.
Most countries plan their military around defense of their territory though, planning around military aggression of their neighbours has only happened for short periods and in a few countries since the invention of railways.
> But wouldn't you want your trains to easily extend into theirs?
Can’t find the link now but I read a fascinating article about the German and Russian use of railways on the WW2 Eastern Front. Long story short IIRC: the Russian rail network was primitive by German standards, primarily lacking the automated signalling that allowed relatively high speed trains. Primitive, but it still worked and was very resilient to damage. The Russians destroyed a lot of their railways as they retreated and could quickly rebuild it when their fortunes changed.
Also, the Russians operated their trains using a centralised planning function that could prioritise strategically important routes and cargoes. In contrast, senior German commanders could in effect commandeer trains to support their local tactical needs, sometimes playing havoc with the overall network.
Yep. Every oil trader and company is monitoring trade routes for insights. It’s almost a rite of passage for new analysts and interns to start playing with vessel data to look for new patterns.
hi, I'm Kpler CTO. Indeed the space is crowded. To succeed you not only need strong algorithm on top of AIS data (ML, Operational research) but also lots of hard facts coming from partners in the supply chains (brokers, ship agents, ports authorities) that you need to extract with NLP and other strategies. Having a strong team of analysts with expertise in each commodity market is needed to interpret and complement the automation.
Last piece is also to have a strategy not only to cover shipping transportation, but also storage, production, consumption of the commodities, so you can deduce more interesting intel. We for example use satellites and drones images as well to know the crude oil inventories onshore.
I come from that side (analyst) and am more of an enthusiast on the technical side- so I have always watched Kpler and some of the other newcomers with a lot of interest as you seem to be pushing the envelope compared with a lot of the more traditional commodity research houses which have been doing this for more years (the likes of Eikon, etc).
Curious, there seems to be a 'center of gravity' in new upstarts in the space in France- why is this?
At my previous employer we had two people working on data analytics for container barge traffic within the Rhine delta in the Netherlands and Germany. They were able to overlay their predictions with the partial barge plannings that we actually had because we supplied software to inland container terminals.
Conclusion was that yes, you can predict locations of the terminals, but it's quite hard to find and produce repetitive schedules from the data because it requires the ships not change their schedule for too long, and the reality is just too dynamic to properly predict this.
For seagoing vessels I expect that they have more predictable schedules, but the hard part there is knowing container volumes, because there is no way to predict from public data what the volume of containers loaded/discharged is at a specific terminal. The closest metric is measuring the turnaround time of the ship, but that is not a great predictor for number of containers moved because handling times per container will differ per port, as will the number of cranes assigned to the ship, as will the ratio between 20ft and 40ft boxes, as well as the capability per port to do twinning (picking up 2x20ft boxes in one handling) and dual-cycling (discharging and loading a container as part of the same crane cycle).
Basically: it's a tough problem, with too many variables you cannot reasonably deduce.
(Also, even if you have the container volumes, I personally think the highest potential for optimization is in the hinterland, but that's a different story altogether)
> because there is no way to predict from public data what the volume of containers loaded/discharged is at a specific terminal
It's doable for a competitor. AIS has the vessels depth. If you assume the weight of a container to be similar you can monitor the vessels depth as they discharge and then load containers. You'll have to account for bunkering, empty containers, etc. Still, it's doable.
There's also some data (not sure if public, should be easy for any shipping company) for anything going to the US. You can use that data to further improve the previous estimates. See https://www.joc.com/regulation-policy/trade-data/united-stat.... Interestingly the JOC is terrible at predicting future trends despite having one of the most detailed data on their trade.
Note: the method above will not give you exact numbers. It's not actually needed to know the exact numbers, seeing the trends and the fluctuations/changes is already quite useful.
I would be very impressed if that was more accurate than turnaround time, but interesting approach, hadn’t thought of that.
But I’m guessing that adjusting for bunkering is tricky since they might fuel up more in ports where fuel is cheaper, so you cannot assume they will be nearly empty close to every port they call. Would love to know if guessing based on this is actually being used though.
Vessel depth (draught) is usually (or always) a static number;hard coded in the ais transponder and never again uodated and thus provides zero information. The only somewhat reliable data are position and heading and their derivatives. Next to that you can get something out of the broadcast frequency, and that's about it. Of course the data can and is easily manipulated by some, and the broadcast is lossy so sometimes a few bits get lost or mangled in transit, there is no proper error correction in the protocol
This is possible, although hard, and is being done at a number of businesses. Most of my experience in this space revolves around working with this business in a consulting capacity a few years ago - https://truebearinginsights.com. The short answer is that it gets easier when you focus on particular types and sizes of vessels.
Ships are classified into size-based categories. For example, a "capesize" vessel is a massive dry bulk ship of 100,000+ ton (and often much bigger) carrying capacity. Since they are the largest ships, there are many fewer of them in the world, they are size-restricted on the ports that they can visit, it is only economical to carry particular types of cargo in these volumes, and because of the volumes, the ports often sit close to the source.
Now, when you collect AIS data for capesize ships, you'll see trade lanes from Australia to China - these mostly carry Iron Ore from Australia's Pilbara region to steel/etc. factories in China (companies like Rio Tinto, BHP and FMG focus on this space). You'll see a similar trade lane from Brazil to China - this is also iron ore and led by a company called Vale. By analysing the trade lanes, cross referencing that with the raw materials that come out of each company, using Google Maps to investigate what the ships might be loading, etc., you can infer what the vessels carry and begin to extrapolate these routes. The same applies for many types of commodities, and is not limited to dry bulk.
It can be easier in some cases to figure out the trading routes for ocean transportation of containers because the carriers often publish their routes in advance, but this obviously doesn't apply to the rail and trucking side of things, containers are often transported a fair distance before reaching a port, and containers will often be trans-shipped (i.e., transferred from one vessel to another at a hub port like Singapore), so it will be difficult to find the root nodes here.
As you go down in vessel size, it gets harder. A small ship like a "handysize" could carry anything really - grain, potash, windmills, yachts (yachts often aren't insurable when crossing the open ocean so the owners typically have them shipped around the world instead). And there are tons of them. And their cargoes might be shipped long distances to the port. You can still find these nodes of course, but it will take a lot more work.
There is also: https://skibstrafik.dk/ which is pretty good.
You can even get a picture (if available) and more information about a ship by clicking on it.
Fun fact - some ships turn off their AIS while being on seas and they are allowed to do so (non-commercial) which is a pain if you are sailing one handed or in bad weather conditions (fog).
Sorry, I'd thought you were referring to navy (non-commercial) ships.
Cue the I'm an admiral / I'm a lighthouse "give way" joke[1]. As of 2020 we've learned (open source: reuters) that naval vessels may be deadly in hot wars, but don't do so well in full and frank discussions ('shouldering' when grey on grey) with either cargo vessels or arctic-hulled vessels.
Back in fax-spam days, the USN even officially responded to this joke:
“The first time I heard of it was — oh, let’s see, how long — about 10 years ago or so, I think. That story’s so old,” [LANTFLT spokesman, Cmdr. Kevin] Wensing said, “it probably started out back in the galleon days, or back when there was a big lighthouse at Alexandria, Egypt.”
Not sure about true data, but i could imagine that cargo vessels are involved in a majority of accidents on sea. They are - fast - like really fast. You cannot outrun them with your sailboat, even with engines. And they won't change course, because they cant.
If they turn the rudder, they need to wait for some 2 miles for it to take effect. Suprisingly many sailors don't know that or neglect it.
So next time there is a war, just send dem cargo ships.
All the deaths in my area are involve fishing boats and pleasure craft.
As a sailboat you are the stand-on vessel so you really shouldn't maneuver. They will see you. They can and frequently do make course adjustments to avoid other traffic.
Well ferry vessels have special rights, you should be careful.
But every big ship will eventually turn, but they'll always scare you. Especially sailors, they don't like sailors...
If you're thinking "I'm the stand on" in your sailboat with regard to a container ship, you've probably missed the opportunity to stay well clear which should have been your guide before getting into that situation.
I love using https://www.vesselfinder.com/ to identify what I can see in the Firth of Forth. It looks like the cluster of parked cruise ships has gone. Wonder if they've resumed service?
I just saw an article that indicates that cruise ships plan on resuming service August 1st, but it was just a headline. I did not read the article to see what country/ies that date refers to.
I run one of the AIS stations that contributes to this. Of course, "RUN" is a bit of an overstatement, the little rPi unit basically just sits there unattended and feeds data received into the system without much need for any attention from me.
I am in the process of adding an uplink from my AIS650 on my boat to also send data received while underway, but need to finish up some other projects first.
I do not have much use for the commercial data side of this, but looking at past routes of various pleasure craft going to or from places I want to go can be informative for planning routes.
It's been a while since I have worked in this space but the underlying protocol (AIS) is incredibly lacking in terms of security due to the nature of plain text transmission & lack of authentication. I'm not sure if these issues have been addressed but below is some great research from 2014 on the matter [1][2].
45 comments
[ 4.7 ms ] story [ 105 ms ] thread(this also makes obvious why the Suez Crisis was so important. I use it as a marker for the UK/US hegemony boundary.)
I started it and so far I got nice visualizations of air and maritime traffic. I should resume it when I find some time.
(note that if one has access to SWIFT data, one can not only watch the goods transported in one direction but also the payments flowing in the other.)
In other graphs, I believe California has (had?) superior petrol environmental standards to the rest of the US because its pipeline network is largely independent: https://www.api.org/~/media/Oil-and-Natural-Gas-images/Pipel...
In yet other graphs, Great War mobilisation was largely a function of the rail networks. To this day, it's possible to buy a "rail" ticket between finland and sweden that winds up "air gapped" via bus transfer.
https://en.m.wikipedia.org/wiki/Variable_gauge
It's not so different from the Soviets choosing an ever so slightly different caliber for their mortars so it's not compatible with the ones from NATO.
Can’t find the link now but I read a fascinating article about the German and Russian use of railways on the WW2 Eastern Front. Long story short IIRC: the Russian rail network was primitive by German standards, primarily lacking the automated signalling that allowed relatively high speed trains. Primitive, but it still worked and was very resilient to damage. The Russians destroyed a lot of their railways as they retreated and could quickly rebuild it when their fortunes changed.
Also, the Russians operated their trains using a centralised planning function that could prioritise strategically important routes and cargoes. In contrast, senior German commanders could in effect commandeer trains to support their local tactical needs, sometimes playing havoc with the overall network.
[1]: https://www.vortexa.com/
[1]: https://www.kpler.com/
[2]: https://www.spinergie.com/
[3]: https://www.refinitiv.com/en/products/eikon-trading-software...
Curious, there seems to be a 'center of gravity' in new upstarts in the space in France- why is this?
Conclusion was that yes, you can predict locations of the terminals, but it's quite hard to find and produce repetitive schedules from the data because it requires the ships not change their schedule for too long, and the reality is just too dynamic to properly predict this.
For seagoing vessels I expect that they have more predictable schedules, but the hard part there is knowing container volumes, because there is no way to predict from public data what the volume of containers loaded/discharged is at a specific terminal. The closest metric is measuring the turnaround time of the ship, but that is not a great predictor for number of containers moved because handling times per container will differ per port, as will the number of cranes assigned to the ship, as will the ratio between 20ft and 40ft boxes, as well as the capability per port to do twinning (picking up 2x20ft boxes in one handling) and dual-cycling (discharging and loading a container as part of the same crane cycle).
Basically: it's a tough problem, with too many variables you cannot reasonably deduce.
(Also, even if you have the container volumes, I personally think the highest potential for optimization is in the hinterland, but that's a different story altogether)
It's doable for a competitor. AIS has the vessels depth. If you assume the weight of a container to be similar you can monitor the vessels depth as they discharge and then load containers. You'll have to account for bunkering, empty containers, etc. Still, it's doable.
There's also some data (not sure if public, should be easy for any shipping company) for anything going to the US. You can use that data to further improve the previous estimates. See https://www.joc.com/regulation-policy/trade-data/united-stat.... Interestingly the JOC is terrible at predicting future trends despite having one of the most detailed data on their trade.
Note: the method above will not give you exact numbers. It's not actually needed to know the exact numbers, seeing the trends and the fluctuations/changes is already quite useful.
But I’m guessing that adjusting for bunkering is tricky since they might fuel up more in ports where fuel is cheaper, so you cannot assume they will be nearly empty close to every port they call. Would love to know if guessing based on this is actually being used though.
Ships are classified into size-based categories. For example, a "capesize" vessel is a massive dry bulk ship of 100,000+ ton (and often much bigger) carrying capacity. Since they are the largest ships, there are many fewer of them in the world, they are size-restricted on the ports that they can visit, it is only economical to carry particular types of cargo in these volumes, and because of the volumes, the ports often sit close to the source.
Now, when you collect AIS data for capesize ships, you'll see trade lanes from Australia to China - these mostly carry Iron Ore from Australia's Pilbara region to steel/etc. factories in China (companies like Rio Tinto, BHP and FMG focus on this space). You'll see a similar trade lane from Brazil to China - this is also iron ore and led by a company called Vale. By analysing the trade lanes, cross referencing that with the raw materials that come out of each company, using Google Maps to investigate what the ships might be loading, etc., you can infer what the vessels carry and begin to extrapolate these routes. The same applies for many types of commodities, and is not limited to dry bulk.
It can be easier in some cases to figure out the trading routes for ocean transportation of containers because the carriers often publish their routes in advance, but this obviously doesn't apply to the rail and trucking side of things, containers are often transported a fair distance before reaching a port, and containers will often be trans-shipped (i.e., transferred from one vessel to another at a hub port like Singapore), so it will be difficult to find the root nodes here.
As you go down in vessel size, it gets harder. A small ship like a "handysize" could carry anything really - grain, potash, windmills, yachts (yachts often aren't insurable when crossing the open ocean so the owners typically have them shipped around the world instead). And there are tons of them. And their cargoes might be shipped long distances to the port. You can still find these nodes of course, but it will take a lot more work.
http://map.openseamap.org/?zoom=9&lat=35.98913&lon=-5.41849&...
https://books.google.ch/books?id=nUtLDwAAQBAJ&pg=PA46&lpg=PA...
meant that submarines used to be able to pass in both directions with engines off.
Still physically possible, but no longer worthwhile: https://news.ycombinator.com/item?id=23548220
Cue the I'm an admiral / I'm a lighthouse "give way" joke[1]. As of 2020 we've learned (open source: reuters) that naval vessels may be deadly in hot wars, but don't do so well in full and frank discussions ('shouldering' when grey on grey) with either cargo vessels or arctic-hulled vessels.
[1] https://www.youtube.com/watch?v=EPqtwV5utUY
Back in fax-spam days, the USN even officially responded to this joke:
“The first time I heard of it was — oh, let’s see, how long — about 10 years ago or so, I think. That story’s so old,” [LANTFLT spokesman, Cmdr. Kevin] Wensing said, “it probably started out back in the galleon days, or back when there was a big lighthouse at Alexandria, Egypt.”
So next time there is a war, just send dem cargo ships.
As a sailboat you are the stand-on vessel so you really shouldn't maneuver. They will see you. They can and frequently do make course adjustments to avoid other traffic.
https://ptyc.net/wp-content/uploads/Do-Sailing-Vessels-Alway...
That said, I’ve been yelled at while racing when crossing ferry pilots. I had the right of way and they had the tonnage.
Unarmed cruise ship 1, Venezuelan navy 0.
This is my space. A very crowded space, and incredibly chaotic, where data is outright wrong most of the time.
Personally, I don't prediction should be the end goal of these analyses, tracking and anomaly prediction would give more insight to the traders etc.
But of course I would only say this cause prediction is hard in this space.
DL (Deep Learning) Et al. don't really help. Traditional stats is more likely to be helpful in this case.
I am in the process of adding an uplink from my AIS650 on my boat to also send data received while underway, but need to finish up some other projects first.
I do not have much use for the commercial data side of this, but looking at past routes of various pleasure craft going to or from places I want to go can be informative for planning routes.
The link is currently: https://marinetraffic24.com/pt/vesselfinder/
The English language page is: https://marinetraffic24.com/vesselfinder/
Hopefully a moderator like dang can edit the link.
Seems to work as expected in Chrome.
[1] https://www.blackhat.com/docs/asia-14/materials/Balduzzi/Asi...
[2] https://www.youtube.com/watch?v=5rt9dzu3I7U