He should've made a 10 minute youtube video with animations of horse's ass, etc just to reveal this in the last 15 seconds... "if you liked this video, please like, subscribe, share and hit that bell icon to bla bla bla!".
That’s my understanding - a lot of the low power devices were big endian as for whatever reason big endian and risc seemed heavily correlated for decades, similarly the early mainframes and whatnot were big endian which is why a lot of network protocols use big endian formats.
I've noticed that decisions like this are sometimes made for the sake of being the opposite of something else and then the two coexist decades within neighboring realms to the frustration and bafflement of users.
"decisions like this are sometimes made for the sake of being the opposite of something else and then the two coexist decades within neighboring realms to the frustration and bafflement of users"
The article says "The reason the early developers at OpenCV chose BGR color format is that back then BGR color format was popular among camera manufacturers and software providers. E.g. in Windows, when specifying color value using COLORREF they use the BGR format 0x00bbggrr.". So it claims that both camera engineers and software used BGR. I'm not convinced though - the Win32 macro to create a COLORREF is RGB(r,g,b), and there are 3 macros to get the R, G and B components out of a COLORREF. Until today I've never realized that a COLORREF stores its components in BGR order, and I've done Win32 graphics programming for 20 years. I think it has more to do with that the 'original' graphics-oriented architectures (SGI workstations and such) were big endian (for no particular reason, probably, although people have come up with retrofact rationalizations probably), and everyone just used that. Some 'standards' don't need a reason, someone just needs to make a choice and everyone else should just stick with it...
Also, I'm pretty sure that COLORREF is xBGR format because colour data in memory is RGBx format and when you read it DWORD at a time on a little-endian machine, you end up with xBGR in registers. So an array of COLORREFs in memory is RGBx format...
On the other hand, BMP is BGR format on disk and in memory[1] when using 24-bit colour. I think "opencv used BGR because BGR was the format of the default image file type on the first platform it was written for" might explain it better. Camera makers probably did the same thing - output BMPs before JPEG was developed.
[1] Yes, technically BMP also has bitmasks that let you store the colour channels in any order with any precision, but almost all BMPs I've come across just use the default BGR format.
Before deep learning is a thing, most computer vision tasks are done either in greyscale, or in a more reasonable colorspace (HSL, YUV etc.). Thus, you most likely just call cvCvtColor the first thing when extracted the frame, and it doesn't matter BGR or RGB.
Only with deep learning, people start to use RGB directly, and this becomes something more interesting.
For DL the colorspace is irrelevant as long as it's consistent between training and inference. I.e. if you train your model in RGB, it better be receiving RGB and not something else from the video feed or input images. Interestingly, models can also be trained in YUV (including its downsampled variants like YUYV), which can make them a tiny bit more efficient, both because there's less data, and because most camera chips can output YUV raw data directly. The net will learn whatever transform it needs to learn to make sense of the data.
I don't think you can fully claim it is irrelevant. Especially not and point out that it can make things a bit more efficient.
I think it is safe to say that the most important part is to be consistent between what you train on and what you infer against. However, it could be that one of the color spaces is more amenable to training. To my knowledge, that has not been completely ruled out. Indeed, it seems supported by your claim that they can get a tiny bit more efficient with some spaces.
That said, this post is my long winded way of asking if you have links on that. :)
What makes YUYV more efficient is downsampling. You represent 2 pixels with 4 bytes instead of 6 due to the loss of spatial color information. If there's no downsampling, and there exists a transform between RGB and a given color space, your network will simply learn slightly differently. Remember that after the very first layer there's no notion of color space anymore.
Intuitively, that makes sense. However, I would still feel more comfortable to have it somewhat proven. Most intuition I have is out the window with deep learning.
Interessting. Why is not the "raw" sensor format, for example Bayer BG8, used more in deeplearning? Would it not contain a little bit more correct information? and if used, we could skip the Bayer-> RGB conversion completly.
Because most models are trained from files, which store in RGB. Using raw sensor format would make sense for inference, but if your model is created for RGB, why bother.
Pretty much. And also, models almost never operate on full-size image, and if you e.g. pixel bin a bayer image you will get a de-mosaiced image, so that's not very interesting. What is interesting is, it'd be pretty cool to train on linear HDR images for instance (i.e. direct 14-stop sensor output without contrast curves and color correction), or on visible spectrum + IR + UV, but I'm not aware of any large public datasets like this. The reason why this is interesting is because a lot of vision models suffer from the lack of robustness. Their performance on the train/validation datasets might be OK, but connect them to a real live video feed and they turn to shit. To some extent this can be addressed by data augmentation during training, but you have to understand that this is an imperfect process: you're operating on the data that's already pre-cooked, so to speak, for _human_ consumption, with color correction, gamma curves, etc already applied, and there's not a whole lot you can do to make things more robust to different lighting conditions, noise in the feed when camera bumps the gain in its ADCs, color drift due to e.g. sodium lamps, over/under exposure, etc. Unsurprisingly, this is especially noticeable in efficient, quantized models, because they don't really have much leeway built in to work around the deficiencies in the input data.
Regarding multi-spectral images, there was a recent Kaggle competition classifying ship-or-iceberg from radar multiple band data. So yeah, CNNs work well for that too.
Public raw image datasets are rare, probably because most cameras, except DSLRs, don't support storing raw images.
Also, the nature of the inference may not require high fidelity raw images since they're going to be down-sampled anyway. Unless research suggests that raw images improve inference metrics, nobody's going to invest effort into building a raw image dataset. One notable recent exception to this is the "Learning to see in the dark" paper on low light image enhancement[1] - they found that training on raw images gives much better enhancement than JPEG.
The data availability problem may start getting solved in the near future because mobiles (atleast Android) support storing and processing raw images, and on-device learning and inference capabilities are getting better all the time on mobiles.
Interesting link! Yhea, I guess JPEG destroys a lot information low light images.
> probably because most cameras, except DSLRs, don't support storing raw images
This may be true for consumer cameras but not for machine vision cameras. Most color machine vision cameras do support the Bayer format. Basler, FLIR (Point Grey), IDS and so on all support the Bayer format. If you roll your own camera you definitely have to deal with the Bayer format from the sensor you are using.
And if you do not want any compression on your data-set images the Bayer format also takes less space than the RGB format.
I'm not arguing it is a good idea, I'm just interested if it works :)
JPEG definitely destroys low light information. I've had some pictures I took with in RAW that I thought had nothing in them, clicked "optimize" in generic photo editor and realized I could make out a ton of detail. Didn't magically recover a good photo, mind, but was a lot more information than originally visible. Tried from the JPEG and got nothing.
It's useful information about industrial machine vision cams, since I don't have any experience with them.
You are right about custom cameras - in fact, I have some camera modules for experimenting with surveillance, night vision and machine learning, and some of their datasheets mention registers that, when set, output Bayer format images. Machine learning in surveillance is still a rather under-serviced market, but it's potentially a rich source of raw image datasets.
Raw format training and inference definitely work. The obstacle is only data availability.
Not quite true. Algorithms like vgg-16 typically involve subtracting out the mean at the beginning for better performance. For example, look at https://pytorch.org/docs/stable/torchvision/models.html. Red has mean .486 and blue has mean .406. While this could be calculated from the dataset, it does seem like a lot of the frameworks like to hardcode it.
No, the post about OpenCV and BGR talks (for about 80% of it's length) about how everything we know in transportation, up to space shuttle, is linked to horse width.
Which, as pointed out in the link I provided, is simply false (or at the very least a gross oversimplification).
One of the key points of the "debunking" seems to itself make a logical error:
> The eventual standardization of railroad gauge in the U.S. was due far less to a slavish devotion to a gauge inherited from England than to the simple fact that the North won the Civil War and, in the process, rebuilt much of the Southern railway system to match its own
But the article earlier states that the North won the Civil War because of the logistical superiority of its rail lines that had settled on using the English standard gauge. Even if the South had settled on a single alternative gauge, they would have been unable to easily acquire rail equipment from abroad without added expense.
Snopes' attempt at debunking achieves the opposite in my reading.
I don't agree with this kind of "just to say" trivialization. Storytelling is an art many enjoy writing and reading. I actually learnt some new things I didn't know before. I had assumed all digital cameras have always been using YUV or variants. It's also helpful that the author reported his conversation with the original author of OpenCV - that kind of authentication is rare and valuable.
I can imagine the comments here if the blog was actually a one line article stating "BGR was used a lot back then".
> E.g. in Windows, when specifying color value using COLORREF they use the BGR format 0x00bbggrr.
Wait a little bit... isn't Windows little-endian [1]? Following the link in that sentence:
> typedef DWORD COLORREF;
> The low-order byte contains a value for the relative intensity of red; the second byte contains a value for green; and the third byte contains a value for blue.
Given that Windows is and has always been little-endian, the byte order of a COLORREF is red first, then green, then blue, that is, the order is RGB. So the reason for BGR can't be Windows' COLORREF.
I thought for a moment that they had the order of COLORREF wrong, but I just double-checked the definitions of the GetRValue, GetGValue, and GetBValue macros. You are correct, writing that DWORD into memory would put them in RGB order not BGR.
Somewhat off-topic, but I just hate that Snopes article. It's wrong, and rather off-base.
Functionally speaking, the main disconnect in the article is the difference between loading gauge and track gauge. 1435mm, or 8'4.5", may be the standard track gauge and uniform between most countries, but the actual standard loading gauges varies quite dramatically between different countries. The UK has a positively tiny loading gauge (about 2.5m), while the "standard" EU loading gauge is 3.1m, the US is slightly wider at 3.2m, and Sweden opts for a roomy 3.4m.
The original notion that the SRB has to be "slightly" larger than the 1435mm track gauge is ridiculous, since as you can see, the loading gauge is over twice the width the track itself. Of course, there's also rooms to play tricks here if you need to. Schnabel cars can laterally offset the loads they're carrying by up to a meter or so, which gives you some extra height to play with if you're dealing with a double-track tunnel with an arched roof.
The other point to mention that's completely omitted is there is a rather big difference between rail carriages and horse-drawn carriages. On rail carriages, the cars sit above the wheels, while horse-drawn carriages sit between the wheels. Something like this:
bbbbbb bbbbbbbbbbbbbbbnbb
b b b b
w b b w w b b w
w bbbbbb w w b b w
waaaaaaaaw waaaaaaaaaw
w w w w
w w w w
(Note that this is one of the reasons that the standard gauge was chosen over a broader gauge).
As for why standard gauge was chosen, well, keep in mind that a) there was a lot of variance of track gauge in the 1800s, and b) the actual tolerance of track gauge was pretty damn massive. A track gauge differing by as much as two inches or so wouldn't have impeded the ability to transfer rolling stock. Stephenson built the first practical steam locomotive on a line that was 4'8", and so people who wanted to reuse his locomotives would have been drawn to a gauge around 4'8"-4'9". Gauges around 5" and slightly larger were selected in part to prevent interconnect between different railroads (basically, the same concept as vendor lockin).
Contrary to the Snopes article, the South mostly standardized on 5' gauge, but it didn't have as dense a network as the North did. The South eventually converted to 4'9" gauge (the Pennsylvania Railroad's gauge)--in 1886, 20 years after the Civil War. Conversion thence to 4'8.5" took much longer, but was much less dramatic because 4'9" and 4'8.5" are within gauge tolerance.
May be the "Roman horse" here is that BGR is RGB in little endian format. Also, it has lots of advantages: little endian pixel handling allows using similar code for 24 and 32bpp (rr.gg.bb.rr.gg.bb vs rr.gg.bb.xx.rr.gg.bb.xx), so you can loop over pixes with offset += pixel size.
The article makes it sound as if moving OpenCV to RGB is as difficult as moving the US to the ISO metric system. Neither is impossible. If Europe manages to abolish national currencies, the US surely can abolish imperial units (and drive faster ;).
But I guess that will only happen once under pressure of chinese overlords. I'm sure the day will come, so why wait?
46 comments
[ 2.3 ms ] story [ 105 ms ] threadAll that writing for a one line explanation...
Oh, you mean a political two-party system?
On the other hand, BMP is BGR format on disk and in memory[1] when using 24-bit colour. I think "opencv used BGR because BGR was the format of the default image file type on the first platform it was written for" might explain it better. Camera makers probably did the same thing - output BMPs before JPEG was developed.
[1] Yes, technically BMP also has bitmasks that let you store the colour channels in any order with any precision, but almost all BMPs I've come across just use the default BGR format.
Only with deep learning, people start to use RGB directly, and this becomes something more interesting.
I think it is safe to say that the most important part is to be consistent between what you train on and what you infer against. However, it could be that one of the color spaces is more amenable to training. To my knowledge, that has not been completely ruled out. Indeed, it seems supported by your claim that they can get a tiny bit more efficient with some spaces.
That said, this post is my long winded way of asking if you have links on that. :)
https://www.kaggle.com/c/statoil-iceberg-classifier-challeng...
Also, the nature of the inference may not require high fidelity raw images since they're going to be down-sampled anyway. Unless research suggests that raw images improve inference metrics, nobody's going to invest effort into building a raw image dataset. One notable recent exception to this is the "Learning to see in the dark" paper on low light image enhancement[1] - they found that training on raw images gives much better enhancement than JPEG.
The data availability problem may start getting solved in the near future because mobiles (atleast Android) support storing and processing raw images, and on-device learning and inference capabilities are getting better all the time on mobiles.
[1]: https://www.youtube.com/watch?v=qWKUFK7MWvg&feature=youtu.be
> probably because most cameras, except DSLRs, don't support storing raw images
This may be true for consumer cameras but not for machine vision cameras. Most color machine vision cameras do support the Bayer format. Basler, FLIR (Point Grey), IDS and so on all support the Bayer format. If you roll your own camera you definitely have to deal with the Bayer format from the sensor you are using.
And if you do not want any compression on your data-set images the Bayer format also takes less space than the RGB format.
I'm not arguing it is a good idea, I'm just interested if it works :)
You are right about custom cameras - in fact, I have some camera modules for experimenting with surveillance, night vision and machine learning, and some of their datasheets mention registers that, when set, output Bayer format images. Machine learning in surveillance is still a rather under-serviced market, but it's potentially a rich source of raw image datasets.
Raw format training and inference definitely work. The obstacle is only data availability.
Which, as pointed out in the link I provided, is simply false (or at the very least a gross oversimplification).
> The eventual standardization of railroad gauge in the U.S. was due far less to a slavish devotion to a gauge inherited from England than to the simple fact that the North won the Civil War and, in the process, rebuilt much of the Southern railway system to match its own
But the article earlier states that the North won the Civil War because of the logistical superiority of its rail lines that had settled on using the English standard gauge. Even if the South had settled on a single alternative gauge, they would have been unable to easily acquire rail equipment from abroad without added expense.
Snopes' attempt at debunking achieves the opposite in my reading.
I can imagine the comments here if the blog was actually a one line article stating "BGR was used a lot back then".
Wait a little bit... isn't Windows little-endian [1]? Following the link in that sentence:
> typedef DWORD COLORREF;
> The low-order byte contains a value for the relative intensity of red; the second byte contains a value for green; and the third byte contains a value for blue.
Given that Windows is and has always been little-endian, the byte order of a COLORREF is red first, then green, then blue, that is, the order is RGB. So the reason for BGR can't be Windows' COLORREF.
[1] "A decision was made VERY long ago that Windows would not be ported to a big-endian processor." https://blogs.msdn.microsoft.com/larryosterman/2005/06/07/th...
Functionally speaking, the main disconnect in the article is the difference between loading gauge and track gauge. 1435mm, or 8'4.5", may be the standard track gauge and uniform between most countries, but the actual standard loading gauges varies quite dramatically between different countries. The UK has a positively tiny loading gauge (about 2.5m), while the "standard" EU loading gauge is 3.1m, the US is slightly wider at 3.2m, and Sweden opts for a roomy 3.4m.
The original notion that the SRB has to be "slightly" larger than the 1435mm track gauge is ridiculous, since as you can see, the loading gauge is over twice the width the track itself. Of course, there's also rooms to play tricks here if you need to. Schnabel cars can laterally offset the loads they're carrying by up to a meter or so, which gives you some extra height to play with if you're dealing with a double-track tunnel with an arched roof.
The other point to mention that's completely omitted is there is a rather big difference between rail carriages and horse-drawn carriages. On rail carriages, the cars sit above the wheels, while horse-drawn carriages sit between the wheels. Something like this:
(Note that this is one of the reasons that the standard gauge was chosen over a broader gauge).As for why standard gauge was chosen, well, keep in mind that a) there was a lot of variance of track gauge in the 1800s, and b) the actual tolerance of track gauge was pretty damn massive. A track gauge differing by as much as two inches or so wouldn't have impeded the ability to transfer rolling stock. Stephenson built the first practical steam locomotive on a line that was 4'8", and so people who wanted to reuse his locomotives would have been drawn to a gauge around 4'8"-4'9". Gauges around 5" and slightly larger were selected in part to prevent interconnect between different railroads (basically, the same concept as vendor lockin).
Contrary to the Snopes article, the South mostly standardized on 5' gauge, but it didn't have as dense a network as the North did. The South eventually converted to 4'9" gauge (the Pennsylvania Railroad's gauge)--in 1886, 20 years after the Civil War. Conversion thence to 4'8.5" took much longer, but was much less dramatic because 4'9" and 4'8.5" are within gauge tolerance.
But I guess that will only happen once under pressure of chinese overlords. I'm sure the day will come, so why wait?