Fwiw, a deep-learning startup that's doing some pretty cool satellite imagery analysis is Orbital Insight http://orbitalinsight.com/. So deep nets are all over deep space...
I wonder if this can be used to study impacts of urban expansion and how natural ecosystems around big cities get affected. Also it can be a great way of exploring natural resources conservation.
What interesting about this is that it is appears to be the next wave of PRACTICAL deep learning that can start in image recognition and move the same models into other areas. CNNs have limited (for now) use cases, this can actually be applied to some very large-scale problems using the single layer approach (which is the difference) and scale to massive sets. Very cool. Very.
Thanks, interesting. Have been doing the same with hyperspectral aerial images. Did these guys open source their trained weights or any source code? Couldn't see any links
I have some doubts about this. Deep learning moves fast and DBNs are pretty much outdated models, even for unsupervised pre-training. It doesn't make much sense to me that unsupervised pre-training would help for this problem to begin with, seeing as their dataset totals around 65TB.
The paper is worth checking out: http://arxiv.org/abs/1509.03602
I haven't read it in full, but based on a quick skim, the convnet architectures they evaluated seem laughably tiny and shallow (at most three convolutional layers) by today's standards -- although I appreciate that there may be other constraints at play here (limits on training time etc.).
But to claim that DBNs are better suited for this problem than convnets based on these results is quite far-fetched. I'm confident that a convnet could crush these results, given enough effort and time spent on hyperparameter tuning.
I find this part particularly misleading (section 6, page 13): "shape/edge based features which are predominantly learned by various Deep architectures are not very useful in learning data representations for satellite imagery. This explains the fact why traditional Deep architectures are not able to converge to the global optima even for reasonably large as well as Deep architectures."
The whole point of learning features is so that they are better suited for the task at hand. If "shape/edge based features" are not suitable to perform a particular task, then a properly trained convnet should not learn them. I think the conclusions drawn from this work would have been very different if the chosen network architectures were more sensible.
+1. There are several fishy statements throughout this paper. Another one in conclusion:
"For satellite datasets, with inherently high variability,
traditional deep learning approaches are unable to converge to a global optima even with significantly big and deep architectures."
this quote points to some basic misunderstandings of how/when these models work. "Inherent high variability" is suddenly some kind of a problem? Unable to converge to a global optima? The modern view of the deep net optimization landscapes based on several recent studies argue against these outdated interpretations.
I just downloaded the dataset, and color is such a powerful feature that training a random forest on images downsampled to a single pixel results in 95% and 98% accuracies! (for the 4-category and 6-category versions, respectively)
And you can easily exceed 99.5% by adding more features to the forest, which is far above their DBN accuracy.
I have no idea how they were able to get an accuracy as low as 69% when they evaluated random forests.
I read the paper, and I also have some reservations. The procedure they used to extract and randomize their data seems biased towards large homogenous areas.
In short, in their procedure, it seems possible to rope off a large contiguous area of Mojave desert, ground-truth it using their GUI system as "barren", and have that area be carved up into 28x28 pixel chips and spread equally into the training and test sets.
In such a case, the training and test sets are not really independent. And their 6 classes, as you point out, are amenable to color features.
Having done classification of remote sensing data...the above is not a good test of accuracy at any useful task. You have to test accuracy on representative data.
That means training within a few areas, and testing on geographically distant but ecologically similar areas. (I.e., same class, but statistically independent.). And, varying things like time of day, observing geometry, and seasonality. Color features will be quite fragile in such tests.
And, testing on a more diverse sample, to see if "none of the above" can be detected, because their class decomposition is nothing like exhaustive.
Since you seem fond of Deep Learning projects, what do you think something like automatic classification of streets (and transportation network in general) from imagery is viable yet? Seems like it would be useful for OpenStreetMap. The corpus of valid classification is tremendous (pretty much all (>95%?) of NA is classified , and the data is readily available.
The subjects themselves don't seem too complex either: lines are small roads, thick lines are major ones, and then there's intersections which semantically interlink them.
I was wondering about this also, especially for the case of Humanitarian OpenStreetMap where they map e.g. West Africa and allow you to map without visiting the area (normally not allowed on OSM). The maps can so sparse before we map a region, that any 'AI' would not have to be perfect - it would anyway be a vast improvement on what already exists.
Maybe a good option would be a mapping tool for humans, that traced e.g. a building and then said to the user 'I think this is a building, press Yes to accept'. That would speed up my mapping times by maybe a factor of 5, especially once I got comfortable with the AI being reliable, and could click Yes after just a cursory check.
Right, human assist would probably be needed for final verification and unfortunately it's impossible to correctly name the streets (unless everywhere were like Manhattan); number might be doable.
It just seems like a perfect fit for Deep Convolutional neural nets.
I stopped reading when I hit this sentence: But armed with two of these, and a new slant on deep belief networks, they are breathing new life into an established deep learning knowledge base and proving how a new model for deep learning is proving itself at scale for NASA’s terabytes of near-range satellite data—and potentially for other areas as the work expands.
Being one of the authors of this paper, I think its my responsibility to clarify some of the doubts raised by some of you in this thread.
Firstly, I would like to thank everyone for taking the time to go through the paper and raising your questions. I think most of these are valid questions and I will try to answer them here.
First of all, regarding the dataset creation method and the chances of having overlapping patches between training and test sets - as we mentioned in the paper, care was taken to avoid this specific scenario where a homogeneous landcover area was first sampled and then 28x28 patches extracted from it occur both in training and test sets. For this, once we selected the set of 1500 tiles from the NAIP dataset, we separated the tiles into spatially/geographically distant areas and then used these two areas for extracting the training and test patches respectively.
Now, regarding the comparison with raw pixel based DBN and CNN. As benanne suggested that we claimed that DBN is better suited than CNN for this task, in contrast, I would like to clarify that in the paper, we claim that the integration of Haralick features with the DBN yields better results than either of DBN or CNN with raw pixels. In fact, validating the popular belief, we show that DBN alone is indeed inferior to CNN on this task same as what benanne states. Now, the question arises - why didn’t we use bigger CNNs with higher number of layers - may be 5 or 6 for the experiments? Here, I would like to point everyone to the paper “ImageNet Classification with Deep Convolutional Neural Networks” where the authors trained a deep CNN with 5 convolutional layers and 3 fully connected layers. But we can see that the dataset used in that paper consists of 256x256 images. As the theory of CNN suggests, a CNN models the human visual cortical system that builds a hierarchical model of the image in such a way that bigger images require deeper nets to encode all the contextual dependencies between neighboring pixels and to perform a hierarchical clustering from the features to labels. So, in contrast to their dataset, we see that the images in our dataset are only 28x28 which means that we need to scale down our model to avoid significant overfitting which can’t even be avoided with L2 regularization or Dropout. Also, if we consider results on another object recognition dataset, namely CIFAR-10, which is 32x32 (roughly equal in size to our images), most of the state-of-the-art results on this dataset (like Deeply Supervised nets and Network in Network) uses an architecture with 3 convolutional layers and 1 fully-connected layer same as the one we considered. Also, in the Imagenet classification paper, while using the significantly being and Deep networks, in order to avoid overfitting the authors use techniques like data augmentation and dropout. So, effectively the model would be bigger and take longer to train while the integration of the texture based features with the Deep Belief Network saves us this overhead which is significant for a 65 TB dataset (It should be noted that our goal in this research is not just to handle this dataset but to use this as a pilot to develop an algorithm that can scale for landcover classification to the whole of continental US).
Now someone might ask, why we chose 28x28 in the first place? Its because an important difference between satellite image classification and object based classification is that given an image with a bigger context, we say if a particular object is present in a scene or not (for e.g., CIFAR, Imagenet etc.). On the other hand for satellite imagery, what we need is a near accurate per-pixel labelling. So, choosing a smaller window removes most of the contextual information required for a per-pixel scene classification and choosing a bigger window means loosing the statistical properties of the object of interest - e.g., a tree or bush might be much smaller than a 64x64 window which covers a spatial area of 64mx64m. To conclude, I would like to stress the fact that out of...
I don't think overfitting will be an issue (at all!) on a 65TB dataset. A bigger CNN model should be more effective at this task than a DBN, (almost) regardless of the features added. If we can use convolutional generative models (from raw noise -> images) to make this kind of stuff [1][2] I see no reason why it shouldn't be super effective to classify with a CNN.
The equivalent DBN-type generative model for CIFAR10 is pretty far behind [3], but not terrible. It is believable that a DBN could do well on this task, but it would be very, very surprising if a DBN beats a well trained CNN of the appropriate capacity in anything related to classification.
All that said, some of the other comments have highlighted larger potential issues than model choice here - achieving up to 98% accuracy using single pixels, random forest, etc seems to point to potential issues in the dataset that will block any kind of model evaluation or further research.
I would look for data leakage, and reconsider CNNs in your future work - especially something using larger patches and VGG style features. With respect to your comment on scalability - if the model converges and can't really learn more (small/limited capacity network) processing more training data is just a waste of time. Convergence is not really epoch based, but update based, especially in big datasets like this.
A small network will be very fast to apply but if that is the goal there are lots of papers on model approximation - a big ensemble of networks that are engineered and model approximated to fit your compute budget might work better than limiting the network capacity initially.
All said - does it really matter much if you get 95% vs. 97% vs. 98% vs. 99.99%? Aren't there meta techniques like CRF to resolve occasional blips in model prediction for neighboring patches anyways? Maybe a linear model with decent features or random forest + follow-on cleanup will work better with the constraints necessary.
I am all about neural networks for most things, but if you have a serious computational constraint linear models (or random forests) are stupid fast [4], and pretty good on many tasks. Adding on a "meta model" to resolve anomalous errors with this could be good enough for your task. Just something to consider.
I want to share some quick results I had from trying out the dataset. First off, I want to say thanks for your work and the dataset. I'm glad to see more people getting excited about machine learning in this field!
I used a simple Lenet-like architecture to train both Sat4 and Sat6. This seemed reasonable, especially considering this network works well on the 28x28 patches for MNIST. After only 10k iterations, I achieved accuracies of 98% on Sat4 and 97% on Sat6. I haven't gone back yet, but I assume these results could be increased with tweaking the network slightly or playing around with learning parameters. I also spent a while looking for any bugs in my evaluation, but things seemed to check out. This would suggest that CNN may be able to at least match, if not potentially outperform (97% vs 94% on Sat6) the methods in the paper. I'm curious to hear if anyone else has gotten similar results with CNN or other methods? I'm also curious to know if you tried other CNN architectures other than those described in the paper?
19 comments
[ 4.1 ms ] story [ 58.9 ms ] thread(I'm aware of them because I work on another deep learning project, http://deeplearning4j.org)
The paper is worth checking out: http://arxiv.org/abs/1509.03602 I haven't read it in full, but based on a quick skim, the convnet architectures they evaluated seem laughably tiny and shallow (at most three convolutional layers) by today's standards -- although I appreciate that there may be other constraints at play here (limits on training time etc.).
But to claim that DBNs are better suited for this problem than convnets based on these results is quite far-fetched. I'm confident that a convnet could crush these results, given enough effort and time spent on hyperparameter tuning.
I find this part particularly misleading (section 6, page 13): "shape/edge based features which are predominantly learned by various Deep architectures are not very useful in learning data representations for satellite imagery. This explains the fact why traditional Deep architectures are not able to converge to the global optima even for reasonably large as well as Deep architectures."
The whole point of learning features is so that they are better suited for the task at hand. If "shape/edge based features" are not suitable to perform a particular task, then a properly trained convnet should not learn them. I think the conclusions drawn from this work would have been very different if the chosen network architectures were more sensible.
"For satellite datasets, with inherently high variability, traditional deep learning approaches are unable to converge to a global optima even with significantly big and deep architectures."
this quote points to some basic misunderstandings of how/when these models work. "Inherent high variability" is suddenly some kind of a problem? Unable to converge to a global optima? The modern view of the deep net optimization landscapes based on several recent studies argue against these outdated interpretations.
I just downloaded the dataset, and color is such a powerful feature that training a random forest on images downsampled to a single pixel results in 95% and 98% accuracies! (for the 4-category and 6-category versions, respectively)
And you can easily exceed 99.5% by adding more features to the forest, which is far above their DBN accuracy.
I have no idea how they were able to get an accuracy as low as 69% when they evaluated random forests.
In short, in their procedure, it seems possible to rope off a large contiguous area of Mojave desert, ground-truth it using their GUI system as "barren", and have that area be carved up into 28x28 pixel chips and spread equally into the training and test sets.
In such a case, the training and test sets are not really independent. And their 6 classes, as you point out, are amenable to color features.
Having done classification of remote sensing data...the above is not a good test of accuracy at any useful task. You have to test accuracy on representative data.
That means training within a few areas, and testing on geographically distant but ecologically similar areas. (I.e., same class, but statistically independent.). And, varying things like time of day, observing geometry, and seasonality. Color features will be quite fragile in such tests.
And, testing on a more diverse sample, to see if "none of the above" can be detected, because their class decomposition is nothing like exhaustive.
The subjects themselves don't seem too complex either: lines are small roads, thick lines are major ones, and then there's intersections which semantically interlink them.
Maybe a good option would be a mapping tool for humans, that traced e.g. a building and then said to the user 'I think this is a building, press Yes to accept'. That would speed up my mapping times by maybe a factor of 5, especially once I got comfortable with the AI being reliable, and could click Yes after just a cursory check.
It just seems like a perfect fit for Deep Convolutional neural nets.
Firstly, I would like to thank everyone for taking the time to go through the paper and raising your questions. I think most of these are valid questions and I will try to answer them here.
First of all, regarding the dataset creation method and the chances of having overlapping patches between training and test sets - as we mentioned in the paper, care was taken to avoid this specific scenario where a homogeneous landcover area was first sampled and then 28x28 patches extracted from it occur both in training and test sets. For this, once we selected the set of 1500 tiles from the NAIP dataset, we separated the tiles into spatially/geographically distant areas and then used these two areas for extracting the training and test patches respectively.
Now, regarding the comparison with raw pixel based DBN and CNN. As benanne suggested that we claimed that DBN is better suited than CNN for this task, in contrast, I would like to clarify that in the paper, we claim that the integration of Haralick features with the DBN yields better results than either of DBN or CNN with raw pixels. In fact, validating the popular belief, we show that DBN alone is indeed inferior to CNN on this task same as what benanne states. Now, the question arises - why didn’t we use bigger CNNs with higher number of layers - may be 5 or 6 for the experiments? Here, I would like to point everyone to the paper “ImageNet Classification with Deep Convolutional Neural Networks” where the authors trained a deep CNN with 5 convolutional layers and 3 fully connected layers. But we can see that the dataset used in that paper consists of 256x256 images. As the theory of CNN suggests, a CNN models the human visual cortical system that builds a hierarchical model of the image in such a way that bigger images require deeper nets to encode all the contextual dependencies between neighboring pixels and to perform a hierarchical clustering from the features to labels. So, in contrast to their dataset, we see that the images in our dataset are only 28x28 which means that we need to scale down our model to avoid significant overfitting which can’t even be avoided with L2 regularization or Dropout. Also, if we consider results on another object recognition dataset, namely CIFAR-10, which is 32x32 (roughly equal in size to our images), most of the state-of-the-art results on this dataset (like Deeply Supervised nets and Network in Network) uses an architecture with 3 convolutional layers and 1 fully-connected layer same as the one we considered. Also, in the Imagenet classification paper, while using the significantly being and Deep networks, in order to avoid overfitting the authors use techniques like data augmentation and dropout. So, effectively the model would be bigger and take longer to train while the integration of the texture based features with the Deep Belief Network saves us this overhead which is significant for a 65 TB dataset (It should be noted that our goal in this research is not just to handle this dataset but to use this as a pilot to develop an algorithm that can scale for landcover classification to the whole of continental US). Now someone might ask, why we chose 28x28 in the first place? Its because an important difference between satellite image classification and object based classification is that given an image with a bigger context, we say if a particular object is present in a scene or not (for e.g., CIFAR, Imagenet etc.). On the other hand for satellite imagery, what we need is a near accurate per-pixel labelling. So, choosing a smaller window removes most of the contextual information required for a per-pixel scene classification and choosing a bigger window means loosing the statistical properties of the object of interest - e.g., a tree or bush might be much smaller than a 64x64 window which covers a spatial area of 64mx64m. To conclude, I would like to stress the fact that out of...
The equivalent DBN-type generative model for CIFAR10 is pretty far behind [3], but not terrible. It is believable that a DBN could do well on this task, but it would be very, very surprising if a DBN beats a well trained CNN of the appropriate capacity in anything related to classification.
All that said, some of the other comments have highlighted larger potential issues than model choice here - achieving up to 98% accuracy using single pixels, random forest, etc seems to point to potential issues in the dataset that will block any kind of model evaluation or further research.
I would look for data leakage, and reconsider CNNs in your future work - especially something using larger patches and VGG style features. With respect to your comment on scalability - if the model converges and can't really learn more (small/limited capacity network) processing more training data is just a waste of time. Convergence is not really epoch based, but update based, especially in big datasets like this.
A small network will be very fast to apply but if that is the goal there are lots of papers on model approximation - a big ensemble of networks that are engineered and model approximated to fit your compute budget might work better than limiting the network capacity initially.
All said - does it really matter much if you get 95% vs. 97% vs. 98% vs. 99.99%? Aren't there meta techniques like CRF to resolve occasional blips in model prediction for neighboring patches anyways? Maybe a linear model with decent features or random forest + follow-on cleanup will work better with the constraints necessary.
I am all about neural networks for most things, but if you have a serious computational constraint linear models (or random forests) are stupid fast [4], and pretty good on many tasks. Adding on a "meta model" to resolve anomalous errors with this could be good enough for your task. Just something to consider.
[1] https://twitter.com/AlecRad/status/645830923150299136
[2] http://soumith.ch/eyescream/
[3] http://www.icml-2011.org/papers/591_icmlpaper.pdf
[4] https://github.com/ajtulloch/sklearn-compiledtrees/
I used a simple Lenet-like architecture to train both Sat4 and Sat6. This seemed reasonable, especially considering this network works well on the 28x28 patches for MNIST. After only 10k iterations, I achieved accuracies of 98% on Sat4 and 97% on Sat6. I haven't gone back yet, but I assume these results could be increased with tweaking the network slightly or playing around with learning parameters. I also spent a while looking for any bugs in my evaluation, but things seemed to check out. This would suggest that CNN may be able to at least match, if not potentially outperform (97% vs 94% on Sat6) the methods in the paper. I'm curious to hear if anyone else has gotten similar results with CNN or other methods? I'm also curious to know if you tried other CNN architectures other than those described in the paper?