“Installing a new neural network library is typically a tremendous pain because of onerous dependencies, python version hell, complicated Makefiles, and massive code bloat.”
Finally people have started speaking up and doing something about it! May we see more of such mentality in the future, and computer industry will be a better place to work!
Bullshit. These days installing any major NN framework is pretty much just "pip install X". Unless you want GPU support, in which case it's just slightly more complicated.
Developers still underestimate how much code bloat and dependency proliferation harm security and administration.
Static linking encourages even more bloat by vendorizing dependencies.
Please make your applications modular. Optional dependencies are there for this reason.
Minimal dependencies; people are starting to get fed up with having to bootstrap and/or compile the planet just to be able to run some software. The situation has gotten completely out of hand, especially if one is not on GNU/Linux.
Generally when I see links to github projects on HN I expect something useful or extraordinary and this looks like my Neural Networks 101 assignment, only far more polished. But people seem to like it(over 400 stars right now) so maybe I am missing something.
People like it because it does what it says on the tin without any further dependencies and having to pull in the entire planet, as with the respective node.js binaries.
So... if its usefulness is equal to something you've made, AND it's available to the public, AND it's more polished... why exactly are you shitposting about it?
I just pointed out that this is a very basic code. It's definitely not something I would use in any real life scenario(see other posts itt). And if you want to learn about NN you'd be better off googling something like 'backpropagation example' as this project does is virtually not documented. I personally don't find it useful and it takes up the space that can be filled with more interesting content. How is that 'shitposting'?
One of the core requirements of modern deep neural networks is performance. The high accuracy you get from very fast training over large amounts of data is the only reason to look at neural networks over very fast, very accurate systems like XGBoost.
If something doesn't deliver that, then it's worth pointing out that it is inadequate for anything but toy tasks.
It's not really worth pointing this out when the thing in question is a 200 line C library. People won't be expecting it to save the world and/or butter their toast.
+1 also these tiny libraries are great for learning the basics. All of the code has actual meaning behind it and it's super easy to get it up and running.
Please make your point without gratuitous inflammation like “shitposting”. Its effect on the threads should be apparent, and we're trying for better here.
When I'm looking for libraries to use on my Cortex M0 and M3 embedded devices, I prefer small code size to fast execution (as long as it's fast enough) and don't have SIMD and no use for threads. Sometimes size does matter greatly.
The included test script demonstrates that it can learn to recognise the semeion digits with reasonable accuracy in under a second. I've tested it on the pendigits data (8 points to describe a digit) and it can learn to recognise those with 93% accuracy in 1.5 seconds. What more do you want from 200 lines of code?
Writing clean C loops are a good way of writing vectorized code.
Checking the vectorization report of icc 2018 for fprop() shows that the loops got inlined, unrolled and vectorized (with AVX512 and AVX2 instructions using XMM/YMM registers). I doubt that doing it by hand would be faster, but it would make the code a lot more complex.
You have a point on the parallelization part though, it can be cleanly parallelized with a few OpenMP Pragmas.
A thing of beauty happens when you write simple code in C and C++: the compiler has a field-day optimizing ad-nauseum. look at all those zmm(AVX-512) registers being used if your architecture supports it (on top of all the inlining and unrolling)!
godbolt link of the source + assembly -- all I did was combine the header, source and test files:
It's commonly optimized out, yes. Rule of thumb is typically to pass pointers to a struct if it gets fairly large and complex. Even then, you should really profile to see that the becomes a bottleneck, compilers inline aggressively and a pointer might defeat some other optimizations down the call chain.
It might get optimized by inlining, these functions are pretty small and would make good candidates.
Passing small structs by value isn't as expensive as it once was, modern ABIs pass the initial fields in registers. However, this struct does have ten pointer/integer fields, and I'm not aware of a mainstream ABI that allows that many. x86-64 allows up to six on Linux, IIRC. A quick check of the AArch64 ABI says it takes up to eight. So yeah, there will be stack traffic involved if these function calls are not inlined.
It might be easier to answer if you explained why you think it would be better.
There may be cases where by-reference (yes, that's an absolutely acceptable thing to say, even in C) is more efficient: If the struct is a kind of "this pointer" (as in object-oriented programming) that is passed to a bunch of functions but relatively rarely used, then passing it by reference causes much less register pressure. Passing it by value would mean occupying a lot of registers with values that are probably not accessed.
On the other hand, if you expect the callee to do actual computation on all or most of the fields of the struct you pass in, passing it by value should probably be the way to go. This way, the values to be processed are already in registers and don't have to be stored to the stack by the caller and loaded back from the stack by the callee.
There is no general answer that is uniformly best, it depends on the characteristics of the actual program and their interactions with the compiler's optimizations. With inlining in particular, which should often allow the compiler to generate the same code for both versions.
> There may be cases where by-reference (yes, that's an absolutely acceptable thing to say, even in C)
I don't want to be nit-picking. But I would like to know why "by-reference" is an acceptable thing to say.
When I learnt C, C++, Java, etc. I was repeatedly told that pass by reference exists only in C++. In C, a pointer is passed by value. In Java, a reference is passed by value.
Doesn't calling "pass pointer by value" as "pass by reference" lead to confusion and distortion of the true meaning of "by reference"?
What do you do when you want to find the value a pointer points to? You dereference it of course!
C++ and Java have their own specific language-lawyer definitions of the word reference, but outside of those languages, "reference" is a catch-all term that includes pointers, handles, and indexes.
C implements the concept 'pass by reference' using the mechanism of 'pointer passed by value'.
You have no problem using the terms 'loop' or 'recurse' to describe C code, right? But these high-level ideas are implemented without using those terms in the code. Same with 'pass by reference'.
It is fair to say KANN "only applicable for rather small problems". However, KANN is not so "similar" to Tinn. Tinn only implements MLP with a single hidden layer. KANN supports a lot more features such as 1D/2D convolution, RNN/LSTM, weight sharing and mini-batching. It is also more optimized.
You're absolutely right that kann is more optimized and full-featured. I thought to mention your library because it is also a very portable neural network library written in C.
kann is a stunning piece of engineering, and your code is magnificent, and that is precisely why I mentioned it.
I apologize, I did not clearly express my intent.
What exactly is the output of the example in `test.c`? I.e. what do the numbers mean? The error in the prediction I guess, but the prediction of what? What is the first entry in the semion data set? a "0"?
61 comments
[ 3.3 ms ] story [ 113 ms ] threadQ. It seems that it only does 1D feature vectors?
A. Convolution and other inductive biases are only necessary when you have small data.
The third line of the readme says "We are finally at our April 1 Release (v4.1.2018)." The date may be relevant. ;)
Finally people have started speaking up and doing something about it! May we see more of such mentality in the future, and computer industry will be a better place to work!
"We are finally at our April 1 Release (v4.1.2018)."
Disclaimer: I'm a contributor ;)
Please make your applications modular. Optional dependencies are there for this reason.
Most good packaging tools support them.
Essentially this library is a demo toy that will need a lot to work to make useful in real conditions.
It's 200 lines of C, what were you expecting?
Let people enjoy things.
One of the core requirements of modern deep neural networks is performance. The high accuracy you get from very fast training over large amounts of data is the only reason to look at neural networks over very fast, very accurate systems like XGBoost.
If something doesn't deliver that, then it's worth pointing out that it is inadequate for anything but toy tasks.
[1] http://neuralnetworksanddeeplearning.com/chap1.html#implemen...
Checking the vectorization report of icc 2018 for fprop() shows that the loops got inlined, unrolled and vectorized (with AVX512 and AVX2 instructions using XMM/YMM registers). I doubt that doing it by hand would be faster, but it would make the code a lot more complex.
You have a point on the parallelization part though, it can be cleanly parallelized with a few OpenMP Pragmas.
godbolt link of the source + assembly -- all I did was combine the header, source and test files:
Passing small structs by value isn't as expensive as it once was, modern ABIs pass the initial fields in registers. However, this struct does have ten pointer/integer fields, and I'm not aware of a mainstream ABI that allows that many. x86-64 allows up to six on Linux, IIRC. A quick check of the AArch64 ABI says it takes up to eight. So yeah, there will be stack traffic involved if these function calls are not inlined.
There may be cases where by-reference (yes, that's an absolutely acceptable thing to say, even in C) is more efficient: If the struct is a kind of "this pointer" (as in object-oriented programming) that is passed to a bunch of functions but relatively rarely used, then passing it by reference causes much less register pressure. Passing it by value would mean occupying a lot of registers with values that are probably not accessed.
On the other hand, if you expect the callee to do actual computation on all or most of the fields of the struct you pass in, passing it by value should probably be the way to go. This way, the values to be processed are already in registers and don't have to be stored to the stack by the caller and loaded back from the stack by the callee.
There is no general answer that is uniformly best, it depends on the characteristics of the actual program and their interactions with the compiler's optimizations. With inlining in particular, which should often allow the compiler to generate the same code for both versions.
I don't want to be nit-picking. But I would like to know why "by-reference" is an acceptable thing to say.
When I learnt C, C++, Java, etc. I was repeatedly told that pass by reference exists only in C++. In C, a pointer is passed by value. In Java, a reference is passed by value.
Doesn't calling "pass pointer by value" as "pass by reference" lead to confusion and distortion of the true meaning of "by reference"?
C++ and Java have their own specific language-lawyer definitions of the word reference, but outside of those languages, "reference" is a catch-all term that includes pointers, handles, and indexes.
You have no problem using the terms 'loop' or 'recurse' to describe C code, right? But these high-level ideas are implemented without using those terms in the code. Same with 'pass by reference'.
Structs on the stack I pass by value. Structs on the heap I pass by pointer. It's a mental thing that I find helps with large codebases.
- a student trying to get a better grasp on neural networks
[0] https://github.com/attractivechaos/kann
kann is a stunning piece of engineering, and your code is magnificent, and that is precisely why I mentioned it. I apologize, I did not clearly express my intent.
https://github.com/codeplea/genann
The second row is the probability of each given the input.
e.g. This output means it's 98.6% certain that the input represents the digit 7.
https://github.com/100/Cranium