I mean, yes, Boost itself is big. But when I think a heavyweight dependency, I think of something that itself pulls in a shitton of dependencies Javascript-style. And AFAIK Boost doesn't have any dependencies.
I’m afraid that’s a very JS viewpoint. A dependency doesn’t need dependencies itself to be heavy.
Start using some of the heavier weight sub-libraries and you can quickly explode your compile times. I have boost beast/asio on one project and it takes over a minute to compile one of the source files…
Boost is (was?) considered to be such a heavy dependency (especially in the days before distribution package managers and build systems got their act together and collaborated or at least acknowledged the reciprocal efforts of one another) that it's probably single-handedly responsible for easily at least half of the so-called "NIH" libraries/ad-hoc solutions I encountered for a particular decade of my career as a C++ developer.
At one point, I had a curated SVN repository of certain boost header-only includes that didn't require any sort of build integration, had no dependencies of their own (even other boost libraries), and didn't require any build system integration - and I know I wasn't the only one.
The world of native software development was just so unbelievably different from that of web development - it's really no surprise developers from that lived through that era have such a hard time accepting the modern-day age of PWAs in Electron sandboxes masquerading as native programs. People that cut their teeth developing desktop software with something like Python (just as an example) instead would have a completely different perspective on all this, of course.
It's worse than that. The GitHub title is "Folly: Facebook Open-source Library". The bolded sentence in the readme is "Folly contains a variety of core library components used extensively at Facebook". In fact, the entire first paragraph is about how it's used at Facebook.
It takes about a minute to find any information at all about this library besides the fact that it's used at Facebook (by the way, did you miss that? it's used at Facebook! Facebook!!). And even that information is empty bromides about how it's "designed with practicality and efficiency in mind". Hmm, I prefer my software to be impractical and inefficient...
All I take away from this is that Facebook has an irrationally optimistic expectation of how the world views its engineering prowess.
People are so grouchy over even mildly ambiguous titles.
Right in the intro section, second paragraph, it says "It complements (as opposed to competing against) offerings such as Boost and of course std". Come on, what more do people want?
Rule one is don't waste people's time. Waiting until the second paragraph to get to the point violates that rule. And that's assuming your reader knows what boost or std are.
Instead of pointing a finger at Facebook, I'd scoff at C++ and the way in which the language has grown where orgs build up their own utility libraries rather than depend on a standard library.
That said, this is an issue for other languages as well. Another example: the "Google core libraries for Java": https://github.com/google/guava
You mean because it has gotten so unreadable and vast?
I've never worked with a place using C++ that didn't make their own core libraries. As a low-level language, there's almost always a way to get some performance benefit by starting from scratch for your specific case.
I've also never worked in a C++ place that didn't make their own core libraries, but when you ask, their rationale for building and maintaining their own libraries (as opposed to using the Standard Library) is often very weak. It's usually based on very outdated assumptions and Cargo Cutting, such as "the STL[1] is slow (but we haven't profiled ours to prove it's faster)." Or "We need our own string type because decades ago, there was something that std::string didn't do." Or "Our founding developer didn't like templates, so he build this vast copy of the standard library but using explicit data types."
(yes I've heard all of these "reasons")
1: Just calling it "the STL" is a key sign of living in the past--pretty much nobody uses the actual STL anymore: What we use is called the C++ Standard Library.
Often these libraries exist as glue to an existing ecosystem. To that end, I have also seen portability libraries that provide a single interface to multiple implementations of system functions or datatypes on different operating systems.
> pretty much nobody uses the actual STL anymore: What we use is called the C++ Standard Library.
I won't deny your prior, but my prior is that people who complain about the use of the word "STL" to mean "C++ Standard Library" (and ignore that the latter is 10 times as many keystrokes) do so only to flaunt their knowledge, not because anyone actually experiences confusion nowadays.
To me, the STL is specifically Stepanov's idea for how to do generic programming in C++ in the early 1990s, and so it makes sense to distinguish the STL from unrelated stuff that happens to be in the C++ Standard Library.
For example iostream isn't part of Stepanov's idea, Stroustrup created that bundle of craziness in the 1980s and today it's part of the Standard Library, but it was not part of STL which only came into existence in the 1990s (although Stepanov's ideas about generic programming are older).
But why is that an issue at all ? Having boost and other utilities is great. And since it's all templates there's in practice in your binary no added cost if you're using folly::foo or std::foo or boost::foo, you can pick and choose the exact data structures that allow exactly what you want (or write your own if you think that you can do a few percent better than folly of abseil in your specific use case).
If I was using java or python I'd definitely not use the standard things most of the time either because it's more than likely that there's a more efficient way in some other library, and because my main paycheck comes from "can you make this as fast as technically possible given this average consumer hardware"-problems
boost::variant will allocate to save state during construction of a new value type, in order to be able to roll back if the constructor throws an exception.
Why? Having millions of libraries is a good dilemma to have, even if it means a bunch don't work well together so you make your own anyway. Better than having no choice.
Instead of pointing the finger at C++, I'd scoff at any language that promotes the belief that a standard library could possibly meet all the needs of a project.
> I'd scoff at C++ and the way in which the language has grown where orgs build up their own utility libraries rather than depend on a standard library.
This comment makes no sense at all. I mean, would it make any sense to also criticize JavaScript/NodeJS for their relatively spartan standard modules and extensive use of vended modules from the likes of npm?
And have we learned nothing from the mistake of going with the "everything and the kitchen sink" that is java which failed to actually satisfy basic needs and resulted in projects like guava and apache commons and vavr stepping in to provide alternatives? And how about C's infamous standard library with it's security vulnerability-inducing standard functions which remain in the standard?
Perhaps we should learn lessons and avoid basic mistakes such as setting libraries in stone. C++ learned that lesson, and so did C#/.NET, and up to a point Rust. Why unlearn them?
> would it make any sense to also criticize JavaScript/NodeJS for their relatively spartan standard modules and extensive use of vended modules from the likes of npm?
>> And even that information is empty bromides about how it's "designed with practicality and efficiency in mind". Hmm, I prefer my software to be impractical and inefficient...
the opposite from practical is general. for example is std not practical but general. Its api is not optimized for 95% of the use cases but rather so that everything is possible and equaly verbose.
the opposite of efficient can for example be trade of for readability, portability and cost (as in development time).
For me as a possible user of the lib i found this information critical. I write always similar info in the readme for my libraries.
No, it's certainly not at all. Those are completely unrelated spectrums. Something can easily be practical and general, or impractical and specific.
> the opposite of efficient can for example be trade off for [...]
And if they stated what trade-offs they made, then that paragraph would actually start to say something useful. Unfortunately, they don't.
Also, if you're developing documentation for libraries, I really would recommend finding a better source of inspiration. Take Redis, in the very first paragraph:
> What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.
That's all you need to do: before anything else, tell the user what the thing does. Then you can give some basic details about implementation: "written in Go", "runs on Linux & macOS", "uses GC, so not suitable for RT applications", etc. After that you can get into trade-offs, real narrow implementation details, setup, handwaving abstract adjectives, etc. But lead with what the hell it actually does.
Looks like that's because the library isn't for anything specific, other than general C++ programming at Facebook. They might worded it a bit better, like "core library" or common library" or something.
I agree, but HN policy is to use the title from the link verbatim and IME even slight editorializing to provide context tends to get reverted by the moderators
This[1] is a c-flavoured c++ STL replacement, uses minimal number of c++ features to be easy to use yet simple. We built it and use it internally where I work.
It's not c but almost c, it could be useful for you
Thank you. You don't seem to provide a c_str() equivalent (a method that gives a pointer to or copies out a C-string) or views ? Sorry if I missed them.
This is still all in c++ correct? why not just using STL instead, am I missing something here.
There are a few pure-C made STL alike containers for C that uses no c++ code at all, a random github search finds this: https://github.com/assyrianic/Harbol , there are quite a few of them just not recalling them now.
It looks like part of it is a similar design to something I came up with myself.
Do you know if Synchronized<> protects against accidental recursive locking?
There is no explicit protection for recursive locking. In practice, write locks will deadlock, and read lock under read lock will be fine.
However, the API encourages practices that make it very hard to recursively acquire the lock (Synchronized is about access to objects, not critical sections).
Having reviewed even just a few diffs with Synchronized since joining Facebook, it absolutely should not be part of the standard library (nor, I would argue, part of folly). I have found far too many cases where engineers have threading issues in their code because they're saving references to objects after .lock() has returned.
Writing thread-safe code isn't easy, and it isn't made easier by libraries that offer footguns like Synchronized. It's easier to get it right with compiler thread annotations than with Synchronized.
How do compiler thread annotations solve the problem of persisting references to the object? Is static analysis able to detect that? Seems a hard problem (basically escape analysis).
Synchronized cannot guarantee thread-safety, but in my experience wrappers that do not allow unsynchronized access to the object are better than the alternative. Annotations are just another way to guard access.
They don't, but C++ programmers especially are familiar with not letting inappropriate things escape scopes; they're much less familiar with not letting inappropriate things escape _expressions_. I've reviewed over a half million lines of code from engineers of every level of experience, and I've seen far more bugs from the latter than the former. For instance, a former employer had a function similar to folly::get_default, except it returned the default by reference, and engineers were always running afoul of lifetimes using that function. (It was especially egregious with std::string temporaries returned from string literal defaults.)
Thread annotations also effectively prevent unsynchronized access to the object, but they do it in a way that programmers make fewer mistakes with, and they do it with a less error prone mental model: mutexes are seen as guards of _class invariants_, not merely as guards of _data members_. Apart from encouraging a superior mental model, thread annotations also encourage a one-mutex-per-class model which I've found far more successful and error prone (both in code I've reviewed and in code I've written) over a one-mutex-per-member model which inevitably (in my experience) results in fewer bugs.
Not the OP, but active user and contributor to folly, so I can add some context.
folly is a community effort within Meta; it doesn't have a specific theme about what's included in it, but it is a set of core libraries that are likely to be depended on by most C++ software within the company, and are enough high-quality and self-contained that they can be useful externally, or showcase implementations undergoing standardization efforts (for example experimental/coro for coroutines).
Main focus is server applications, but it is also used in mobile, in particular Thrift (the serialization/RPC infrastructure) depends on it.
A very personal list of favorites:
- experimental/coro is the coroutines infrastructure, which is now widely adopted.
- SharedMutex is a heavily optimized mutex which supports shared/upgrade/exclusive semantics, and allows near-linear scalability in shared mode. It is used pretty much everywhere as it is the default mutex for Synchronized, a wrapper to safely synchronize access to objects. Its footprint is just 4 bytes.
- DistributedMutex, OTOH, is an exclusive-only mutex that supports flat combining, and performs extremely well under high contention.
- MPMCQueue and UnboundedQueue are lock-free MPMC queues, respectively bounded and unbounded, which are used in almost all our thread pools (see the executor/ directory)
- Userland implementations of RCU and hazard pointers, for deferred reclamation.
- LifoSem and Baton are other fundamental synchronization primitives that are not available in most core C++ libraries. They do a magic trick internally, where if they're waiting long enough, they unmap the unused suffix of the stack, so idle threads don't hold unused stack memory.
- Function is like std::function, but move-only, so it can hold non-copyable function objects (think lambda that captures unique_ptr). It is the vocabulary type for callbacks used in futures and executors.
- F14 is a SIMD-optimized hash table. Similar to Abseil's SwissTable, it was published around the same time.
- TDigest is a highly scalable quantile estimator, widely used for service counters (see fb303, also open source).
- (shameless plug, as I wrote this one) enumerate() is a replica of the Python function, so you can do
for (auto&& [i, element] : folly::enumerate(collection)) { ... }
This type of libraries often sell themselves as high performance featuring bunch of fancy practices, yet in everywhere I had seen people using them, some of the mostly inefficient code are produced, yes that includes Facebook.
When I write a README for a library I always try to include 1-2 code samples of the highlights of said library. So the reader sees quickly what to expect.
Such a massive lib. Interesting, if not too extensive read.
I understand the FB rationale to have it, but what whould be a reason for other projects to adopt this lib?
I won't imagine maintaining such a lib as a dependency by ourselves. Maybe for one short-term purpose, but then it's still safer to go with standard or boost, or, God forbid, carve out some pieces and wrap them up.
I've considered it for some special things, mainly the F14 hash table because I needed a highly performant implementation (I ended up writing a custom one that takes advantage of domain knowledge to go faster). It seems that it has classes and constructs not available in other libraries. A lot of them are very specific, but at that point having a reliable, tested, fast implementation of a complex implementation might be worth the hassle of bringing in a new dependency.
Yes, in the end I had to go to a custom hashmap/linked list combination to get the best performance for our specific use case (basically a flow table with LRU cache behavior and also expiration of flows by time limits).
I like a lot of things in Folly, but as others have mentioned: you have to pull in quite a lot, even if you care about only a tiny part. It's not very modular in that way.
However, I did have decent success taking the classes I was interested in (CPUThreadPoolExecutor, and surrounding), stripping the library down to just those bits, and using them. Obviously a PITA to stay current with upstream, though.
If nothing else, it's worth reading for some of the highly-optimized stuff going on in there.
The stdlib implementations of associative containers have serious drawbacks. At the very least, most projects should consider adopting absl::flat_hash_(set|map) and absl::btree_(set|map) (or similar replacements) over the stdlib containers.
I am also partial to explicit error handling over exceptions, and there is a lot to like about absl::Status (and absl::StatusOr).
Disclaimer: I work for Google but the views expressed are my own and don't necessarily reflect those of my employer.
I wouldn't start a green field C++ project if I can help it[1], but if I had to I would probably go with the standard library with maybe as little boost as possible.
I've been very interested in native C++ development for web-services and other applications. However the development experience always seems like an after thought. I really don't want to role my own package manager or build process - why isn't there a pip/cargo/mvn etc. for C++ devs?
Personally I find C++ more fun to use than Rust, even though I use Rust as part of my day job. I'd genuinely love to find a full-time C++ gig and spend the rest of my career focused on that single language. There are tons of flaws with the language, sure, but they're documented and known flaws. And best practices to get around them. This really appeals to me.
Funny, I have a full time C++ job, and really don't see the appeal. Rust feels superior in every regards and especially dev experience. Want to exchange jobs ;-)?
Developing in c++ vs (for example) python does seem like more work. I think it is a consequence of offering more control, which is expected in the c/c++ community.
What the two options offer.. Is different. With a higher level language, you can get more work done for less cost [1]. With a lower level language, you get more control —- which is great, but you shoulder that responsibility yourself. And you recognize this already..
If you want to move with the crowd (which the higher level abstractions are likely tailored for), then you will be able to move fast and far. When you need to exert more control, however, things get tricky.
For example managing the OpenSSL library used in your project independently of the system library?
folly [https://www.wordnik.com/words/folly]
fŏl′ē
noun
Lack of good sense, understanding, or foresight.
An act or instance of foolishness.
A costly undertaking having an absurd or ruinous outcome.
At the risk of starting a meta conversation (pun intended) I wonder why the team chose this name for a set of core library components. The GitHub page indicates it is a loose play on an acronym.
“acronymed loosely after Facebook Open Source Library”
Yet why accept a name with a negative connotation.. The name leads the product, so why not at least make it _neutral_?
83 comments
[ 0.54 ms ] story [ 159 ms ] threadhttps://news.ycombinator.com/item?id=4059356
https://news.ycombinator.com/item?id=22875090
Which unfortunately means Folly is an extremely heavyweight dependency to pull in compared to Abseil, if you're not already using Boost
I mean, yes, Boost itself is big. But when I think a heavyweight dependency, I think of something that itself pulls in a shitton of dependencies Javascript-style. And AFAIK Boost doesn't have any dependencies.
Start using some of the heavier weight sub-libraries and you can quickly explode your compile times. I have boost beast/asio on one project and it takes over a minute to compile one of the source files…
At one point, I had a curated SVN repository of certain boost header-only includes that didn't require any sort of build integration, had no dependencies of their own (even other boost libraries), and didn't require any build system integration - and I know I wasn't the only one.
The world of native software development was just so unbelievably different from that of web development - it's really no surprise developers from that lived through that era have such a hard time accepting the modern-day age of PWAs in Electron sandboxes masquerading as native programs. People that cut their teeth developing desktop software with something like Python (just as an example) instead would have a completely different perspective on all this, of course.
Right now the title reads essentially: "Here's a library".
It takes about a minute to find any information at all about this library besides the fact that it's used at Facebook (by the way, did you miss that? it's used at Facebook! Facebook!!). And even that information is empty bromides about how it's "designed with practicality and efficiency in mind". Hmm, I prefer my software to be impractical and inefficient...
All I take away from this is that Facebook has an irrationally optimistic expectation of how the world views its engineering prowess.
Right in the intro section, second paragraph, it says "It complements (as opposed to competing against) offerings such as Boost and of course std". Come on, what more do people want?
Professionals often don't have that choice. Cue many thousands of comments on HN why working in c++ sucks.
Looking to implement a memory efficient search engine.
That said, this is an issue for other languages as well. Another example: the "Google core libraries for Java": https://github.com/google/guava
I've never worked with a place using C++ that didn't make their own core libraries. As a low-level language, there's almost always a way to get some performance benefit by starting from scratch for your specific case.
(yes I've heard all of these "reasons")
1: Just calling it "the STL" is a key sign of living in the past--pretty much nobody uses the actual STL anymore: What we use is called the C++ Standard Library.
I won't deny your prior, but my prior is that people who complain about the use of the word "STL" to mean "C++ Standard Library" (and ignore that the latter is 10 times as many keystrokes) do so only to flaunt their knowledge, not because anyone actually experiences confusion nowadays.
Supporting evidence: https://github.com/microsoft/STL
For example iostream isn't part of Stepanov's idea, Stroustrup created that bundle of craziness in the 1980s and today it's part of the Standard Library, but it was not part of STL which only came into existence in the 1990s (although Stepanov's ideas about generic programming are older).
If I was using java or python I'd definitely not use the standard things most of the time either because it's more than likely that there's a more efficient way in some other library, and because my main paycheck comes from "can you make this as fast as technically possible given this average consumer hardware"-problems
Who would want to work in a language where you could not make these choices?
std::variant does not do this.
C++ is a massive beast but it works.
This comment makes no sense at all. I mean, would it make any sense to also criticize JavaScript/NodeJS for their relatively spartan standard modules and extensive use of vended modules from the likes of npm?
And have we learned nothing from the mistake of going with the "everything and the kitchen sink" that is java which failed to actually satisfy basic needs and resulted in projects like guava and apache commons and vavr stepping in to provide alternatives? And how about C's infamous standard library with it's security vulnerability-inducing standard functions which remain in the standard?
Perhaps we should learn lessons and avoid basic mistakes such as setting libraries in stone. C++ learned that lesson, and so did C#/.NET, and up to a point Rust. Why unlearn them?
Uh .. Yes? Yes, very much so.
the opposite from practical is general. for example is std not practical but general. Its api is not optimized for 95% of the use cases but rather so that everything is possible and equaly verbose.
the opposite of efficient can for example be trade of for readability, portability and cost (as in development time).
For me as a possible user of the lib i found this information critical. I write always similar info in the readme for my libraries.
No, it's certainly not at all. Those are completely unrelated spectrums. Something can easily be practical and general, or impractical and specific.
> the opposite of efficient can for example be trade off for [...]
And if they stated what trade-offs they made, then that paragraph would actually start to say something useful. Unfortunately, they don't.
Also, if you're developing documentation for libraries, I really would recommend finding a better source of inspiration. Take Redis, in the very first paragraph:
> What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.
That's all you need to do: before anything else, tell the user what the thing does. Then you can give some basic details about implementation: "written in Go", "runs on Linux & macOS", "uses GC, so not suitable for RT applications", etc. After that you can get into trade-offs, real narrow implementation details, setup, handwaving abstract adjectives, etc. But lead with what the hell it actually does.
Let me guess: English is not your native language, right?
https://github.com/facebook/folly/blob/main/folly/docs/Overv...
This has library contents: it’s a mix of various utilities - type conversion, data structures, memory management, random numbers, concurrency, etc.
More precisely, one that defines a datatype (no SDS-like trick) and has all goodies like SSO, CoW, views, thread-safety, etc..
I'm making such a lib but can't seem to find other examples to compare.
It's not c but almost c, it could be useful for you
[1] https://github.com/MoustaphaSaad/mn/blob/master/mn/include/m...
There are a few pure-C made STL alike containers for C that uses no c++ code at all, a random github search finds this: https://github.com/assyrianic/Harbol , there are quite a few of them just not recalling them now.
However, the API encourages practices that make it very hard to recursively acquire the lock (Synchronized is about access to objects, not critical sections).
Writing thread-safe code isn't easy, and it isn't made easier by libraries that offer footguns like Synchronized. It's easier to get it right with compiler thread annotations than with Synchronized.
Synchronized cannot guarantee thread-safety, but in my experience wrappers that do not allow unsynchronized access to the object are better than the alternative. Annotations are just another way to guard access.
Thread annotations also effectively prevent unsynchronized access to the object, but they do it in a way that programmers make fewer mistakes with, and they do it with a less error prone mental model: mutexes are seen as guards of _class invariants_, not merely as guards of _data members_. Apart from encouraging a superior mental model, thread annotations also encourage a one-mutex-per-class model which I've found far more successful and error prone (both in code I've reviewed and in code I've written) over a one-mutex-per-member model which inevitably (in my experience) results in fewer bugs.
folly is a community effort within Meta; it doesn't have a specific theme about what's included in it, but it is a set of core libraries that are likely to be depended on by most C++ software within the company, and are enough high-quality and self-contained that they can be useful externally, or showcase implementations undergoing standardization efforts (for example experimental/coro for coroutines).
Main focus is server applications, but it is also used in mobile, in particular Thrift (the serialization/RPC infrastructure) depends on it.
A very personal list of favorites:
- experimental/coro is the coroutines infrastructure, which is now widely adopted.
- SharedMutex is a heavily optimized mutex which supports shared/upgrade/exclusive semantics, and allows near-linear scalability in shared mode. It is used pretty much everywhere as it is the default mutex for Synchronized, a wrapper to safely synchronize access to objects. Its footprint is just 4 bytes.
- DistributedMutex, OTOH, is an exclusive-only mutex that supports flat combining, and performs extremely well under high contention.
- MPMCQueue and UnboundedQueue are lock-free MPMC queues, respectively bounded and unbounded, which are used in almost all our thread pools (see the executor/ directory)
- Userland implementations of RCU and hazard pointers, for deferred reclamation.
- LifoSem and Baton are other fundamental synchronization primitives that are not available in most core C++ libraries. They do a magic trick internally, where if they're waiting long enough, they unmap the unused suffix of the stack, so idle threads don't hold unused stack memory.
- Function is like std::function, but move-only, so it can hold non-copyable function objects (think lambda that captures unique_ptr). It is the vocabulary type for callbacks used in futures and executors.
- F14 is a SIMD-optimized hash table. Similar to Abseil's SwissTable, it was published around the same time.
- TDigest is a highly scalable quantile estimator, widely used for service counters (see fb303, also open source).
- (shameless plug, as I wrote this one) enumerate() is a replica of the Python function, so you can do
for (auto&& [i, element] : folly::enumerate(collection)) { ... }
I understand the FB rationale to have it, but what whould be a reason for other projects to adopt this lib?
I won't imagine maintaining such a lib as a dependency by ourselves. Maybe for one short-term purpose, but then it's still safer to go with standard or boost, or, God forbid, carve out some pieces and wrap them up.
However, I did have decent success taking the classes I was interested in (CPUThreadPoolExecutor, and surrounding), stripping the library down to just those bits, and using them. Obviously a PITA to stay current with upstream, though.
If nothing else, it's worth reading for some of the highly-optimized stuff going on in there.
I am also partial to explicit error handling over exceptions, and there is a lot to like about absl::Status (and absl::StatusOr).
Disclaimer: I work for Google but the views expressed are my own and don't necessarily reflect those of my employer.
[1]: http://cliffle.com/blog/prefer-rust/
Conan
Vcpkg
Cpm.cmake
A lot of projects these days use conan.
Developing in c++ vs (for example) python does seem like more work. I think it is a consequence of offering more control, which is expected in the c/c++ community.
What the two options offer.. Is different. With a higher level language, you can get more work done for less cost [1]. With a lower level language, you get more control —- which is great, but you shoulder that responsibility yourself. And you recognize this already..
If you want to move with the crowd (which the higher level abstractions are likely tailored for), then you will be able to move fast and far. When you need to exert more control, however, things get tricky.
For example managing the OpenSSL library used in your project independently of the system library?
[1] https://xkcd.com/353