281 comments

[ 2.8 ms ] story [ 255 ms ] thread
what does this mean for the future of clojure?

I am a bit worried that a startup has acquired two companies that have extremely deep experience in specialised languages Platformatec (Elixir) and now Clojure (Cognitect)

The answer is in the blog post ("Nubank understands the unique value of Clojure and Datomic, and is committed to continuing both without disruption." and the bullet points that follow.)
That's what they _say_, it's not necessarily the answer to the question.
Well then what would you prefer the answer to be instead?
"Nobody knows"
It sounds like not much will change.

I do wonder about datomic, though, as people I’ve spoken to at Clojure conferences seem to have made use of Cognitect’s consulting as a kind of more advanced support mechanism and they say that they will stop the consulting work.

All Datomic consulting will continue.
It says this in your post.

Cognitect will continue to offer professional services for Datomic customers, but will transition out of general consulting development.

Does this mean your general consulting services are going away?

(comment deleted)
I friend of mine hired the principles at Cognitect to kick off a Clojure/Datomic project. They were like a force of nature, really, and very quickly got things set up and helped with architectural decisions, etc. I would imagine that they have helped hundreds of customers this way.

From the blog post, I think that the answer to your question is yes: they will stop general consulting services.

Here's hoping they release a getting started project now that they've dipped out of general consulting.There must have been a standard-ish template in use.
Great, so it sounds like not much will change for external users of Clojure and Datomic. Sounds like the the biggest change is in terms of funding.

Sounds good to me, congratulations!

EDIT: I guess my confusion was between professional services for Datomic and general consulting, since both were mentioned in the context of Datomic.

Would Nubank be still considered a startup?
Disclaimer: I am a Nubank employee, so there might be bias.

Regarding the future of Clojure, I believe that not much will change actually. Before Clojure was sponsored by Cognitect and had Rich's / Stu's and Alex's stewardship and that's gonna continue to be the case just swapping the sponsoring company behind it. IMHO it's only on Nubank's best interest to have a strong community around Clojure and Datomic as well.

About Platformatec I think that was a very different thing, it was more an acqui-hiring, and before Elxir's stewardship was under the company (but having very different development dynamics compared to Clojure) and after the agreement, it was transferred to the community. So the company was pretty much dissolved after the contract, and back then, that was clear and transparent, as it was announced to happen.

(comment deleted)
I think the history of large companies hiring maintainers/authors of programming languages/libraries they use has so far gone good.

Take for example php, perl(booking.com, bbc, craigslist), js(fb etc).

The language needs to be supported financially though. Companies are not living beings, typically its some SVP making decisions. Sometimes they do support, but sometimes big shots do feel programmers are spending too much time working for greater good, while they should be working for company profits.

So it depends on the bosses you report to. There is no easy answer to this question, because many times it just comes to down to one person making decisions.

What would somewhat worry me somewhat is what the situation would be if Nubank went pop, and Cognitect were no longer in a position to step back in as sponsors.
The acquisition of Plataformatec will have almost no impact on Elixir, as Jose Valim had already moved to/created a new firm, Dashbit.
Nubank’s co-founder and CTO offers his perspective: https://building.nubank.com.br/welcoming-cognitect-nubank/
> Clojure is and will remain open-source and Rich Hickey will continue to lead its efforts (as the BDFL).

This is great! Languages need strong laser-focused vision to avoid becoming a soup of ideas.

He makes mention of 2,300 Datomic databases. Can anyone shed some light on how a company maintains (or requires?)... 2,300 databases? What business needs lead to something like this? What kind of reporting pipeline possibly could pull so many dbs with cross concerns together under one roof?
Brazilian tax law, Brazilian financial regulation, Brazilian fraud prevention...

Things get byzantine fast in Brazil.

Let’s say the company has four databases and 500 technical employees. That could get you to 2k personal development instances. Take your non-development instances and multiple for failover, blue-green deployments, multiple data centers. It doesn’t necessarily sound nice but it’s conceivable.
In datomic the concept of a database is pretty light, creating a db doesn't imply spinning up a new backend stack.
First plataformatec (company behind elixir) now this. Interesting.
It was more of an aquahire and Jose Valim the creator of Elixir left Plataformatec to start his own consultancy. All assets related to Elixir were transferred to the community. The rest of the core contributors are not affiliated to Plataformatec therefore the acquisition had no effect on Elixir.
Wonder if this makes datomic more likely to become free/open source

Not sure what their license revenue is but hard to imagine it would be significant to a major fintech

Also lets hope they acquire cursive ide too and offer it for free.

Clojure deserves an IDE for larger adoption.

Clojure has many free IDEs. Many people currently use Emacs w/CIDER, Atom w/Chlorine, Visual Studio w/Calva, Vim w/Fireplace, etc. For that matter, Cursive also has a free non-commercial edition.
Thanks Alex.

Wishing you and your team many congratulations and kudos on this occasion.

I've found CIDER superior to Cursive for Clojurescript, specifically. I use a combination of IntelliJ (with the emacs keybindings) and emacs depending on which language I'm working with. Most of my projects involve two or more languages, so I'll usually have both editors open all day :)

As an aside, I'm starting to become a fan of Kotlin for backends (with some Clojure sprinkled in where it offers a big advantage), and Clojurescript (re-frame) for the frontend (because it offers a big win over javascript there). I've taken a step back from full Clojure for everything, being that I have to work with others who aren't as in to "alternative" languages as I am.

Datomic is something I probably won't mess with until (and if) it becomes open source.

Using Spring with Kotlin or other?
Yea, spring boot. When 90% of what you're doing on the backend is gluing a bunch of different Java technologies together, Clojure becomes hard to justify. Fortunately, mixing JVM languages together on the same project isn't a big deal so you can bring in some targeted Clojure where needed.

Kotlin is great because it's the closest thing to a general-purpose Java replacement I've seen. The interop support between Kotlin and Java is outstanding.

Funny, I left Cider for Cursive a few years ago, because back then, Cursive had the better cljs support.
It's probably a result of my using shadow-cljs rather than figwheel. Cursive will catch up eventually, I'm sure.
Are there any screen casts out there showcasing the utility of CIDER? I guess I have just never been a big emacs guy so it felt sort of clumsy to me but I am probably just missing something. Could also be that I am just too familiar with the Cursive workflow.
ItelliJ w/Cursive too works great.
There's a free non-commercial license. If that's not enough, wouldn't they have to acquire IntelliJ too since they have a similar licensing model?
The bull case here is if Nubank goes full Amazon and uses banking as a beachhead to create the next layer up from AWS, which Clojure/Datomic is amazingly perfect for – data all the things
[deleted]
Is a "transaction log" not exactly the technical shape of a bank?
(comment deleted)
And we'll have to go back using sqlite again
there is also an on Prem solution, and I believe that's what such customer wants.
Ha! I was going to reply to this comment - "If you're interested in data-all-the-things, then you should take a look at Hyperfiddle" - until I recognized the user handle and realized that you already knew about Hyperfiddle. (for those reading along, dustingetz built Hyperfiddle)
still building! we killed the saas thing for now and are releasing a web framework soon - declarative crud apps from nothing but function defs, specs, and datomic! All IO is managed and composes like Datomic Peer fns compose. Userland has no async, error handling etc. Complex pickers with typeahead search, query params etc are declared in one LOC
Apparently Nubank has offices in Berlin, wonder if at some point there could be jobs there for working on open source Clojure things.
Did anyone else notice that the biggest non-academic supporters of functional programming seem to be financial institutions?

Is it due to the nature of the problem space? Is it because finance people tend to be more analytical? Something else?

I think the answer to both of your questions is yes. I think the related concepts of immutability and pure functions without mutable state allow for cleaner analytical modeling (and in some cases in strong functional languages, mathematical proof) of what that the code is actually going to do in production. That kind of predictability is essential for dealing with financial computations.
I would assume it's because it's easier to provide a mathemathical proof for the code with functionnal programming.
Immutability is a very powerful idea when record keeping and dealing with transactions. I know of at least one major bank whose ledgers are written Haskell.
any relevant links to share about this?
Perhaps: Standard Chartered Singapore?

I tried unsuccessfully to get a job with them about 4-5 years ago. They are a huge Haskell shop.

Sounds like a broken hiring process right there. Anything you care to share about why that didn't work out? I can't imagine a company having an opportunity to hire you and choosing not to.
I don't mind sharing about that. Standard Charter has the reputation for being a great Haskell shop and I have been trying for years to up my Haskell skills.

I simply didn't have enough background in Haskell. The interview was fun. The interviewer did his PhD in Haskell and compiler theory.

Around the same time, I applied to Facebook's Haskell group, and I got to interview with Simon Marlow, which was great. He also said that I didn't have enough Haskell experience and suggested that I do more Haskell consulting work, and re-apply in a year.

Anyway, I love coding in Haskell but my skills need improvement.

Functional languages and databases seem to be a natural fit.

Given you have a basic grasp of bookkeeping, accounting and financial statements: Would you rather program these things in a functional or imperative language? Would you rather store the transactions in a accreting, value oriented database or one that has no inherent notion of these concepts?

Or more generally, if your application has a temporal reporting feature, then you need to be able to ask questions like: "What were these values at date X and what are they going to be at Y?" etc.

It is nature of the problem space which requires you to produce high quality code. If you have a small error in your program, the financial market participants will use it against you to profit. What you loose, others win! Google "fat finger" for examples. In banking you have to keep track of every transactions, see "double accounting". You never delete a transaction, you only retract! Mutablity can cause you a lot of trouble there. SQL DELETE and UPDATE are extremly dangerous! Clojure and datomic solves this through immutibilty. Lastly time is relativistic, meaning that every IT system has a slightly different time. Normaly you never notice this. But they are the cause of tricky race conditions and cost you real money. Think about bank transactions, where you have a transaction date (date you send money) and valuta date (date your friend receives money). One transaction 2 different dates, depending which perspective you take (perspective is relativistic, Einstein is right even in IT!). Datomic linerialies transactions thus this problem does not occure on database level.
> It is nature of the problem space which requires you to produce high quality code.

Wouldn’t a strongly-typed language be a better choice here?

I wouldn't think that strong types are an advantage. They may be in classic software with long compile times and complex builds. However, the current landscape for financial institution doesn't require that. Immutability and functional paradigms seem to be much more in line with the needs of the business.
I do not think so. The regulation is constantly changing and the meaning of names change frequently. Thus a "Verified Account" can mean different things over the years. The problem with types and object orientation is, that the names used in the domain diverge from the name used in source code (class name, types). Think about a class diagram with class names relating to each other. To represent the domain language better, you need to change a lot in a class diagram. Dynamic languages reduce the problem, as a lot less names are needed. Clojure spec is used for specification of data instead of types, but there is also clojure typed (which uses javas type system).
Standard Chartered, Mercury, Tsuru Capital (Haskell), Jane Street (OCaml) don't seem to have a problem with using a statically typed language in the financial space.
maybe they do have a problem if their net worth is so low in comparison :D
this is perhaps controversial but i actually think datomic is so powerful, it's worth using clojure for. in other words, the database is driving the choice of programming language.

the idea of viewing a database as an immutable state of the world at a given instant t0, and time becoming a parameter on that state of the world (in order to show changes as time goes forwards [or backwards!]), is extremely, extremely attractive for things like finance, whose first class citizens are among others:

- capability for audits e.g. show me the history of transactions from any particular account. since datomic is basically a collection of immutable facts over time, this is "free"

- distributed computing - datomic runs nicely across your own internal compute (often needed for financial stuff)

- transactions are no longer strings, but are actual data structures - this makes the gnarly steps of things like transferring assets across instruments a lot easier (i'd imagine). think about how you'd implement a shopping cart with transactions in postgres vs. how you'd do it with access to raw data structures

Moreover, transactions are reified and can have arbitrary metadata, so you can query the transaction log itself (for example, "show me all transactions issued by person X").
It takes the will and discipline to use it, but what Clojure Spec (and other schemata libs) offer is, in a lot of ways, more powerful and flexible than traditional type systems. A spec can be thought of more as a contract with the data, one with enough detail that it can be used to auto-generate conformant example data even. If, for instance, you have a function, that needs to work on either, integers, or numeric strings, and textual fractions like '1/5'. Enforcing this constraint on input, and getting informative exceptions on bad data is easy with a spec, and the function code does not need to contain all of the noise to validate or coerce data. Sure, you don't see the problem at compile time, but if you can auto-generate test data, the tests that you should already have become easier to write.
Unlikely. At least not for domain modeling.

Especially in that industry where your domain is changing all the time, where regulations are changing all the time, where the ability to reason about your domain at different points in time is essential.

Having more flexible types like maps is one of the building blocks to avoid complexity (there are more, more important ones) It sounds counter-intuitive, but it certainly is working out for companies embracing clojure.

I love immutability and Datomic as much as the next Clojure programmer. But this sounds quite a lot like post-hoc reasoning.

For example, there's quite a few situations where you require mutability in bank data (e.g. with government requirements, or when you want to rollback fraudulent trading). Sure, both are solvable with Datomic, but they don't scream "huge advantage with an immutable datastore".

And there are plenty of highly competitive industries where code quality matters, but they're not flocking to functional programming.

Don't get me wrong, I think functional programming is an absolutely great way to structure and think about programming (and have coded professionally in Clojure for the last 5 years or so). But I suspect the prevalence of FP in some industries is as much chance/social reasons than it is for some inherit superiority in the approach.

Aren’t “rollbacks” done as separate transactions, that return account amount to Previous value?
Depends on the legal requirements of that rollback
small note: Banks do not want to forget fraud, the rollback would be akin to a git revert. Even if the history presented to the customer makes the fraud vanish it won't be forgotten by the bank.

With government requirements I'm assuming you mean something like gdpr. Datomic supports actually removing data via excision. it's just not the default behaviour. I personally prefer a system that doesn't forget by default whilst preserving the option to do so.

you can also support destruction of personal data via other means such as key shredding.

> It is nature of the problem space which requires you to produce high quality code.

Would love to see any actual data or studies showing that functional programming implicitly produces "high quality code".

That’s unfortunately hard to find, since hardly any customers require that these days. Fast and cheap is where it’s at.
Or, as I'm asserting, it's just totally bogus.
I have quite a few gripes with this article, and overall question the validity of its assertions, but interesting nonetheless.
There's a ton of problems with empirical studies about software in general. Very hard to conduct reliable experiments. In particular, I think analyzing public GitHub projects is pretty much the worst corpus possible.

For example, almost all of the projects in this study are infrastructure projects (I'm not familiar with all of them so I can't say that it's definitely all). I'm much more interested in application projects, and even if you (general you) aren't, you have to admit that an infrastructure project has a totally different set of characteristics than your average business application.

I think anything we can do to get more empirical data related to software the better, as we have devolved into strong personalities and conviction making pretty much all of the major decisions in our industry, which is really deeply sad. But we have to do better than just mining open source Github projects.

Yeah, I wasn't necessarily trying to get into an argument with the person who responded, but there's so many factors not being accounted for in that study that it might as well be measuring nothing at all.

Choosing Github projects and measuring defects on them has almost nothing to do with the quality of closed-source code (as we were discussing, functional programming languages in the wild). I also briefly started down the path of mapping that study's measurements to overall language popularity (as I think they're related - more C++ code available = more bugs), but gave up as I remembered that A.) Nobody's opinion changes as a result of a reasonable argument on the internet, and B.) Convincing this person nets me nothing at all.

You get higher quality code by hiring people capable of producing higher quality code. A great way to find people that are capable of producing higher quality code is by hiring for languages with extremely small talent pools - nobody got there because it was easy or the odds of getting hired were good. Functional programming might seem popular on HN, but in the wild is really not popular at all. Clojure developers probably care a great deal more about the quality of their work than, say, some random J2EE developer, as the Java dev might not care at all about anything other than staying employed.

I guess my argument summarizes down to "Functional programmers care more about their work", which, to my original point, has nothing at all to do with the languages they're using (as the person I responded to was saying). To assert that "Functional programming languages produce higher quality code" is like saying "This brand of hammer hammers better." It's just nonsense.

That's oversimplification. Good programmers care about their work, doesn't matter which language.

Also, use the right tool for the job.

A language that forces you to think about values rather than mutable objects, will produce higher quality code as the number of ways you can shoot yourself in the foot are drastically reduced.

Clojure will make you run your code constantly in the REPL. Paired with a dead-simple testing library, the desire to keep your majority of functions pure, it is not hard to see why Clojure code has higher quality.

> Good programmers care about their work, doesn't matter which language.

Yes, but the whole purpose of my post is to point out the difficulty in _finding_ the good programmers.

> Also, use the right tool for the job.

Not sure how this applies. You can build banking software in literally any programming language.

> A language that forces you to think about values rather than mutable objects, will produce higher quality code as the number of ways you can shoot yourself in the foot are drastically reduced.

Again, this is just totally unsubstantiated. You're using "higher quality code" to mean "code I like more".

> Clojure will make you run your code constantly in the REPL. Paired with a dead-simple testing library, the desire to keep your majority of functions pure, it is not hard to see why Clojure code has higher quality.

Where have we demonstrated that clojure has implicit higher quality? Also, where have we demonstrated that all clojure devs keep their functions pure? Or test them? And how does clojure enforce that?

Your last bit kinda proves my whole point, I think. You like a language a lot, and you make good use of its tools. But again, when you're looking at large groups of programmers, they're going to drastically lower the bar on what you "should" do - talking about "strong desire" and "dead simple" is totally unrelated to whether people will _actually do_ something in the wild. And what's the best way to keep a codebase good? Well first, tooling, but second, not letting bad devs get ahold of it.

Not really a Clojure user, but I made a small API with it once and it blew my mind, especially the design patterns involved. An interesting one is the ports-and-adapters (a.k.a. hexagonal architecture) [1][2] . Basically, all the business logic will be kept at a layer, and all of the functions there should be pure (i.e. they will always return the same information according to your input, and these functions won't cause side effects [3]). Then you would have layers where you can plug databases and REST handling.

And Nubank take testing really seriously. REPL and pure functions makes it very easy to use TDD.

[1] https://github.com/nubank/basic-microservice-example#ports-a...

[2] http://wiki.c2.com/?PortsAndAdaptersArchitecture

[3] https://practicalli.github.io/clojure/thinking-functionally/...

Mainly because of the problem space. Banking/finance is a complex field with many moving parts: card processing, validation, encryption, currency exchange and tons of certification stuff (I believe Nubank's primary focus is card processing). Didn't even touch other financial fields like investing, credit handling, or any kind of trading.

The software is always expected to be correct, never fail and fast. Otherwise, someone will lose the money, and usually, you (bank or institution) will have to recoup that, plus penalties, plus name damage.

Source: worked there.

Doesn’t banking actually use double accounting to ensure not losing money instead of relying on something as brittle as software?

Then again, Wirecard happened. But perhaps that wasn’t exactly banking.

In haskell, a lot of the uptake in financial services is because they are writing internal DSLs to do things like modelling exotic options.
As a counter example, Stripe uses Ruby and Mongo.
Never discount networking effects. I think it's good for the problem, but better languages are better for all problems, so that doesn't explain it enough.

Considering that finance is decently geographically distributed, I'd say it's more SV has become javascript hell than finance is especially FP oriented.

I think the best argument is that there are a lot of small financial institutions (small referring mainly to the tech groups at them, that is, so you might hear that some big bank uses eg Haskell but really it’s just some group within their IB arm. I’m sure google has some Haskell around too), and often the kind of people they employ tend to like weird languages.

Plenty of financial institutions use lots of c++ or java and plenty use cobol too.

Nubank has a very inspiring story, both for engineers that are fans of Clojure & Datomic and for entrepreneurs that want to do "crazy" things like entering the bank sector.

This must be very validating and exciting for both Cognitect and Nubank.

I'm excited of the future impact on Clojure and Datomic. I get the impression that _all_[0] involved parties deeply care about what made them great so far.

[0] I'm getting that vibe based on some talks I saw of Lucas Cavalcanti and Edward Wible, available on Youtube.

Yeah, I saw the Nubank folks present in person in NYC just prior to COVID. I am a Clojure fan, to be sure, and also a banking/money dweeb. Soup to nuts it was one of the most impressive presentations / achievements I have seen. They were clearly operating in the future. Congrats to both and looking forward to growing the community.

Plans to release a SAAS banking core in the US? Pretty please?

Literally ‘functional banking’ :)
Throwaway account, for obvious reasons.

This is great for nubank, but how is this supposed to work for other users of Clojure and Datomic? Who is really going to be interested in building on top of tech owned by a (new, unknown, regional) bank?

Someone said that this is just like Amazon (who would build on top of tech from a bookstore??), but I hope it's obvious the two situations are very different.

I expect that I'll have to be looking for a new job in a year's time :-(

A lot of people don't seem to mind building on top of tech owned and developed by a telecommunications company - Erlang.
Yep, telecomms and banks, very similar dynamics and incentives /s
(comment deleted)
In what relevant way are they different? (I find that your comment has more sense without the /s tag)
Yes? What incentive exactly are you implying banks have that make them less suitable than other mega-corporations for the purposes of developing a programming language?
Banks and fintech often view their in-house tech as competitive advantages, and they usually don't care a whit about building cred in dev communities (a common reason why projects are OSS in the first place, to drive recruitment or promote a complimentary platform).

A good outcome isn't impossible, but it might be unprecedented

Rich Hickey literally bet his retirement savings on making Clojure, and owns all the IP. It seems unlikely he's just going to let them smother it.
Nubank has presented their tech several times at a number of conferences and I believe they often sponsor those conferences. It's not a traditional bank, like Airbnb is not a traditional travel agency, and Amazon is not a traditional bookstore. This is not your local neighborhood credit union or ancient behemoth running COBOL mainframes.
And telecommunications companies don't see their technology as a competitive advantage?
Is it better to be owned by an adtech company (React, Angular, Go)?
If you asked me that 15 years ago, I'd say "no". In hindsight, it sort of makes sense given their objective of increasing time spent online and the quality of the experience.

The proposition here is that nubank will be the first bank to successfully turn into a software vendor. Maybe my worries will be nothing in 10 years, but no one can say its an obvious outcome.

Bloomberg and Jane street (not banks but still finance). They don't own OCaml but are heavily involved in its development.
Bloomberg has sponsored BuckleScript in the beginning, but now it is sponsored by Facebook. Other than that their influence on OCaml development is minimal (and technically BuckleScript is a fork of the OCaml compiler).
Nubank already ships software, Cognitect is already a healthy consultancy, no particular reason to kill either one. Surely we can agree that banks also care about good UX? Nubank in particular is known for it.

(I appreciate that you may not have heard of Nubank but I don't think calling them "unknown" is fair. They have attracted a literal billion in funding in a little over a year. They have over 10 million users.)

Per https://building.nubank.com.br/welcoming-cognitect-nubank/, "Nubank does not own Clojure, just as Cognitect never owned Clojure. Nubank will take over Cognitect’s role as a corporate sponsor for Clojure, funding the core development effort".
Outside of the license, OF COURSE Relevance, then Cognitect, now nubank owns clojure, in every other way that matters.

Now do datomic :-)

I am pretty sure that whoever owns Cognitect is in no position to boss Rich around regarding Clojure. Besides, Clojure is rather simple and stable. No need to change it dramatically...
(comment deleted)
> Who is really going to be interested in building on top of tech owned by a (new, unknown, regional) bank?

As opposed to building on top of tech "owned" by a small software consultancy? Perhaps I simply lack imagination, but how is this worse?

We all know the history of acquihires. Key tech and products inevitably are swallowed, never seen outside again.

Besides that, datomic really is a great solution for banks and fintech....are other firms like that going to buy from another bank, maybe a competitor?

Nubank is a company valued at over $10 billion, with investing from major Silicon Valley firms. It is not unknown to millions of people, and is expanding internationally so I don't think they'll be regional much longer. Clojure and datomic being such a good fit for their tech stack of hundreds of microservices and thousands of databases should really encourage people to use clojure like Erlang's history in telecommunications encouraged people to adopt Elixir.
Since Nubank, is not in the business of selling software does this mean, they might open source datomic ?
Will the license change?
No change in Clojure's license (it is still EPL 1.0).
No. It's just a "cooperation" to use technologies.
It sounds like Nubank might have the biggest Datomic installation in the world (2000 servers). This is exciting news for people like me who think Datomic is the most interesting database out there. Putting more resources behind Datomic might finally allow it to go mainstream.
I was unable to bring Datomic in-house. Mainly because our organization has a huge agreement with Oracle and all our database servers are essentially "free" to teams.

So I cheated. I created some schema designs that were immutable. I added a GUID, timestamp, and a deleted flag (value of 'Y' or 'N') to tables. Basically, all selects are against views that are defined over the tables to select the tuple associated with the max(timestamp) for that tuple along with the tuple having a deleted flag value of 'N'. This means that any select only sees the most recent tuple value for a GUID if it hasn't been "deleted".

There was a little bit more hiding in there to handle dirty writes.

But this worked very well for my requirements. By really only doing inserts, I was able to do similar "point in time" looks at the db as an immutable value.

Fun stuff.

Reminds me of the time I wrote an eavt (entity, attribute, value, transaction-time) store on top of rethink db with a friend of mine.
Would you be willing to have an email chat about that? I'm thinking about doing a similar strategy with our relational database and wanted to get your experiences with that.

sam at havenconnect dot com

Good for them, well deserved success for the principles, Clojure, and Datomic!

I donated to the Clojure dev fund early on, and I had two consulting customers over a few year period who mandated Clojure. I stopped using Clojure when the paid work stopped because I really like Common Lisp better for my use cases. I have on my radar for sometime this year to update my cooking website [1] that is written in Clojure, so I will at least get a small opportunity to kick the tires on modern day Clojure.

[1] http://cookingspace.com

Are there any technical posts from Nubank on how they manage such a large Clojure codebase (I guess it must be Clojure's largest user in the industry).

I'm looking for things like how to refactor safely in the absence of static typing, for example. Do they make use of spec?

I'm a big fan of static typing, but when using a dynamic language with immutable values and pattern matching, like Erlang, I don't miss it that much. Maybe with Clojure is the same?

Not sure we have any talks about that specifically, but what I can say is that on the Clojure codebases:

- We rely heavily on testing (unit tests, property-based/generative and integration)

- We adopt a micro-services architecture, so most codebases are small enough that refactoring is easy, or sometimes we just replace the service altogether

- We leverage Schema (https://github.com/plumatic/schema) and/or clojure.spec (https://clojure.org/guides/spec) to annotate functions and data schemas, but it's opt-in

- We have static checking of data schemas across boundaries (e.g. checking data producers did not break consumers over HTTP or Kafka), which is where we found the most value (replaced complicated end-to-end tests)

Thanks! Can you talk a bit more about the static checking of cross-boundaries schemas? Any relation to tools that allow a schema to evolve without breaking compatibility, like protobuf?
It's not based on protobuf. There's some in-house tooling that checks the schemas of producers against consumers against known versions (aka the versions in production), and we run it as part of the CI.

Since services implement standard components/protocols there are standardised hooks to extract these schemas definitions from each HTTP endpoint or Kafka consumer.

Does this rely on code generators or maybe a schema generator from code? I’m curious about how schema and code are kept in sync.
Yes. The schema definition is part of the code/data itself, so you can reference a particular schema by its git rev hash.
Curious, did you really find that microservices helped?

The downsides can be so tremendous -- I wonder if nubank would have done better to services in the low digit range.

(context: my thoughts on it -> https://stopa.io/post/236)

I believe micro-services change the trade-offs, and it's a matter of what you value more.

We certainly have to invest more on infra-structure, monitoring, standardisation, and we have been doing since day one - by now it's pretty robust, to the point our internal tooling allowed us to migrate from raw EC2 to kubernetes (https://www.cncf.io/case-study/nubank/) and it was a transparent change to all services and teams.

On the other hand, failures are more localised and we gain robustness and speed to do certain kind of changes.

Your point about getting domain boundaries wrong is true, but IMO it's a pain orthogonal to micro-services. The solution is the same as if you defined a sub-optimal class interface on a big mono-repo: relentless refactoring. We are often strangling (https://martinfowler.com/bliki/StranglerFigApplication.html) and replacing legacy services.

Questions: transactions are really important in banking systems. If you have 100s of Microservices how do you manage transactions across them? Or maybe all of your microservices run on top of a monolith database with transaction safety?
It's probably a misconception that database-level transactions are the primary way to implement business transactions - many business processes w/ customers or 3rd parties rely on ACKs and/or compensating transactions, escaping the scope of a single database-level transaction.

In practice, if the services are loosely-coupled enough, transactional boundaries will be local in scope and transactions spanning multiple services are unnecessary. Downstream services can be eventually consistent if idempotency is guaranteed and retries/replays are free.

Inside the scope of a single service, when needed, we use Datomic, which provides ACID guarantees for that (https://docs.datomic.com/on-prem/acid.html) and also helps w/ idempotency guarantees by allowing time-travel (https://docs.datomic.com/on-prem/getting-started/see-histori...).

I've watched a number of Clojure centric talks from Nubank employees. Actually, I think it was one particular employee, but I don't remember his name. One was on how they do testing. The others were about Fulcro, which is a very interesting web frontend library/framework.
Nubank also acquired Plataformatec and the Elixir team. Very interested to watch how this plays out.
I am genuinely curious. I haven't heard the name Cognitect before. I checked the website and they seem like another software consultancy firm. Anything I am missing? HN is very picky, so they must be doing something special.
(comment deleted)
This is the company that employs Rich Hickey and most of (all?) other maintainers of Clojure, the programming language.
> Nubank has grown to 600 Clojure developers, running 2.5 million lines of Clojure code in 500 microservices

Wow, that's enormous especially for a concise language like Clojure. Very happy about this announcement as a Clojure user. Sounds like it's going to be viable for at least another decade.

I'm just remembering the sad collapse of RethinkDB...

The task of making a viable commercial database in 2020 is herculean and I can really empathize with the team's need to both keep cashflow and have a sufficiently sized deployment to debug against.

Can also say that everyone I've met at nubank are really passionate about the ecosystem and seem like they'll be excellent partners

Datomic is 8+ years old and by all accounts has a healthy customer base, it's not like they are in the takeoff stage. But of course they have to keep up like everyone else, especially now that there is competition from the OSS side (Crux).
I remember being very excited by RethinkDB, it was a huge shame to see that go down. I feel like they signed their own death warrant by making a product that was super easy to use and was well documented.. so no support contracts, because simply not needed. Such a shame.
Nubank is one of the biggest startups in Brazil and I am a Nubank customer and I must say they have the best service available. Treating customers as customers.
The customer support is the best! No waiting, no pre-recorded messages, no muzak, no menu. A real person picks the phone and talks to you. Oh and the app is a delight!
Customer here, as well. IMHO, the best company currently in Brazil.
AWS has Quantum Ledger DB, which is a lot like what nubank architected on top of datomic.

Anyone know what language qldb is built on ?

I believe the intersection of use cases for QLDB and Datomic is the empty set.
Congratulations to Cognitect. Hopefully the acquisition will help Rich's retirement funding. (see 'History of Clojure' for the not-so-insider joke here)

I have had the pleasure of "using Clojure in anger" and for a few years was very dedicated to learning how to use it effectively. This was a significant step up for my general programming skills and the simplicity of the language often saved me from myself with respect to bad design or choices in code. Whatever programmer "maturity" I can claim is probably due to Clojure and teaching introductory programming and databases (not with Clojure and Datomic to be clear).

This acquisition is more abstractly interesting to me in that a VERY dedicated Clojure/Datomic company made it. We didn't see here, for lack of a better example, a FAANG company grabbing Cognitect. That may have been a choice! Maybe Cognitect went with an acquisition where they retain a high degree of autonomy with the benefits of being acquired.

I never made the leap into a larger organization with Clojure at its core. I was able to use Clojure only in my small shop where I have a small degree of autonomy on development language choice. For a few years, I was somewhat actively seeking an opportunity to join a larger Clojure-oriented team but struggled to gain much traction when applying to openings. It seems possible to build a Clojure career, but I've not been very good at that.

(for anyone who doesn't get it, Rich funded Clojure development via his retirement savings from 2005 until he ran out in 2009 https://dl.acm.org/doi/pdf/10.1145/3386321)
Oh wow! I didn't know that
(comment deleted)
gotta ask - what did he do after 2009? is he alright now?
He became the CTO of Cognitect. :) He's doing great.
He did a community drive in 2009/2010 or so and a lot of us pitched in a little. Then he joined forces with Relevance to create Cognitect and that has funded his Clojure development since. Hopefully he gets a nice payday from this acquisition too.
Rich Hickey is an artist and a hero, I owe him so much.
Pretty strange how no one else sees that banks are very very fragile institutions, plus this one is heavily VC-ed, which makes everything they claim in public irrelevant (this is if we want to trust what any bank says at all).

What is relevant is that Cognitect will now have 100% of income coming from the single (and fragile) source, they now do not have to compete and survive on their own, they can't say "no" when having a choice between having to conform or quit the only money source.

I am not discussing personalities here - people are free to choose how to sell their products or themselves. Just thinking what this means to me as a software consultant who makes 100% of money from Clojure (yes, a fragile single choice because of sort of falling in love)

Yeah, very strange for a bank to buy a technology stack company. Why? It strategically doesn't make any sense, unless the bank wanted Clojure/Datomic to prioritize their features and requirements.

What would it mean for adoption?

How is a bank a fragile institution? Banks are the most heavily insured and protected institutions in any modern country.

In the US for example, banks not only contribute to federal insurance but also have built joint private reserves to weather economic turmoil. In the modern world banks rarely go bankrupt because if they did that would tank any confidence in the economy and create incredible chaos.

Even in failed states like Venezuela banks are still a major pillar of economy confidence.

This is simply false - banks are not insured against failure and not protected unless some political move "too big to fail" is made, and this only makes systemic financial system failures worse.
Whether this is false depends on particular country laws. For example in the EU you can't call yourself a "bank" without being subject to regulation, and in Poland for example, if you are a "bank", your money is actually protected against failure, up to €100,000.
Deposits insurance has nothing to do with that.
> Banks are not insured against failure and not protected.

Well. Except they are. In the US not only at the federal level but also at the state level:

- https://en.wikipedia.org/wiki/Deposit_insurance

- https://www.fdic.gov/deposit/

- https://en.wikipedia.org/wiki/Federal_Deposit_Insurance_Corp...

- https://www.cnn.com/2020/04/17/business/bank-earnings-defaul...

You are mistaking what is insured here. Deposits are insured, the banks themselves are not.

Here's the list of banks that have failed in the US and for which their deposits entered receivership managed by the FDIC; looks like a handful every year: https://www.fdic.gov/Bank/individual/failed/banklist.html

I'm not. I was trying to point to the OP that banks aren't as fragile as he think they are. When the very nature of your business, which is handling people's money, is federally insured, you have mitigated a lot of risk.

Yes banks can fail and they can go bankrupt, and if that happens to Nubank then this acquisition goes to hell which makes the Clojure ecosystem weaker. I give that to OP.

But saying that banks are "very very" fragile institutions I think is fundamentally wrong.

The fact that deposits are insured has nothing to do with bank solvency. People just get some of the money back when the bank blows up. Arguably this makes banks to be even more reckless and to hide more dangerous risks under the deposit insurance claims.

And the function of the regulator is to stop and DISSOLVE the bank BEFORE it goes potentially too deep into the pocket of deposit insurance fund in case of running astray. So this is a social instrument but again does not solve the institutional risk problem. Maybe, just maybe makes some banks more paranoid, but they still blow up at very high pace.

Just to clarify, Nubank is not an actual/formal bank in Brazil. They have operating licenses as Financial institution and Payment institution though.
> In the US for example, banks not only contribute to federal insurance but also have built joint private reserves to weather economic turmoil.

did you miss how the banks all had to be bailed out a few years back?

Well I’m talking about present time. Past your cynical take, the reality is that the last crisis was exactly the reason why banks in the US have been growing their cash on hand the last decade faster than they did the two decades before that, while putting aside money on other reserves.

https://www.cnn.com/2020/04/17/business/bank-earnings-defaul...

With respect, I see where you are coming from, but would argue it isn't necessarily the right read, either in terms of banks, or in terms of Clojure, though there is nuance.

Banks are systemic actors in the financial system. They have privileges granted only to them, requirements that only they have to meet. It is HARD to build and operate a bank, whether in the US or in another country.

From a "fragility" perspective- there are a lot of definitions of fragility, but when looking at attributes like- may fail at any time; may lose customer assets; may take on inappropriate risks given their systemic position; have negative systemic impact on larger financial system operations- whatever attributes you think of, the amount of energy that goes into oversight around those attributes is significantly larger for banks than any other "systemic" industry (real estate, health, whatever).

The number of health oversight agencies in the US is, like, 5. The number of financial oversight agencies is in the dozens, just at the federal level. (This is why in the US you sue for medical malpractice but not financial malpractice, and why there is government backed financial insurance of various kinds and much less health insurance. You are on your own in the health industry. In finance there are armies around).

Banks have like 10 different kinds of risk that they and their supervision are monitoring for. Yes, banks fail, but that's because people suck and are bad at things and the second law of thermodynamics and so forth. It isn't necessarily the same in other countries but banking, like many other industries, is increasingly global, and banking core systems are increasingly international.

Nubank is clearly doing a LOT of things right- I saw a presentation in person from them in February and it blew my mind the level of capability and abstraction they were working at. They are a solid entity with a strong growth path.

Cognitect, in contrast, is a tiny consulting firm. Maybe 50 employees? Technical consulting at scale is an incredibly hard business, particularly when your solution domain is limited to a particular toolchain. There was really no chance Cognitect was ever going to become Thoughtworks, or Deloitte, or anyone like that. That's an infinitely more fragile financial situation for Clojure than living under the roof of a business with a valuation in the billions.

The way to think about this is that all technologies have a sponsor. Clojure moves from basically being sponsored by a guy in his backyard to, like PHP (Etsy has a market cap of around $10B and employs the PHP leader). That's huge.

Beyond that, having what should become a more well known and reputable player in the financial industry backing it- you could well see A LOT MORE adoption from enterprises playing copycat. It's not going to happen right away, but it is a seminal step in COMPLETELY changing the enterprise adoption calculation.

However, everything is a tradeoff. If you aren't working with Datomic, or in finance- this move is a stronger statement that maybe Clojure doesn't have the strongest future for you. It isn't going to become a lot of other things that it could become. The finance industry has a fondness for languages with weird syntax, and those languages often stay in the finance industry niche. That's probably where Clojure stays. There are all sorts of wicked cool JVM things like LMAX and Azul that are also financial industry specific, and Clojure and Datomic get to become one of those. Very successful, very cool, but niche.

Anyway- hope that's helpful, happy to continue the conversation.

Cheers.

I have no issues with financial things per se - I have built several financial systems (from crypto to funds management) with Datomic and Clojure myself.

And this is also one of the reasons I think having sole bank as your owner is a really really bad thing.

Say more? Would be helpful for others to further articulate your observations and opinions. No one has all the answers. For all who have a stake a full diversity of views is best.
I'll be really short: the bank lends money in Brazilian real to credit card holders and, in addition, keeps half of that in financial derivatives. The last real slide (3x in value) was in 2011-2016. Current world situation is.. interesting, to say at least, and 1 billion of VC money are not charity.
Ah, yes, got it, that's helpful.

Are their financial reports available and published in English? Would be interesting to read.

I don't know details of their financial model, but will say:

* there is a long bootstrap process for a bank that involves being selective with what customers they can serve with what products at what margins in what order

* debt is a hard game, though in many niches now is the best of times

* currency is also a hard game

* in general, banks are valued at a small multiple of assets and do not attract VC

* in general, SAAS businesses are valued at a large multiple of revenue and do attract VC

So my assumption would be that they are using equity to build the ability to get into the banking core business as a SAAS, which is both lucrative and exploding as an opportunity (see recent nCino news). If they succeed, that's an extremely durable business, and not a bad place for Clojure to be. There are a number of languages that still only "exist" because they form the basis of a banking core.

Cheers. Thanks for the engagement.

An excellent assessment - Rich, Stu and team clearly must have assessed the option similarly. Congratulations to Cognitect and here's wishing them well.
“If you aren't working with Datomic, or in finance- this move is a stronger statement that maybe Clojure doesn't have the strongest future for you.”

I don’t follow. The company the python BDFL last worked for was a storage outfit with a valuation roughly equal to Nubank. The Rails BDFL is at a pretty small software shop in Chicago. Ruby BDFL was at a division of Salesforce last I checked.

Seems a bit reductive to say that Clojure creator being employed by a bank makes Clojure a worse choice outside finance.

Sure, reductive is fair. In response:

a. ruby and especially python (and PHP for that matter) are top 15 languages with very wide adoption, following, name recognition, use cases. Great documentation, lots of books, conferences, blogs. Robust business and consulting ecosystem. etc. Lots of onramps and growth curves for individuals. And communities still growing steadily. Clojure is substantially younger (10 years vs 30 for py, 25 for php and ruby) but it is comparatively tiny and niche and arguably its rate of growth is lower

b. syntax is part of the story here. Clojure is a better language than kotlin, but kotlin is seeing explosive general purpose use case growth that Clojure can only dream about. Clojure syntax is niche.

c. there are a very large number of technologies, including languages, that have wide usage within finance and banking and zero use outside. No matter how neat k is, there is no market for tutorials, no community growth (well, maybe modulo data sci world). No one is ever going to use jbase rather than postgres. Many of these also have niche syntax.

d. I think the Nubank news on balance is terrific, but I don't think it helps Clojure's community growth story at all, which is a chicken/egg problem in the context of general purpose use outside of finance. Nubank committing to that outcome...seems doubtful to me, compared to Nubank wanting to have more control over what it considers to be its strategic differentiator.

This last could well be wrong, and what we'll get from Nubank-owned Cognitect is developer experience, onboarding, a completed Spec, a comprehensible REPL-based developer onboarding, error messages, documentation- all the UX things where Clojure's superior design ergonomics are undone by nits in the experience.

My understanding is that Nubank is concerned about these things for its internal people, that their hiring rate has forced them to solve for onboarding. We will see. The Clojure team itself likes the sharp tools for experts model. There would need to be a funded change in that emphasis for the acceptability of Clojure solutions in areas where it wasn't already known to grow.

Cognitect is not winding down Datomic, Datomic Cloud, or Datomic consulting, and they have big name customers including Facebook and Netflix.
Nubank raised nearly a billion $ since 2018, hard to see that being bad in any possible way for Datomic to have access to those resources
That billion is not a charity, but comes with some people pursuing some things. Most of the time that "things" are to make a PR stunt, inflate and sell.