Being a massive fanboy of jOOQ, I have to ask if this is a "for fun" project or is there something nice / different I'm not picking up on from scanning the repo / blog post[0]?
Looking at the sample code makes me wonder if Java does not have type inference like auto" or "var" in C#/C++? LINQ like code (or STL in C++) is much nicer if you don't have to explicitly declare variable types.
Yes Java 10 now has local variable type inference so you can now use "var", this repository looks like it was last touched 2 years ago so probably why the examples aren't using it.
Java 11 in September is going to be the first stable version supporting inference for locals, and IMHO that's when it starts to be reasonable to write code that doesn't run on today's LTS.
C# LINQ’s prowess is not just about the syntax, it’s about the underlying expression trees that linq queries create.
The same LINQ expression once translated to expression trees by the runtime can then be translated to SQL, MongoQuery,C# IL, etc at runtime depending on the linq provider.
Indeed. My understanding of LINQ is that a where clause, for example, creates a matching clause in the SQL it generates, so the database only fetches the elements that you want. In contrast, the Java solution seems to "select *" in SQL then iterate over the results in the application itself.
LINQ itself just creates expression trees. The expression tree is then translated into code by another provider (for lack of a better term). The code generated can either operated on an in memory collection or generate any other type of query.
var seniors = customers.Where(c => c.age > 65)
Could either represent C# IL, a MongoQuery, a SQL query, an ElasticSearch query, etc. depending on what type of variable “customer” is.
Just to be a little pedantic and to inform those that might not be familiar, LINQ only creates an expression tree when the type of the source object is IQueryable. If the type of the source object is IEnumerable only (such as an in-memory list), it will not create an expression tree and instead will use the Enumerable static class methods passing in the lambdas as delegates (aka function pointers). In the latter case, there is no "code generated", it will directly enumerate the IEnumerable and do things like an "if" statement inside a "foreach" loop in the case of "where", for example.
Also, you don't need a better term than "provider" as that's what they're called. :-) The IQueryable interface includes a property called Provider that returns an IQueryProvider which can operate on an expression tree.
Also, in my experience knowing when you will be working with an IQueryable and knowing when you will be working with a realized IEnumerable is a very important thing to be learned when working with Linq and datastores. You can easily load full tables into memory if you aren't careful.
This times 100. Ended up having a week of intermittent outages when a Dev on my team created a single helper method which took in an Func<> not an Expression<Func<>>. Having the same interface for both is imo dangerous
> Indeed. My understanding of LINQ is that a where clause, for example, creates a matching clause in the SQL it generates, so the database only fetches the elements that you want. In contrast, the Java solution seems to "select *" in SQL then iterate over the results in the application itself.
That's the difference between IQueryable and IEnumerable. If you use IEnumerable, it'll filter on the client side rather than the server side. Potentially a performance and resource and even security/privacy problem. With IQueryable, the filtering is done on the server side.
Personally, the best thing to do is to use stored procedures. ORMs and query generation on the client side is such a bad idea. But it's easy, so people go for the easy solution.
Stored procedures? No. It makes it much more difficult to keep the code and the stored procedures in sync. Also unit testing is more difficult and slower. With linq, you can unit test your queries without ever touching a database by mocking out your context and using Lists (https://msdn.microsoft.com/en-us/library/dn314429(v=vs.113)....)
Also, it’s a lot easier and less error prone to compose dynamic queries with Linq - ie keep chaining Where clauses and OrderBy’s based on a condition.
You can also do things like create methods that take predicates.
I’ve even had one project where the backing store could either be SQL Server or Mongo. We were able to use the same interface with methods like the Find method above
and the repos translated the predicates appropriately.
> Also, it’s a lot easier and less error prone to compose dynamic queries with Linq
That's the general assumption of people not familiar with SQL, RDBMs and full stack architecture. One of my issues is that ORMs and LINQ have become crutches. Also, you are ignoring the major problem of security and auditing. If you are working on a local toy project, then sure, you can just do whatever you want. But if it is an enterprise project with real security concerns, the idea of giving read/write access to web server accounts is amateur hour. Trust me, I've seen it with my own eyes. People who spend 2 days reading "Learn web dev in 21 days" and "Learn SQL in 21 days" and putting today something akin to "Hello world" projects because their managers didn't know any better.
> - ie keep chaining Where clauses and OrderBy’s based on a condition.
You can use LINQ, chaining, lazy evaluation and all that on data/Enumerables/sequences retrieved from a stored procedure.
How is this meaningfully different from the Java 8 Streams API? It seems to be functionally equivalent right up to lazy evaluation, just with SQL vocabulary (select, where, etc) instead of functional vocabulary (map, filter, etc).
SQL vocabulary is better than functional vocabulary as for many people lot of understanding (and actual use) of set manipulation and transformation starts with SQL. Why have one set of terms for code and a different set on the DB? Thats like saying why is there English, Spanish is functionally equivalent etc. Except that I already learned English (SQL) so why don't you just let me speak English (SQL) in my code?
Java streams are great. Having to go from collections to streams (`.stream()`) and back (`.collect(...)`) is annoying, but understandable from a compositional point of view.
For fun, you can further reduce the size of the code above (and IMO, make it clearer), but using `Arrays.stream(product)` and a method reference for the `map`:
The point of LINQ is that it isn't like java streams, it gets to decide where to "where".
Java streams read data from spinning rust and then does the selection in RAM. LINQ can send the "where" to the RDBMS and the RDBMS can use an index/btree lookup to read just the data needed where java streams would read umpty megabytes.
Yeah, I get that, but this library doesn't appear to be doing that. It seems like it's replicating the facade of LINQ, but not the "hard" parts such as switchable backends, which isn't useful since we already have Streams to handle that use case.
It replicates the API and provides a distinctly primitive implementation of that API. By contrast java streams provides an API that can never, ever be implemented in such a way as to issue a clever SQL select.
I don't know whether the JINQ people plan to write a good query generator. But if I were the maintainer, my plan would definitely be to focus on the API first and only then work on optimal backend implementation(s).
Quill is really neat and I agree that it's a lot further down the road than most libraries like this.
But it all feels a bit too "magical" for me. Seems like you're trading one class of bugs - hand-written SQL mismatching your types - for another - this magic doing something unexpected when you try to do something weird. But maybe it's really really good and that second class of bug never happens. I'm just hesitant (and happy with doobie for now).
The point of jOOQ is for developers to write SQL (almost) exactly as if Java natively supported the SQL language. Not hide SQL behind some "more idiomatic" collection API.
So, jOOQ will never become quill or end up looking like it.
A pet peeve of mine is how these libraries put the 'old' alternative in the worst possible light, creating the fake impression (at a casual glance) that the library makes your code much nicer. It doesn't.
Passing in an array is weird, so lets assume we're not working with really outdated codebases and a List<Product> is passed in instead.
You can do the snippet with bog standard Java8+ stream API like so:
Even in old style java6 syntax it's short, sweet, and to the point (as short as this library can let you write it):
List<ProductDisplayInfo> getDisplayList(List<Product> products) {
var pdiList = new ArrayList<ProductDisplayInfo>();
for (var p : products)
if (p.getCategory() == 2 || p.getCategory() == 5 && isProductUnderDiscount(p))
pdiList.add(new ProductDisplayInfo(p));
return pdiList;
}
Separate from that, as scarface74 mentioned, part of the point of LINQ is that the structure of the expression tree can be applied elsewhere, for example, to an SQL statement.
Probably the biggest difference between this and Java Streams is that it produces an Iterable<T>, which implies that, like LINQ, the thing can be iterated multiple times. For better or for worse, Java 8's Streams can only be iterated once. The difference can potentially be a big deal, depending on your use case.
Repeatability is a bonus, but it's also a limitation - not all sources of data can be restarted. Overall, I think restartable enumerations are ok to lose given that they are rarely restarted in practice - they tend to be expensive, or at least seem expensive.
When you're working with external data, yes, they can be expensive.
But most of what I've seen LINQ to Objects used for is simply mapping over data that already lives in memory. Oftentimes you want to just pass along a lazy projection of that data, on the grounds that re-running the operation is less expensive than allocating a bunch more memory in order to cache the result to a list.
In the cases where a source can't be restarted, it's always seemed easy enough to me to just build up your LINQ query and then pass its enumerator back rather than the original enumerable. Returning an IEnumerator<T> should be sufficient to communicate that you only get to enumerate the thing once, and tacking a ".GetEnumerator()" on the end of things requires barely more typing than tacking ".ToList()" on the end.
By contrast, with streams, I feel like the Java maintainers gave me an API that's less powerful than the standard I'm used to on other platforms, on the grounds that they think I'm not a smart enough developer to handle a more powerful API. I don't really appreciate that. All it's really meant is that I now need to work with two different libraries depending on whether I'm working with data that lives in memory or not, which does worse things to the cognitive load it takes to understand the codebase than simply having to keep track of the difference between an Iterable<T> and an Iterator<T> would have.
I find it hard to get worked up about it. You can just hand around a Supplier of your stream if you need restarting. Monadic composition of further operations is a bit clumsier then, but it's still very doable.
You can turn a Stream<T> into an Iterator<T>, and since Iterable<T> is a functional interface whose single method supplies an Iterator<T>, you can make an Iterable<T> by wrapping a Stream<T> in a simple lambda:
I strongly doubt it. It's only mimicking the IEnumerable<T> bits of LINQ; there's nothing like IQueryable<T> in there.
I think that one thing a lot of people are missing when they talk about the Expressions bit of LINQ is that LINQ is really two different things that have two different killer features. There's the set of extensions on top of IEnumerable<T>, which are mostly used for writing declarative replacements for for-loops, and also supporting things like PLINQ. As far as I'm aware, IEnumerable<T>'s MSDN documentation doesn't even mention expression trees. And there's the (much more complicated) IQueryable<T> interface, which is the bit that supports things like the LINQ interface to Entity Framework, and is all about expression trees.
The IEnumerable<T> bits aren't as fancy, but they are still quite useful.
I don't think people understanding the problem LINQ attempts to solve with expression trees. Let me give you a more Internet'ish example. Let's say I have a REST api that's something like this:
https://movies.org/movie
which returns all movies
but if you want only certain movies, you issue:
https://movies.org/movie?year>2016
to get movies created after 2016.
Let's say you didn't issue the query but instead you got the entire set of movies and filtered locally:
var movies = get("https://movies.org/movie")
foreach( var movie in movies) {
if (movie.year > 2016) addToTable(movie)
}
Let's do this with LINQ:
foreach( var movie in movies.Where(movie => movie > 2016)) {
addToTable(movie)
}
The naive observer will assume the two pieced of code are about the same. But in actuality, if you had a fictitious "LINQ to Movies", it could, internally, construct the correct REST API for you.
What I'm saying is that, this code:
var list = movies.Where(movie => movie > 2016))
See how elegant it is compared to the other examples using Java, Node, etc.
Doing this requires meta-programming, which many other languages have. I don't know much about Clojure, Scala or Ruby, but I'm pretty sure stuff like this can be (and possibly already are) built using the existing meta-programming constructs.
I agree that it’s very easy to use, but it’s extremely hard to implement new LINQ providers. When I needed that, I implemented the subset of the functionality without LINQ (1). See the usage example (2).
If you find jinq useful, I'd like to also suggest a pet project of my own:
http://jpad.io/
It let's you run java snippets and view and share results. e.g. Any collections get converted to HTML tables with a column for each "getter".
52 comments
[ 3.8 ms ] story [ 98.4 ms ] thread[0] - https://vivekragunathan.wordpress.com/2016/04/02/jinq/
The same LINQ expression once translated to expression trees by the runtime can then be translated to SQL, MongoQuery,C# IL, etc at runtime depending on the linq provider.
https://blogs.msdn.microsoft.com/charlie/2008/01/31/expressi...
https://stackoverflow.com/questions/623413/expression-trees-...
var seniors = customers.Where(c => c.age > 65)
Could either represent C# IL, a MongoQuery, a SQL query, an ElasticSearch query, etc. depending on what type of variable “customer” is.
Also, you don't need a better term than "provider" as that's what they're called. :-) The IQueryable interface includes a property called Provider that returns an IQueryProvider which can operate on an expression tree.
To be further pedantic: it creates an expression when the method accepts an Expression<T>, which IQueryable happens to do.
That's the difference between IQueryable and IEnumerable. If you use IEnumerable, it'll filter on the client side rather than the server side. Potentially a performance and resource and even security/privacy problem. With IQueryable, the filtering is done on the server side.
Personally, the best thing to do is to use stored procedures. ORMs and query generation on the client side is such a bad idea. But it's easy, so people go for the easy solution.
Also, it’s a lot easier and less error prone to compose dynamic queries with Linq - ie keep chaining Where clauses and OrderBy’s based on a condition.
You can also do things like create methods that take predicates.
CustomerRepo.Find( c=>c.age > 65)
And define the Find method.
List<Customer> Find(Expression<Func<Customer,T>> predicate).
I’ve even had one project where the backing store could either be SQL Server or Mongo. We were able to use the same interface with methods like the Find method above and the repos translated the predicates appropriately.
List<Customer> Find(Expression<Func<Customer,T>> predicate)
Should be
List<Customer> Find(Expression<Func<Customer,bool>> predicate)
That's the general assumption of people not familiar with SQL, RDBMs and full stack architecture. One of my issues is that ORMs and LINQ have become crutches. Also, you are ignoring the major problem of security and auditing. If you are working on a local toy project, then sure, you can just do whatever you want. But if it is an enterprise project with real security concerns, the idea of giving read/write access to web server accounts is amateur hour. Trust me, I've seen it with my own eyes. People who spend 2 days reading "Learn web dev in 21 days" and "Learn SQL in 21 days" and putting today something akin to "Hello world" projects because their managers didn't know any better.
> - ie keep chaining Where clauses and OrderBy’s based on a condition.
You can use LINQ, chaining, lazy evaluation and all that on data/Enumerables/sequences retrieved from a stored procedure.
But hey, what do I know.
For fun, you can further reduce the size of the code above (and IMO, make it clearer), but using `Arrays.stream(product)` and a method reference for the `map`:
Java streams read data from spinning rust and then does the selection in RAM. LINQ can send the "where" to the RDBMS and the RDBMS can use an index/btree lookup to read just the data needed where java streams would read umpty megabytes.
https://news.ycombinator.com/item?id=17414383 BTW
I don't know whether the JINQ people plan to write a good query generator. But if I were the maintainer, my plan would definitely be to focus on the API first and only then work on optimal backend implementation(s).
That started out as a LINQ port but turned much more interesting.
Life would be much easier for everyone with expression trees in the language, though.
But it all feels a bit too "magical" for me. Seems like you're trading one class of bugs - hand-written SQL mismatching your types - for another - this magic doing something unexpected when you try to do something weird. But maybe it's really really good and that second class of bug never happens. I'm just hesitant (and happy with doobie for now).
So, jOOQ will never become quill or end up looking like it.
A pet peeve of mine is how these libraries put the 'old' alternative in the worst possible light, creating the fake impression (at a casual glance) that the library makes your code much nicer. It doesn't.
Passing in an array is weird, so lets assume we're not working with really outdated codebases and a List<Product> is passed in instead.
You can do the snippet with bog standard Java8+ stream API like so:
Even in old style java6 syntax it's short, sweet, and to the point (as short as this library can let you write it): Separate from that, as scarface74 mentioned, part of the point of LINQ is that the structure of the expression tree can be applied elsewhere, for example, to an SQL statement.But most of what I've seen LINQ to Objects used for is simply mapping over data that already lives in memory. Oftentimes you want to just pass along a lazy projection of that data, on the grounds that re-running the operation is less expensive than allocating a bunch more memory in order to cache the result to a list.
In the cases where a source can't be restarted, it's always seemed easy enough to me to just build up your LINQ query and then pass its enumerator back rather than the original enumerable. Returning an IEnumerator<T> should be sufficient to communicate that you only get to enumerate the thing once, and tacking a ".GetEnumerator()" on the end of things requires barely more typing than tacking ".ToList()" on the end.
By contrast, with streams, I feel like the Java maintainers gave me an API that's less powerful than the standard I'm used to on other platforms, on the grounds that they think I'm not a smart enough developer to handle a more powerful API. I don't really appreciate that. All it's really meant is that I now need to work with two different libraries depending on whether I'm working with data that lives in memory or not, which does worse things to the cognitive load it takes to understand the codebase than simply having to keep track of the difference between an Iterable<T> and an Iterator<T> would have.
I think that one thing a lot of people are missing when they talk about the Expressions bit of LINQ is that LINQ is really two different things that have two different killer features. There's the set of extensions on top of IEnumerable<T>, which are mostly used for writing declarative replacements for for-loops, and also supporting things like PLINQ. As far as I'm aware, IEnumerable<T>'s MSDN documentation doesn't even mention expression trees. And there's the (much more complicated) IQueryable<T> interface, which is the bit that supports things like the LINQ interface to Entity Framework, and is all about expression trees.
The IEnumerable<T> bits aren't as fancy, but they are still quite useful.
var movies = get("https://movies.org/movie") foreach( var movie in movies) { if (movie.year > 2016) addToTable(movie) }
Let's do this with LINQ: foreach( var movie in movies.Where(movie => movie > 2016)) { addToTable(movie) }
The naive observer will assume the two pieced of code are about the same. But in actuality, if you had a fictitious "LINQ to Movies", it could, internally, construct the correct REST API for you.
What I'm saying is that, this code: var list = movies.Where(movie => movie > 2016))
would translate into:
var movies = get("https://movies.org/movie?year>2016")
Not so hypothetical. This is how OData works from C#.
Look at the C# example under "Step 2: Requesting an individual resource" at http://www.odata.org/getting-started/understand-odata-in-6-s...
See how elegant it is compared to the other examples using Java, Node, etc.
Doing this requires meta-programming, which many other languages have. I don't know much about Clojure, Scala or Ruby, but I'm pretty sure stuff like this can be (and possibly already are) built using the existing meta-programming constructs.
(1) https://github.com/Const-me/EsentSerialize/tree/master/Core/... (2) https://github.com/Const-me/EsentSerialize/blob/master/Demos...
2. JINQ (as a name) already exists: http://www.jinq.org