Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user.
Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.
I read about Hobbes years ago. The embeddability certainly caught my attention.
I wrote Empirical specifically for the use case of wanting to read a CSV file while inferring the type at compile time. Basically, a ton of compile-time function evaluation mixed with type providers.
Yeah, I had this type constraint system for doing that kind of thing, so I can load a file at compile time to get the type structure and then compile/evaluate to process data in the file or files. Same thing for network connections between communicating processes (type constraints at compile-time decide a protocol, machine code at run-time assumes that protocol).
I recently did a project using Dyalog APL. I really wanted to love it.
On the pro side, the community was great and I loved the language, but only to a point. There are way too many instances of utter astonishment at behaviors that seemed to be just how the language works. I had a lot of trouble finding good, mature libraries for things I’ve taken for granted in other languages.
I’ve found that Julia provides 98% of the Array Programming value in a modern “batteries included” package.
This reminds me of what a BASIC programmer friend of mine said about C when he first tried it. After many (mostly senseless) debates, I eventually convinced him to try C after showing some bodged-together flashy graphics demo.
He got me to teach him basic C syntax and then took off on his own, writing a game. After a few weeks of hacking, he presented me with a decent Asteroids clone, "I don't see the point of C."
The funny thing is, when I looked at his main.c file, the entire code was written inside the main function, with copious use of gotos. He had essentially tried to speak BASIC using C.
APL isn't an Algol-like, and you're probably going to have a bad time writing APL trying to use the cognitive tools and intuitions you have as a programmer of Algol-like or even functional languages. This includes the tendency to reach for libraries, when you can, for example, implement an entire U-Net CNN in 30 lines of APL and be competitive with PyTorch performance.
In my (budding) experience, to "get" APL you have to laser-focus on writing beautiful, concise code from the onset. It feels painful as a student to spend weeks writing and re-writing just to produce 2 lines of code, but in the end those 2 lines will be doing the work of 20 lines of Julia, or 200 lines of C, and feel like a pure, direct mathematical expression of the algorithm you intend.
I think this is the first comment I’ve seen that describes my general approach to learning and designing programming languages. The only other topics such statements are likely to be made are concerning Lisp (and it `mind expanding` properties).
I really wish more people would focus on using/making languages that cater to a strict niche but compile to a sensible and readable common root language/IR.
As an array language developer I may be a bit heretical in the community for my view that array languages are a domain-specific tool, not a general-purpose tool. They are the best by far in their niche of working with arrays, but they can be harder to fit to some other kinds of tasks, especially when it comes to more semantic programming.
That's why my APL compiler, April (https://github.com/phantomics/april), can be called within Common Lisp with CL data structures as its input and output. It's also trivial to port functions from CL and its libraries into an April workspace. You can see an example on page 2 of this paper: https://zenodo.org/record/6381963
In this way April puts the entire CL ecosystem at your fingertips. If you want to do something like making HTTP requests or ingesting XML files, you can write a function to do it and express the specific parameters for your use case in CL, and then have the function available within April using a simple monadic/dyadic argument API. This can sidestep the need to port every necessary library into APL itself.
Oh, I have to try this! APL is in the family tradition, I literally learned it from my mom. Probably the first programming language I sincerely enjoyed! I sure do miss the glyphs. Mom uses R now and often notes some things used to be easier with APL…
Well this is unexpected - didn't think my little study group would appear on HN! :O Well anyhoo, all are welcome to join; details in the linked thread (including playlist of last week's videos so you can catch up).
As it happens, my interview on The Array Cast was just published today:
In it I discuss my thoughts on array programming, including in PyTorch, numpy, Tensorflow, and other libraries and languages. If you haven't seen it before, The Array Cast is a really great podcast including interviews with some of the most interesting folks in the array programming world. My fave episode so far is the one with Aaron Hsu (@arcfide) of co-dfns fame:
My personal interest in APL is for teaching (my daughter and her friend who I tutor really like learning math via APL, and we've successfully covered territory that previously I'd had no luck making progress on using more traditional methods) and for studying notation (which is what APL was originally designed for).
I also think schools are missing out massively, but not only by not teaching them at every chance, but because they do not combine these things. If I had been taught math and computer programming combined, implementing things one learns, damn, I would have been waaay more interested in math in later school years and that would probably have lead to me being much better at math later on at university too. It just gives you something practical to do and a feeling of success, when you make the computer do it for you, while at the same time you learn about computer programming.
I hope some day we see those 2 things married into one superb class at schools.
I don't know if APL supports graphics, but ngn/k, an implementation of K, a language in the APL family, has JS support in its official online playground [0], which can be used for displaying graphics.
The author also has this video: https://dyalog.tv/APLSeeds21/?v=iC9floP7POU "Tomas takes you through the API of a 3D-engine and shows you how to connect to it from APL, enabling you to create a simple scene with graphics objects moving in real-time. He also explores some of the more advanced rendering techniques." (I haven't watched it)
I did a bit of the last AoC in BQN. I really liked the language but got stuck a bit when it came to the more tricky loops where you had to do some reassignment if I remember correctly... I should go back and finish the rest of it. Saw there were a few others that had solutions I could take inspiration and learn from.
28 comments
[ 4.8 ms ] story [ 60.9 ms ] threadhttps://code.kx.com/q/ref/
Now days we refer to tables-within-a-language as "dataframes". These are widely available in R, pandas, Spark, etc.
The q language inspired me to write my own language a few years back that added static typing to dataframes:
https://www.empirical-soft.com
Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.
I built a similar typed PL and compiler for Morgan Stanley back in 2013: https://github.com/morganstanley/hobbes
I wrote Empirical specifically for the use case of wanting to read a CSV file while inferring the type at compile time. Basically, a ton of compile-time function evaluation mixed with type providers.
On the pro side, the community was great and I loved the language, but only to a point. There are way too many instances of utter astonishment at behaviors that seemed to be just how the language works. I had a lot of trouble finding good, mature libraries for things I’ve taken for granted in other languages.
I’ve found that Julia provides 98% of the Array Programming value in a modern “batteries included” package.
He got me to teach him basic C syntax and then took off on his own, writing a game. After a few weeks of hacking, he presented me with a decent Asteroids clone, "I don't see the point of C."
The funny thing is, when I looked at his main.c file, the entire code was written inside the main function, with copious use of gotos. He had essentially tried to speak BASIC using C.
APL isn't an Algol-like, and you're probably going to have a bad time writing APL trying to use the cognitive tools and intuitions you have as a programmer of Algol-like or even functional languages. This includes the tendency to reach for libraries, when you can, for example, implement an entire U-Net CNN in 30 lines of APL and be competitive with PyTorch performance.
In my (budding) experience, to "get" APL you have to laser-focus on writing beautiful, concise code from the onset. It feels painful as a student to spend weeks writing and re-writing just to produce 2 lines of code, but in the end those 2 lines will be doing the work of 20 lines of Julia, or 200 lines of C, and feel like a pure, direct mathematical expression of the algorithm you intend.
I really wish more people would focus on using/making languages that cater to a strict niche but compile to a sensible and readable common root language/IR.
That's why my APL compiler, April (https://github.com/phantomics/april), can be called within Common Lisp with CL data structures as its input and output. It's also trivial to port functions from CL and its libraries into an April workspace. You can see an example on page 2 of this paper: https://zenodo.org/record/6381963
In this way April puts the entire CL ecosystem at your fingertips. If you want to do something like making HTTP requests or ingesting XML files, you can write a function to do it and express the specific parameters for your use case in CL, and then have the function available within April using a simple monadic/dyadic argument API. This can sidestep the need to port every necessary library into APL itself.
Oh, I have to try this! APL is in the family tradition, I literally learned it from my mom. Probably the first programming language I sincerely enjoyed! I sure do miss the glyphs. Mom uses R now and often notes some things used to be easier with APL…
As it happens, my interview on The Array Cast was just published today:
https://www.arraycast.com/episodes/episode31-jeremy-howard
In it I discuss my thoughts on array programming, including in PyTorch, numpy, Tensorflow, and other libraries and languages. If you haven't seen it before, The Array Cast is a really great podcast including interviews with some of the most interesting folks in the array programming world. My fave episode so far is the one with Aaron Hsu (@arcfide) of co-dfns fame:
https://www.arraycast.com/episodes/episode19-aaron-hsu
My personal interest in APL is for teaching (my daughter and her friend who I tutor really like learning math via APL, and we've successfully covered territory that previously I'd had no luck making progress on using more traditional methods) and for studying notation (which is what APL was originally designed for).
I hope some day we see those 2 things married into one superb class at schools.
[0]: https://ngn.codeberg.page/k/#
The author also has this video: https://dyalog.tv/APLSeeds21/?v=iC9floP7POU "Tomas takes you through the API of a 3D-engine and shows you how to connect to it from APL, enabling you to create a simple scene with graphics objects moving in real-time. He also explores some of the more advanced rendering techniques." (I haven't watched it)
And J is almost isomorphic to APL.
APL also has support for GUI apps (Windows only) using .NET, but graphics takes some additional effort: https://www.youtube.com/watch?v=iC9floP7POU
https://beyondloom.com/tools/specialk.html
https://youtube.com/watch?v=AUEIgfj9koc
Related HN story: https://news.ycombinator.com/item?id=24434717
More pics and videos at: https://bloxl.co
My April APL compiler has a terminal graphics application as one of its demos, which you can see in this folder: https://github.com/phantomics/april/tree/master/demos/ncurse...
If you are interested in this deep interconnection and what future might hold, please check out New Kind of Paper series. [0]
[0]: https://mlajtos.mu/posts/new-kind-of-paper
FWIW, s/effect/affect/ .