Thanks, I think I can actually apply this idea actually. I'm writing a chat app for fun and one of the (admittedly small) issues I'm dealing with is cleanly storing the local state of the chat so that it appears correctly on refresh. There's some complexity involved with how messages are served, received, etc. with notifications, dings, animations, whatever -- but maybe saving that information to a file (as an alternative to localStorage or database read) could be a robust alternative to whatever I was trying to do. I'll give it a try.
You need something to deal with follows (cryptographically verify that the follow attempts actually originate from the servers they claim to) and submitting posts to your followers (push based, not pull based in practice) but all the metadata and user lookup stuff can be done with JSON files in the right place.
Rails, by default, will put a “.json” suffix on json routes. This is so supremely useful if you have a Rails api app, speaking to some over complicated front end that might need a quick and dirty mock for something. Just serve your /api/thing.json from somewhere. Only issue is with HTTP verbs other than GET, but usually at that point I need a lot more than this 15second mock. Good for testing little things though.
Not from a modern developer. You need to jump down the rabbit hole of the history of solid and proven solutions.
Indeed. I've noticed an increasing number of people who call themselves "developers" and appear to create software, but all they can do is follow step-by-step tutorials to glue together some massively bloated thing that they have absolutely no understanding of, much less the skills to debug it when something doesn't work perfectly as expected.
At some point, the plane flew so high above the clouds, and it hasn't seen the ground for so long, the passengers might think the clouds is the ground.
When "Hello World" is a cloud instance spun up and billed on your tuition.
That won't really happen, right? No Computer Science (CS) professor would designate special cloud Interactive Development Environments (IDEs) to students.
On the other hand, it would really resolve a lot of "doesn't work on my machine" issues.
I think that would be a great way to make a weeder course?
Setup some obscure software, with poor documentation, build something nice on top of it, then rewrite it because some fundamental assumption is broken, sounds like this kind of first year of cs course we should have.
The idea being weeding out those not as passionate about computers.
Good way to catch unaligned access errors. I seem to recall a similar "you can develop on whatever you want, but to turn it in it needs to work on the Dept systems" back in my college days. Intro to CS was in pascal, and a bunch of us started with Turbo Pascal on our PCs, and figured out pretty quick the Pascal on the AT&T SVR3 boxen was a very different beast.
Azure and AWS both have academic/educational resource buckets and discounts.
GitHub has said one of the largest growing areas for Codespaces (cloud IDE) usage is Education.
It's already happening. It does resolve a lot of "doesn't work on my machine" issues; it does leave some questions about how much students see of the "ground" below the Clouds.
To be fair, I would also say the mainstream IDEs have contributed to this situation to a certain extent, since they lower barrier of entry. That's a very good thing, but also sometimes bad.
Here's what I mean. There's a difference between:
* Using an IDE knowing what you want to do but you just like the convenience. You know what the IDE is doing for you, but you want your tool to do it for you quickly and get out of the way. If the IDE does something you don't know, you try to learn what this was.
* Using an IDE because you literally can't work without one. Not knowing your programming language and depending 100% on autocomplete to do most stuff. Not even knowing how to write the files that the IDE automatically updates when doing certain things.
It's a subtle difference, but the former is more likely to leave better comments in a web-based code review ("pull request") than the latter.
And if a project consists only of the latter type, it's unlikely that there will be comments like:
* "Use this other class with the same name but different package because this one is buffered" (because depending on autocomplete without knowing the language prevented them from even caring about which import to use, or the difference between those, as long as it works).
* "Would this change affect this other file that's not in the PR but calls this function a thousand times per request?" (because not knowing their own projects general structure due to depending on their IDE for navigation, so they're more likely to NOT notice a file that's not mentioned in the PR).
So if a project has only people that depend on an IDE (instead of using an IDE for convenience), it's easy to end up with stuff that technically works, but copies 40MB of strings around multiple times, and thrashes your performance due to reading one character at a time without buffering.
Granted, this technically is not related to IDEs, but in my experience (emphasis on "my"), most of the people I've worked with that use an IDE, are of the "can't program without it" type, and not the "I use it because it's convenient" type.
Not all of them. But most. Even if there's a learning curve at play here, I'd say there's usually overlap between "use IDE because convenient" and "actually bothering to put an effort into learning".
I understand this is an inevitable consequence of lowering the barrier of entry and making programming more accessible, and it doesn't matter if we're talking about npm, IDEs, programming languages, or anything.
> I understand this is an inevitable consequence of lowering the barrier of entry and making programming more accessible, and it doesn't matter if we're talking about npm, IDEs, programming languages, or anything.
Pretty much this. Anybody new to the field will be using an IDE, it's extremely unlikely that someone new will be using vim/emacs. So IDE users will be a mix of newbies and experienced people, but since the field grew so much in recently, the number of newbies using IDEs will outnumber the experienced people using them. In contrast, people using vim/emacs will consist mostly of people with a lot of experience at this point.
So the average IDE user will be less experienced than a non-IDE user, but I don't think it has anything to do with the fact that they are using an IDE. I wouldn't attempt to pre-judge anyone's ability based on if they use an IDE or not.
> I wouldn't attempt to pre-judge anyone's ability based on if they use an IDE or not.
Neither would I, but I will certainly judge them as lacking if they can't write a nontrivial program without using code completion or looking things up on SO/Github/etc.
Of course not, and the bar isn't arbitrary. That you ask this tells me that I was insufficiently clear as to my stance. My stance is that programmers should understand how to write programs, not just copy/paste them together.
I certainly have way too much in my head to remember every nuance of several languages. As such, I lean heavily on multiple sources of external documentation to help me recall the specifics. That being said, this would in no way replace actual software development expertise, so I agree with your clarified stance.
I find using a good IDE with code completion and navigation is essential to efficient use of my time. I _also_ use vi constantly and perform most every other operation from the command line. I know I can get the same from properly setting up vi with the right plugins, but I find the ergonomics less appealing for my habits.
You might not want them in your team maybe, but isn’t it really cool we reached a state where you can actually do this? Democratization of software engineering is a great good!
Dunno if it actually is democratizing software engineering. Looks more like a cargo cult to me. You get a second class of developers that don't quite get it. Not because they aren't capable, but because nobody shared the secrets with them and they had to piece together everything from flotsam.
Cargo cult programming is a starting point. Plenty of people won't go any further, but when what you're putting together is all source code, it's mostly transparent boxes you're putting together. You can look inside and see the secrets whenever you want, if you ever want.
Sure, maybe there's some design principles to share that aren't obvious. But it's hard to share and teach those before someone has experience with at least putting stuff together. Also, it's not like there are principles that are universally accepted or universally applied by trained software engineers.
I'd say it's more the opposite. What characterizes cargo cult development is having too many principles and rules, rather than lacking them, they're rules without any real understanding of why they exist and what they're for or when they're applicable. As long as you adhere to the strictures, good code emerges somehow.
The way to bring a cargo cult developer into the light typically involves debunking or contextualizing a whole bunch of weird ideas.
Maybe you and I have a different definition of cargo cult developers than I do. I'm thinking cut and paste from stack overflow (or now chatGPT) without much understanding or consistency.
Knowing how to apply design principles and rules despite not fully understanding them is a different level, in my mind.
Democratization of software engineering means giving everyone the opportunity to learn. Not handing them a black box with a few instructions then sending them off to plow head first into a brick wall of insurmountable knowledge gaps so they give up.
Democratization of software means that more people than ever can make things. This is absolutely the case with software today. We have kids creating games with drag-and-drop visual programming languages. One developer can get a worthy full-stack prototype in front of an investor in a month (or less!), and these stories are gaining steam.
Soon we may have no-code solutions getting good enough to warrant broader adoption. No-code solutions are as black box as you can get, and it's also as anti-learning as you can get in terms of the underlying technology and its theoretical principles. But this is also an example of more people than ever making things.
Someone wrote "if people are using your tool to turn out a lot of half-baked garbage, then your tool is a huge success", which really reframed my view on these things. It was originally in the context of one of the "game maker" tools, but applies to any kind of simplification.
A tool which only allows perfect solutions by perfect users is one that rejects a lot of work and prevents a huge number of projects from ever happening at all. Useful for safety-critical work, but not everything is safety-critical.
Was this is a joke or something? You could have always done this with whatever framework you were using including Express, FastAPI, Flask & everything else....
They do host some data, but they delegate a lot of connections as well. One of their biggest products is bot mitigation, where they'll inspect incoming connections/traffic and pass them through if they don't seem to be bots.
But i think they terminate the SSL connection, and then re-establish a new one right? It's not like your backend server didnt know it's been MITM'ed. I suppose they should've named MITM as Surprise-MITM.
This technique of producing an “API” from directories of json generated by batch jobs and creative use of symlinks can be surprisingly effective. I wrote a simple program to transform a list of RSS feeds into a directory tree that I’ve been using for years as my feed reader API to great effect.
I really like endoflife! It's great for looking up dates with a consistent UI and also codenames - I could never learn that Debian Bullseye was 11 or all the other silly codenames so I look them up here. Thank you!
We also do something similar on Datatig, which is a tool I'm writing for the general situation of crowdsourcing data in Git repos. It makes it easier for people to submit data and then creates usable data and handy websites. The static website output has an API which is just static JSON files. https://pypi.org/project/DataTig/ or https://datatig.readthedocs.io/en/latest/
Yeah, I've always tried to generate RSS feeds when needed rather than on-demand — it strikes me that the ratio of requests to updates must be so enormous that it doesn't make sense doing anything else.
At my work I struggle with the opposite: all problems are being squeezed into "let's put it into static JSON on the CDN" - which ends up with a complex custom JSON based language (schema) to support sharing information between apps, subsetting information (ie. search) etc - ie. implementing an ad-hoc one-file database.
Ahh... and don't forget about complex CI/CD pipelines and Git setup that allow business users to manipulate data in these files.
I found "just use Postgres for everything" a much saner default in the end.
So it’s simple or complex because I got confused ;)
KISS is ALWAYS good. But simplicity doesn’t mean easy or naïve solutions. Naïve solutions get very complex since the initial input is low and they don’t cover edge cases by design making it tangled mess.
Simple solutions are often very difficult to create, require a lot of input to be simple, but they cover a lot of cases behind the curtains and are easy to describe.
Your “use Postgres” strategy is more KISS than doing complex mocking. Sure, initial effort is higher, but using well known, powerful yet ergonomic singular entity is as simple as one can get on some levels. It’s equivalent of “just use Excel for it”.
I agree with you, but I think tagline is misplaced.
That sounds like a PITA solution more than KISS. That said, I'd rather walk into that business and solve that problem, than walk into a business that is fully cloud serverless hooks all over the place.
So yes, Kiss is good until it isn't... because it wasn't kiss anymore. Moving to the postgres solution sounds like it's the new Kiss.
I don't know how this will ever not be a complete maintenance hell. It is even far worse than the monolithic "I push every bit that needs to be pushed for the whole enterprise - Sync Tool".
I propose a new paradigm: microenterprises. We limit the scope of the responsibility of each monolithic enterprise Sync Tool to something more digestible by limiting the scope of the enterprise itself.
The crazy habits proponent in your workplace is simply ahead of his time: there is the "SQLite for edge" vulgata now and he/she may skim some of the wheel reinvention and funny engineering problems with throwing around SQLite files, like in the good old Access days.
People should not cache something that changes very often.
I do this when possible. Caching can be a pain to get right though. If you’re set up to do hashed filenames for JS and CSS already, you can leverage that for cache busting your json, and then you need to inject the hashed filename into the app somehow.
does this, but more production-ready, honors cache headers, sets Etag, etc.
Also:
caddy respond
can be used to hard-code specific responses if you're testing an HTTP client. It can even spin up on a whole port range and supports templates in the response text right there on the command line:
caddy respond --listen :2000-2004 "I'm server {{.N}} on port {{.Port}}"
I'd say "easy to setup and run" is arguably caddy's entire point:) Single binary file, yes running is that easy for simple stuff, and honestly it's not exactly complicated for more complex stuff. It got a lot of its popularity because it made HTTPS, including automatic cert provisioning, utterly trivial.
It’s pretty easy, but I find the config file format rather confusing. In the classic Unix style, you’re learning yet another little language, so there’s a learning curve. Once you learn it, it’s fairly powerful and expressive, but the docs aren’t very clear.
Caddy excels at making simple [1] use-cases trivial - either with no config, or 3-5 lines of config.
It might seem jarring comming from Apache or nginx - but i think a lot of it comes from the developers having destilled a lot of common use-cases and carefully chosen reasonable defaults.
[1] Simple to articulate; not necessarily simple to execute - eg letsencrypt works out of the box. Caddy will give you https if you ask (or rather unless you go out of your way to avoid it).
It can be a great stop-gap in running an ssl proxy in front of a legacy application server stuck on some horrible old distro, for example - since the binary just needs a Linux kernel (and can ignore old ssl libraries).
NGINX felt jarring coming from Apache. The cycle continues.
To be fair to NGINX, there is a lot of boilerplate config, but if you have a good set of default files, the server blocks can be small. The same could be said for Apache, of course. But yeah, Caddy is great in how little config it takes, no boilerplate, to do what the incumbents do.
I was working on content heavy web app few years ago and realised that lot of database reads are happening to very few "popular" posts then it hit me that we can just host that content as json files using Nginx without API getting involved. That worked like a charm. Decreased usage of database, decrease in latency. etc. In hindsight we could have just used CDN but for the shoestring budgets we had, we achieved lot of scalability with single nginx server.
The python server is not meant for production, serving files from it is incredibly slow and resource intensive. It also is not full webserver. There are better options for serving files
Adding that to the blog post would strengthen the argument.
We shouldn't assume the knowledge level of the reader.
Examples become production all the time.
Using files with unix philosophy is a tried and tested way to be really productive.
Some time ago, a website was posted here (forgot the topic) which consisted of .md files, and a JavaScript library which would then, on the client, render the markdown on the fly.
Why on earth the markdown was not pushed through a renderer so that then the server could serve static HTML is probably a question that only occurs to old timers like us.
I built and API for an internet radio station archive. Initially I tried to build an app but instead I realised I can run a scheduled Github Action that runs every hour, scrapes the archive, builds a Json file and deploys it to Github Pages for free: https://bd.maido.io/api.json
Had a smile reading through this. So many web developers out there who have no idea about history of HTTP. That it was designed for static document serving and since been twisted to serve applications.
For a 'fake' API, I can't overstate just how powerful this is.
I once spent an entire month building a proper API server out of this. Then one day while tweaking some DTOs for some POST endpoints, I realised what I was doing and switched to Postgres.
This is great advice and its most of the time everything you need for a personal website.
A generation that grew up with Wordpress et al. will have dificulties getting that really all there is to a website is some html. I switched from a Spring Boot web app that used Thymeleaf for the templating engine, with a MySQL db on a Linux vps and Heroku (I did it to learn/practice) to a Hugo blog that I publish locally then scp to a VPS with Caddy setup to serve the static content. Nothing to hack/maintain/worry about.
As an extra point I would even complete the suggestion with adding direct HTTP not S, support. Your page will load even faster and you're transmitting some static content anyway.
126 comments
[ 2.6 ms ] story [ 140 ms ] threadBest as I can tell the part that might be non-compliant here is that new followers should be added to the follower collection.
You need something to deal with follows (cryptographically verify that the follow attempts actually originate from the servers they claim to) and submitting posts to your followers (push based, not pull based in practice) but all the metadata and user lookup stuff can be done with JSON files in the right place.
Makes it super simple to implement mocks of your api design before building the api.
…I guess that was kind of the point of REST in the first place but it’s easy to forget.
REST has a wonderful simplicity to it.
I’m yet to hear about an equivalent caching solution for GrqphQL APIs for example.
Indeed. I've noticed an increasing number of people who call themselves "developers" and appear to create software, but all they can do is follow step-by-step tutorials to glue together some massively bloated thing that they have absolutely no understanding of, much less the skills to debug it when something doesn't work perfectly as expected.
That won't really happen, right? No Computer Science (CS) professor would designate special cloud Interactive Development Environments (IDEs) to students.
On the other hand, it would really resolve a lot of "doesn't work on my machine" issues.
Setup some obscure software, with poor documentation, build something nice on top of it, then rewrite it because some fundamental assumption is broken, sounds like this kind of first year of cs course we should have.
The idea being weeding out those not as passionate about computers.
GitHub has said one of the largest growing areas for Codespaces (cloud IDE) usage is Education.
It's already happening. It does resolve a lot of "doesn't work on my machine" issues; it does leave some questions about how much students see of the "ground" below the Clouds.
Welcome to the world of npm and supply-chain hell
Here's what I mean. There's a difference between:
* Using an IDE knowing what you want to do but you just like the convenience. You know what the IDE is doing for you, but you want your tool to do it for you quickly and get out of the way. If the IDE does something you don't know, you try to learn what this was.
* Using an IDE because you literally can't work without one. Not knowing your programming language and depending 100% on autocomplete to do most stuff. Not even knowing how to write the files that the IDE automatically updates when doing certain things.
It's a subtle difference, but the former is more likely to leave better comments in a web-based code review ("pull request") than the latter.
And if a project consists only of the latter type, it's unlikely that there will be comments like:
* "Use this other class with the same name but different package because this one is buffered" (because depending on autocomplete without knowing the language prevented them from even caring about which import to use, or the difference between those, as long as it works).
* "Would this change affect this other file that's not in the PR but calls this function a thousand times per request?" (because not knowing their own projects general structure due to depending on their IDE for navigation, so they're more likely to NOT notice a file that's not mentioned in the PR).
So if a project has only people that depend on an IDE (instead of using an IDE for convenience), it's easy to end up with stuff that technically works, but copies 40MB of strings around multiple times, and thrashes your performance due to reading one character at a time without buffering.
Granted, this technically is not related to IDEs, but in my experience (emphasis on "my"), most of the people I've worked with that use an IDE, are of the "can't program without it" type, and not the "I use it because it's convenient" type.
Not all of them. But most. Even if there's a learning curve at play here, I'd say there's usually overlap between "use IDE because convenient" and "actually bothering to put an effort into learning".
I understand this is an inevitable consequence of lowering the barrier of entry and making programming more accessible, and it doesn't matter if we're talking about npm, IDEs, programming languages, or anything.
Pretty much this. Anybody new to the field will be using an IDE, it's extremely unlikely that someone new will be using vim/emacs. So IDE users will be a mix of newbies and experienced people, but since the field grew so much in recently, the number of newbies using IDEs will outnumber the experienced people using them. In contrast, people using vim/emacs will consist mostly of people with a lot of experience at this point.
So the average IDE user will be less experienced than a non-IDE user, but I don't think it has anything to do with the fact that they are using an IDE. I wouldn't attempt to pre-judge anyone's ability based on if they use an IDE or not.
Neither would I, but I will certainly judge them as lacking if they can't write a nontrivial program without using code completion or looking things up on SO/Github/etc.
Does this arbitrary bar extend to language and library documentation as well?
I certainly have way too much in my head to remember every nuance of several languages. As such, I lean heavily on multiple sources of external documentation to help me recall the specifics. That being said, this would in no way replace actual software development expertise, so I agree with your clarified stance.
I find using a good IDE with code completion and navigation is essential to efficient use of my time. I _also_ use vi constantly and perform most every other operation from the command line. I know I can get the same from properly setting up vi with the right plugins, but I find the ergonomics less appealing for my habits.
Sure, maybe there's some design principles to share that aren't obvious. But it's hard to share and teach those before someone has experience with at least putting stuff together. Also, it's not like there are principles that are universally accepted or universally applied by trained software engineers.
The way to bring a cargo cult developer into the light typically involves debunking or contextualizing a whole bunch of weird ideas.
Knowing how to apply design principles and rules despite not fully understanding them is a different level, in my mind.
Soon we may have no-code solutions getting good enough to warrant broader adoption. No-code solutions are as black box as you can get, and it's also as anti-learning as you can get in terms of the underlying technology and its theoretical principles. But this is also an example of more people than ever making things.
A tool which only allows perfect solutions by perfect users is one that rejects a lot of work and prevents a huge number of projects from ever happening at all. Useful for safety-critical work, but not everything is safety-critical.
As in be a terrorist and produce bombs? These are all ticking time bombs.
Relevant: https://xkcd.com/1988/
Lego's dream has come true.
But only for prototyping and testing the auth and database tools.
Haven’t really made a decision on it vs rolling my own in Rails.
[1] https://supabase.com/
[2] https://github.com/supabase/supabase
It’s been running unattended for a decade….
This is essentially MITM-as-a-service.
We also do something similar on Datatig, which is a tool I'm writing for the general situation of crowdsourcing data in Git repos. It makes it easier for people to submit data and then creates usable data and handy websites. The static website output has an API which is just static JSON files. https://pypi.org/project/DataTig/ or https://datatig.readthedocs.io/en/latest/
At my work I struggle with the opposite: all problems are being squeezed into "let's put it into static JSON on the CDN" - which ends up with a complex custom JSON based language (schema) to support sharing information between apps, subsetting information (ie. search) etc - ie. implementing an ad-hoc one-file database. Ahh... and don't forget about complex CI/CD pipelines and Git setup that allow business users to manipulate data in these files.
I found "just use Postgres for everything" a much saner default in the end.
So it’s simple or complex because I got confused ;)
KISS is ALWAYS good. But simplicity doesn’t mean easy or naïve solutions. Naïve solutions get very complex since the initial input is low and they don’t cover edge cases by design making it tangled mess.
Simple solutions are often very difficult to create, require a lot of input to be simple, but they cover a lot of cases behind the curtains and are easy to describe.
Your “use Postgres” strategy is more KISS than doing complex mocking. Sure, initial effort is higher, but using well known, powerful yet ergonomic singular entity is as simple as one can get on some levels. It’s equivalent of “just use Excel for it”.
I agree with you, but I think tagline is misplaced.
So yes, Kiss is good until it isn't... because it wasn't kiss anymore. Moving to the postgres solution sounds like it's the new Kiss.
I don't know how this will ever not be a complete maintenance hell. It is even far worse than the monolithic "I push every bit that needs to be pushed for the whole enterprise - Sync Tool".
Best of both worlds.
People should not cache something that changes very often.
The fuller version of KISS is "keep it as simple as possible, but no simpler".
Also:
can be used to hard-code specific responses if you're testing an HTTP client. It can even spin up on a whole port range and supports templates in the response text right there on the command line: Here's an example maintenance page: https://caddyserver.com/docs/command-lineCaddy excels at making simple [1] use-cases trivial - either with no config, or 3-5 lines of config.
It might seem jarring comming from Apache or nginx - but i think a lot of it comes from the developers having destilled a lot of common use-cases and carefully chosen reasonable defaults.
[1] Simple to articulate; not necessarily simple to execute - eg letsencrypt works out of the box. Caddy will give you https if you ask (or rather unless you go out of your way to avoid it).
It can be a great stop-gap in running an ssl proxy in front of a legacy application server stuck on some horrible old distro, for example - since the binary just needs a Linux kernel (and can ignore old ssl libraries).
To be fair to NGINX, there is a lot of boilerplate config, but if you have a good set of default files, the server blocks can be small. The same could be said for Apache, of course. But yeah, Caddy is great in how little config it takes, no boilerplate, to do what the incumbents do.
That also tends to provide results faster than the database layer, so better latency and less database load.
It's not as fast as serving static content via nginx or a CDN though. :)
Yes. Just like you can put letters together and form words. It's kind of what it was designed to do.
Why on earth the markdown was not pushed through a renderer so that then the server could serve static HTML is probably a question that only occurs to old timers like us.
https://github.com/caddyserver/website/blob/master/src/docs/...
I once spent an entire month building a proper API server out of this. Then one day while tweaking some DTOs for some POST endpoints, I realised what I was doing and switched to Postgres.
A generation that grew up with Wordpress et al. will have dificulties getting that really all there is to a website is some html. I switched from a Spring Boot web app that used Thymeleaf for the templating engine, with a MySQL db on a Linux vps and Heroku (I did it to learn/practice) to a Hugo blog that I publish locally then scp to a VPS with Caddy setup to serve the static content. Nothing to hack/maintain/worry about.
As an extra point I would even complete the suggestion with adding direct HTTP not S, support. Your page will load even faster and you're transmitting some static content anyway.