The first line really hits me hard. There’s something so incredibly freeing about being a kid and doing stuff like coding. There’s simply no expectations. Even the smallest project felt like such an achievement. But now I code professionally and I don’t know how to turn off engineering brain. I don’t know how to be okay doing something poorly, but on my terms.
Writers often say that your first N books will be crap, and the answer to that problem is to just write more than N books. I feel that’s true of software as well - the first years of the magical programming I did without any expectations meant those projects were utter crap yet functional.
It's because you don't get to write apps from scratch enough. Everything is a POC. You get it working. Optimization is just dumb until the project is so big it doesn't fit in the developers brain anymore. Then you write tests for the core part so you can keep interating fast. You will see parts that clearly need refactorwd before you can move on.
Also, be aware of, but don't get hung up on, engineering best practices. Chances are that someone is going to see your POC and put it in front of a client or promise it to someone. It will enter service being less than perfect, if you are lucky.
When you pull this off people will tell stories about you and brag about you when you are not around. None of them will know anything about the code engineering.
> When you pull this off people will tell stories about you and brag about you when you are not around.
True!
> None of them will know anything about the code engineering.
Not true. In environments where it's very easy to fuck up (hyperconnected codebases, lots of legacy tech debt, fragile systems generally), programmers who create tools/libraries that let other engineers get work done without blowing their feet off or spending huge amounts of time on silly things do get praised. Maybe the praise comes from different people than in your example, but it definitely is rewarded.
Two caveats: this only happens in certain environments; some gigs are just culturally unable to care about quality tools (death march feature factories and the like). Also, you actually have to save time and reduce defect rates with your engineering work; starting an architecture-astronaut rewrite that isn't actually useful for other programmers' day-to-day work doesn't count. Making "one tool to rule them all" that requires a Ph. D or pairing session with the author doesn't count either.
I think you can add a mode to engineer brain, in which it's aware when it's kludging something, but kludging is the appropriate thing to do.
Might help to have enough experience that you have (well-placed) confidence in your gut feel for how much engineering a particular thing needs.
(More common is to not have an engineer mode, or to not have enough experience to do it well.)
If you don't have the kludge mode, try some things where kludges are outright necessary. One time I recall this was when writing Emacs extensions. For example, it used to be that you'd hit a point where normal programmatic navigation/parsing of the buffer was just too slow, and a well-placed regexp operation that worked 99.99% of the time (and the other 0.01% of the time it's OK enough, and the user understands) made the rest of the code viable.
Another example is personal open source projects, when you really want to experiment with an usual implementation approach, and you give yourself permission.
(Maybe don't do this when the open source is done only for resume purposes, where you're trying to demonstrate you follow all the current fashionable conventions. You're almost guaranteed that someday a new-grad on a public forum or not-very-technical hiring manager will stumble across that bit of code, and fixate on the one thing they think they recognize as a bad practice. Documenting the kludge and rationale, in, say, a code comment, is great practice, but, again, it will also draw the attention of the least-skilled. Much like, if you're relating an anecdote in a job interview, and implied are 20 things you did right and 1 thing you did brilliantly, and you note as an aside one mistake you made, most of that will whoosh over the head of the dimmest FAANG interviewer, but they'll latch onto that mistake you spelled out, as a wise insight they have, and that's what's going in their report. :)
Then, armed with an experienced multi-skilled brain, when your startup's imminent MVP launch is facing crazy constraints, you triage what bits need to be rock-solid so they'll absolutely work and not fail, what bits need creative kludging so you can hit your launch window for other key requirements, and what bits you need to mitigate any compromises. Ideally, you have the wisdom to know which is which, and can activate different skills for each kind of work.
That Turbo Encabulator needs to be well-machined, but the new sensor you just decided it needs for a test doesn't need a week to be drilled into the engine block, nor a mount to be designed and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or maybe wad of chewing gum will do.
All such great advice. The one time I’ve found success is when I figure out the entire scope of the little project up front. Then I find myself not even worrying about maintenance or scalability. I know I won’t need that extra class abstraction or modularity or whatever. I just don’t like planning at the start, but it’s probably, counterintuitively, part of the answer.
P.S. you opened a parenthesis and forgot to close it so now everything I read all day is part of your anecdote)
This is something I've had to train myself to overcome. The first step for me was having a place where I clearly signaled that this wasn't my best work. Where the rules were allowed to be broken. That place is my junk drawer of code[1].
I have a zsh alias `alias changes='git add . && git commit -am "Changes" && git push'` that I use for my commits in the repo.
This all may feel silly, but it's what I needed to get back to that time of playing. Where I never worried about the code. But where I also didn't feel I was wasting my time working on code no one could see. I'd definitely recommend trying something like that if you are struggling with it.
i considered a few times to make an alias like this but i feel like git is so precious to get right that i would rather type out each command just to "feel" what i'm doing and be active in it.
> I miss that direct connection. The fast feedback. The lack of making grand plans.
There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated?
But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do.
Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff.
These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...
It includes a reference to Backbone and Knockout JS, which were released in 2010, so presumably it was around that era. The database, though, was probably much older...
Have you had the pleasure of blowing young minds by revealing that production-grade databases come with fully fledged authnz systems that you can just...use right out of the box?
Databases have pretty robust access controls to limit (a sql user's) access to tables, schemas, etc. Basic controls like being able to read but not write, and more advanced situations like being able to access data through a view or stored procedure without having direct access to the underlying tables.
Those features aren't used often in modern app development where one app owns the database and any external access is routed through an API. They were much more commonly used in old school apps enterprise apps where many different teams and apps would all directly access a single db.
This is perfectly fine when you are driving some app that has a per-user experience that allows you to wrap up most of their experience in some blobs.
However I would still advise people to use a third normal form - they help you, constraints help you, and often other sets of tooling have poor support for constraints on JSON. Scanning and updating every value because you need to update some subset sucks.
You first point is super valid though - understanding the domain is very useful and you can get easily 10x the performance by designing with proper types involved, but importantly don't just build out the model before devs and customers have a use for anything, this is a classic mistake in my eyes (and then skipping cleanup when that is basically unused.)
If you want to figure out your data model in depth beforehand there's nothing wrong with that... but you will still make tons of mistakes mistakes, lack of planning will require last minute fixes, and the evolution of the product will have your original planning gather dust.
> Scanning and updating every value because you need to update some subset sucks.
Mirrors my experience exactly. Querying json can get complex to get info from the db. SQLite is kind of forgiving because sequences of queries (I mean query, modify in appliation code that fully supports json ie js, then query again) are less painful meaning it's less moprtant to do everytning in the database for performance reasons. But if you're trying to do everything in 1 query, I think you pay for it at application-writing time over and over.
> These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Really!? Are you building applications by chance or something else? Are you doing raw sql mostly or an ORM/ORM-like library? This surprises me because my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks. SQLite's fluid type system haa been a nice middle ground for me personally. For reference my application layer is kysely/typescript.
> my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks
Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
Practical example: I recently wrote my own SMTP server (bad idea!), mostly to be able to control spam (even worse idea! don't listen to me!). Initially, I thought I would be really interested in remote IPs, reverse DNS domains, and whatever was claimed in the (E)HLO.
So, I designed my initial database around those concepts. Turns out, after like half a million session records: I'm much more interested in things like the Azure tenant ID, the Google 'groups' ID, the HTML body tag fingerprint, and other data points.
Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent, but fortunately modern C# makes that easy as well.
And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
> And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
I do the same thing with serde_json in Rust for a desktop app sqlitedb and it works great so +1 on that technique.
In Rust you can also tell serde to ignore unknown fields and use individual view structs to deserialize part of the JSON instead of the whole thing and use string references to make it zero copy.
because they put everything in JSON. Migration means running a script to parse and edit each JSON item instead of letting the database do database things automatically.
> Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Also I don't understand why you're afraid of migrations, especially since you're the only developer on your own SMTP server.
> That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Did you miss that he’s using sqlite? The dev experience with a sqlitedb is way better than running yet another service, especially for personal projects.
Sqlite is used just as much as an application file format as it is a relational database.
> My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""
Which feels off by six generations or so of memes. It feels prior to "memes" existing in the first place (at least this modern iteration, pics with captions in them; not Dawkins' original verion). I'd guess it is, ironically, chronologically closer to, well, your username here.
How does MongoDB handle someone pulling the power cord out of the server? Because that’s another reason to use something like SQLite, and it often gets used in embedded systems.
You certainly would lose a lot of things, like a well supported path to linking with to the database engine, and a straightforward way to start to introduce relational tables as the project matures. Nothing completely insurmountable, of course, but carry a lot of extra effort for what benefit?
> Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Our solution for a similar situation involving semi-structured data (in postgres) was to double it up: put all the json we send/receive with a vendor into a json field, then anything we actually need to work on gets extracted into regular table/columns. We get all the safety/performance guarantees the database would normally give us, plus historical data for debugging or to extract into a new column if we now need it. The one thing we had to monitor in code reviews was to never use the json field directly for functionality.
This is exactly what I've tried (and failed at) doing! Can I ask how you handle normalization from vendor data when it contains relationships and multilevel nesting? How do you know when to create a new child table, and which ones to create, and their relationships etc. I haven't found a good balance yet.
Odd-shaped miscellaneous data that you only need to retrieve is a good candidate for a JSON field. Once you're heavily using some piece of data, or if you need to index it (which means you are heavily using it), you should insert the data in the database "properly".
If some vendor is giving you a list of categories you don't care about, there's no need to make a vendor categories table and a many-to-many link table until you actually need them.
The point is that putting data properly in the database lets you use database features on it and get database performance.
Basically what the other reply said - handle it the same as you would any complex data. You just don't need to handle all of the json immediately, only the parts you plan on using for the moment.
Recently I’ve been tempted to make an SMTP server that translates emails into a web-hook. Please tell me more horror stories so that I might be convinced not do it.
“Without the need for any migrations” seems like a weird one to me. Of all of the things I don’t like having to do, migrations are pretty low on my list - far below any scripting-in-prod or applying hacks or even just rudimentary business logic changes. Granted, I used to fear them a lot back when I was expected to apply them by hand rather than use CICD and testing to validate/apply them
> Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations.
> And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
That's a migration.
> Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent
That's an effect of not migrating - having to process null and absent fields instead of just null fields. After doing more of these, you'll run into the same thing that made people stop using NoSQL databases: with no schema, your code has to parse all previous versions of the data format and they probably aren't even well-documented. While an RDBMS can just set the new column to null in existing rows.
In my current company, we're using a similar approach: just shove everything into a JSON blob. If you need a constraint or an index, you can create a computed column (in PostgreSQL) that pulls out a field from JSON.
For the data schema, we're using Protobufs with buf validate. This works surprisingly well, you can use the same types in the backend API and on the frontend. We even have a cron job that reads all the data periodically and verifies that the JSON blobs conform to the schema. Our next intern is going to write a PostgreSQL extension to do that on commit/update :)
One real advantage of this approach is that it's easy to do stuff like "search everywhere".
Do you go to the trouble of updating individual values in objects using some kind of deep/partial updating function or do you just accept race conditions that come with updating full objects?
We use optimistic versioning, with a dedicated "version" field (that is actually always pulled out of the blob in all tables).
Classic fine-grained schemas are not that much different. A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.
In addition, our frontend apps also support offline mode. They can get all the relevant objects, and then operate on them locally. So our API was designed from the start to deal with conflicts.
> We use optimistic versioning, with a dedicated "version" field (that is actually always pulled out of the blob in all tables).
All well and good but you do need to handle failures elegantly in code. The nice thing about flat DB tables and SQL is you don't really have to care if your goal is to update a single column authoritatively. The state of the other values in the table are, often, immaterial. It gets even more complicated reconciling deeply nested conflicts in open schemas.
Not knocking your approach, it's just a trade-off I guess.
Yeah, makes sense. To write safe code though at that point don't you have to enforce rigid schemas?
An example would be updating Kubernetes resources, the admission controller will verify the correctness of the change you are trying to make and reject the patch if it's non-comformant. Nested values have... value... in terms of contextualizing and isolating/localizing data leaves but at the end of the day aren't you still dealing with strict schemas and, when those schemas change you have to reconcile schema migrations?
> To write safe code though at that point don't you have to enforce rigid schemas?
Certainly. But you have to do that with SQLite anyway, because (by default, without strict mode [1] enabled) it won't stop you from putting the wrong type of data into a column.
I've worked on a system were a guy "migrated" a database from NoSQL to SQL as a proof-of-concept. Except he really didn't migrate anything. He basically created tables with "ID" (UUID) and "data" (json) columns everywhere. No indexes.
The actual "data" is a mess: different data types for the same JSON field. Imagine storing "price" as both a float and a string, depending on whatever buggy code happened to do the inserts.
It worked enough for a prototype and that was enough for management to believe the project was a success. I can't wait until we actually try and deploy it.
I think you can only judge that by knowing the context, like the domain and the experience of the designer/dev within that domain.
I looked at a DB once and thought "why are you spending effort to create these datatypes that use less storage, I'm used to just using an int and moving on."
Then I looked at the volumes of transactions they were dealing with and I understood why.
Deep down, the optimizer in me wants this to be true, but I'm having trouble seeing how this difference manifests in these days of super powerful devices and high bandwidth.
I guess I just answered my own question though. Supposing there's a system which is slow and connected with very slow connectivity and still sending lots of data around, I guess there's your answer. An embedded system on the Mars Rover or something.
I actually love your approach and haven’t thought of that before.
My problem with relational databases often stems from the fact that remodeling data types and schemas (which you often do as you build an application, whether or not you thought of a great schema beforehand) often comes with a lot of migration effort.
Pairing your approach with a „version“ field where you can check which version of a schema this rows data is saved with would actually allow you to be incredibly flexible with saving your data while also being able to be (somewhat) sure that your fields schema matches what you’re expecting.
Having to write and perform migrations for every small schema change is a bore, but it means your software doesn't have to worry about handling different versions of data. Going "schemaless" with version numbers means moving code from "write-and-forget" migrations to the main codebase, where it will live forever.
I think not doing database migrations only makes sense when you can make do without version numbers (or if you can't do atomic migrations due to performance constraints, but that's only a problem for a very small number of projects).
> Not having to write and perform migrations for every small schema change is a bore, but it means your software doesn't have to worry about handling different versions of data.
You’re correct there. I mostly work on CMSes with page builder functionality, which often bake the content schema into the database columns, which makes changing that schema (for new frontend features or reworking old ones) difficult and often prone to losing content, especially in dev environments.
Best case is obviously that you never have to version your changes, but I‘d prefer making a new schema and writing an adapter function in the codebase depending on the schemas version to spending a lot of time migrating old content. That might just be due to me not being too comfortable with SQL and databases generally.
Same here. If your entities are modelled mostly correctly you really don't have to worry about migrations that much. It's a bit of a red herring and convenient "problem" pushed by the NoSQL camp.
On a relatively neat and well modelled DB, large migrations are usually when relationships change. E.g. One to many becomes a many to many.
Really the biggest hurdle is managing the change control to ensure it aligns with you application. But that's a big problem with NoSQL DB deployments too.
At this point I don't even want to hear what kind of crazy magic and "weird default and fallback" behavior the schema less NoSQL crowd employs. My pessimistic take is they just expose the DB onto GraphQL and make it front ends problem.
>> If your entities are modelled mostly correctly you really don't have to worry about migrations that much
I'm gonna take a wild guess here that you have never worked in unfamiliar domains (like lets say deep cargo shiping or subpremium loans) where your so called subject matter experts provided by client werent the sharpest people you could hope for and actually did not understand what they where doing for most of the time?
Because I on the other hand am very familiar with such projects and doing schema overhaul third time in a row for production system is bread and butter for me.
Schemaless systems is the only reason I'm still developer and not lumberjack.
I agree. Migrations have never been a problem at my company and this codebase is 9 years old. Just regular old postgres with a few JSONB columns of things that don't need to be relational.
Same for the database that was about 8-9 years old at my last company. Migrations are fine. It's the relationship-changing that is painful.
I think they are referring to the fact that software development as a field has matured a lot and there are established practices and experienced developers all over who have been in those situations, so generally, these days, you don't see such code bases anymore.
That is how I read it.
Another possible reason you don't see those code bases anymore is the fact that such teams/companies don't have a competitive comp, so there are mostly junior devs or people who can't get a job at a more competent team that get hired in those places
"The rowid is implicit and autoassigned, but we want developer-friendly IDs." maybe
But of course, the obvious solution is to have one table with just ROWID and data, and another table with the friendly IDs! If you time the insertions really well, then the ROWIDs in both tables with match and voilà.
yeah, I've seen tables with a single row and single columns a few times, for alright reasons. Sometimes you do just want to store a single global value!
Yes, but both these have very different properties. He said (I don't know if its the case) that the db didnt have an autoincremental type. Postgres uses these sequence objects to implement autoincremental ids as he was referring to, they are implemented in-engine and are very fast and have already solved data races.
In the article, what he complains about is not about what a sequence is, but about implementing it manually with a table that is read, incremented and then saved. This us more expensive, and depending on how it was implemented you need to take care of the whole data flow so you are unable to allocate the same id twice. That's what he considers odd
The scary thing to me about that setup is how the global value is updated. Every individual script must successfully increment the value to avoid duplicate keys.
Really hope they had a “get key” stored procedure to handle that.
Sqlite only has automatically assigned row ID. You can type a column as "integer primary key" to make it an alias for the row ID, or use ROWID directly (not recommended for compatibility).
I think the intriguing part was purposefully using the same sequence value for rows in multiple tables.
I've worked with globally unique (to our application) integer keys, and per table integer sequences (which obviously aren't globally unique), but I don't recall seeing anyone use a global sequence but purposefully reuse elements of the sequence before.
If the caller/consumer isn't choosing the idempotence key then I don't think it would act as a particularly good one.
In this scenario the database is choosing the id during a transaction and so you'll only find out the value if the transaction successfully commits and the response makes it back to you.
Generally you'd want the client to provide the idempotency key such that whatever failure occurs you know the value and can provide it on a retry such that the server can use it to prevent double entry / etc
I think you missed the crucial part: every related record across all tables would have the same sequence item (the same id). That's really not normal in the database world. It sounds a lot like ECS though.
Hey former colleague - just had to say hello on a comment where you might see.
I started reading this article and everything started feeling so familiar... as soon as you told me Munch was the resident shaman, everything clicked.
My favorite factoid for others was that when I was there, we used split-horizon DNS to squat on an in-use domain name for tons of internal services, including Github. I kept wondering what would happen if the owner realized & set up his own services to catch people who weren't on the VPN.
The problem is maybe not so much the splitting and putting extra columns in a separate table. It's that you even have a table that large that it necessitates such a thing. Worst case you have a main table and a detail table that has a one to one correlation to the main entity table.
Because that means your data is highly denormalized and has plenty of duplicates. But in all likelihood it means no one knows wtf this table actually represents and you should be firing people.
I've seen this play out. Usually the many columns is because everyone misuses the table and eventually their special little business scenario or "filter" needs to be a column. Bonus points is whoever has to reference this table, they have to copy over whatever the hell your PK seems to be, and the cycle repeats, this time a bit worse.
Last place I did a brief project in had this. Queue 1000 tables spread across 25 schemas, each table having wide PKs, 20 redundant indexes on each table, and despite all this the database performs poorly. No one can tell you what each table represents, the table names are meaningless and the same data is everywhere. In order to get anything done you have to ask a small cabal of priests that knows the processes that write between these tables. After about 10 years, a partial rewrite happens and you now have 1/3rd of the data on each side with plenty of duplicate and overlap because hey.
I feel torn, I really wanna name&shame this company as a warning to all developers thinking about working there.
I feel you... I mean "...you should be firing people", this is my day to day way of thinking.
My thoughts are that, hyper-specialization, and grind breed this type of data structure. But so many companies are forced to choose, and generally tend to sacrifice on the database side of things. Then you end up with this type of unruly structure.
Database theory and practice should be a MUST on all software development courseware.
> But in all likelihood it means no one knows wtf this table actually represents and you should be firing people.
That's almost exactly the opposite of my experience, but then I've worked on smaller teams with long-term team members, so perhaps that's why.
Our tables are wide because law require we store the data for 10 years, and it's easier and faster to store things in a single row, rather than spread out over several dozen detail tables.
We don't denormalize in the sense of storing only invoice items, repeating the head data on each item. We do denormalize in the sense of storing full name and address for each party rather than storing that in a detail table. So minimal, if any, duplication of data. If we duplicate it's almost always due to the law, ie data might change in one table but shouldn't be changed in the other to preserve history.
For child tables we always have a unique ID, sequence or autoinc, and the FK to the parent, but we include the root-level ID as well. This way we can select all the child rows in a single select regardless of level, again without tons of joins.
This way our core data is just a handful of tables, and there's not much doubt about what goes where.
- there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it)
- it’s back to school again and everything goes. You can implement features in any way you want because the constraints the system imposes. Granted, sometimes it’s painful to add functionality
- there’s usually no room to subjective topics like clean code and architecture. The most important thing with these systems is correctness (and this is usually an objective topic)
- nobody can blame you for something that doesn’t work. It’s always the fault of the legacy system
I wouldn’t recommend working on such systems to junior engineers, though.
I don’t really like to work on “perfect” codebases where everyone follows the same pattern, with linters, where if something breaks is because of your shitty code (because the codebase is “clean”). It’s very frustrating and limiting.
I mean, it is kind of nice to notice that something isn't going to work because you have the linters and tests. When your developer count goes up, chances that erroneous behavior is included also goes up.
I've created proof-of-concepts that worked perfectly but would make you cry if you looked at how they worked. Eventually they became that mess. Everything is a self-contained unit so it doesn't mess anything else up. Of course, there is always time to keep adding new stuff but never to refactor it into what it should be.
I prefer the way with linters and tests, it at least lessens the chances of whatever is put in being broken (or breaking something else). (Then again, somebody putting "return true" in a test _will_ surprise you sooner or later)
> All that remained were ragtag interns and junior developers.
For many people, their first job in software engineering is the worst codebase they will deal with professionally for this reason. The first job hires lot of people with little/no experience. As soon as someone gains some experience than can move on to better paying jobs, where there are better developers with better standards.
Worst code bases are often ones taken over from IT consultancies. They drive young, inexperienced developers working many hours to deliver functionality. While the project may start out “clean” using whatever is the current hotness in technology, at some point getting stuff developed and throwing over the wall to QA is the important part.
I had dubious pleasure of working with similar codebases and devs. I'll remember one of those guys forever, because whenever he wanted to work on a new branch he would clone the repo, make changes to the master branch, and push code to a new repo numbered repo0001, repo002, ... He refused to change his ways, because "I have a PhD so you are wrong".
Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. That made porting code fun.
> Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type.
The standard does not have a boolean type. It's a postgres extension that the other open source databases adopted (because, yeah, it's obvious). But the proprietary ones insist on not having.
The official recommendation is using byte on MS SQL and char(1) on Oracle. Both are ridiculous.
In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.
Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).
Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.
This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.
Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.
It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.
This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...
MSAccess is a tragedy, imo. Right-up until, say, Access 2007, it was a simple (in a good way!) RAD platform following those zombie 4GL predecessors, but it’s been left to stagnate without any real effort to modernise it, and it’s clear from how Microsoft’s been downplaying Access that it’s a product they’d really rather not have to support, but they know if they do kill it then it would weaken on-prem Office’s moat and everyone will probably just rush to Google Firebase.
Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.
Even those before git - what they're describing sounds a bit like RCS, the precursor to CVS, which came before SVN.
I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
I recall something similar from my first job, except the shared file locking was a full on feature in Macromedia dreamweaver.
CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.
Macromedia flash / .fla files weren't particularly ideal for collaboration either, though I still feel a bit nostalgic for working with flash in general
SVN provided the ability to steal locks, and locks were opt-in so e.g. you wouldn't have made CSS need locks because it's a text file and merges fine. Mandatory locking was mostly for binary work files e.g. PSD and the like.
So they had a problem, got 2 years of approved development effort of a small team to solve it property which they did successfully, and then you resigned? After they fixed the problem?
Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.
I presume there were other reasons for resigning, or you just like massive refactoring projects.
It was a little tongue in cheek, but yes. I had large grievances with the software culture there, but after I got sign off on the project to modernise our build process, I couldn't bring myself to abandon ship in the middle of trying to fix it.
After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
Thank you for the clarification. Because you said “it took a small team … and then I resigned”, it was unclear that you were part of that small team and instead made it sound like you left because the problem was fixed.
I worked at a company recently where I couldn't get a team to do the fairly simple task of cleaning up our git ignore (so we wouldn't have to painstakingly add certain files to the index every time we changed something) so I take it as a massive accomplishment moving to git within two years.
If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.
We still use VSS for a couple codebases. We’re a small enough team that conflicts are rare, but the occasional Teams message “hey, let me know when you’re done with module.cpp” is not unheard of.
I'm so sorry. We were using it in the late 90s, and on a monthly basis it would be down for a day or two while the admin plunged out the database. Good times.
Did you know, theres a GitHub repo called [you-poor-bastard](https://github.com/SirkleZero/you-poor-bastard)? It converts a vss repo to git (not very well, but well enough), ignoring the VSS "passwords."
IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.
The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.
> IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.
IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).
I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).
I’ve worked on “check out” code bases very similar to this. I mean. Nothing so insane as scripts patching class files in to jars Willy nilly, but it seems the “just check out a file so nobody else can work on it” thing is a “common” “solution” to this.
I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)
Had a similar thing on a small team at a bigger company. We didn't have any type of version control so the team did all development on a shared dev server that we SSH'd into. We'd use vim to edit files one at a time and rely on vim's file locking to make sure that we weren't editing anything at the same time as anybody else.
Oddly, the project was one of the best I've ever worked on.
I think a lot of people who've only learned programming in the last 10 years or so don't realize that Git, and even more so the large-scale popularity of Git, is actually pretty new. A lot of programming happened before Git was created, and before it became mature and popular enough to be the default for everyone.
Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.
Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.
I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.
I use CVS, RCS and Subversion on my personal projects for several years before I learned git. I don’t remember any of them being a pain to setup for small projects and Sourceforge provided free CVS hosting for small projects.
I started with Rational Clearcase, which I interacted with via Eclipse, and I remember that you did have to lock a file to make changes to it... pretty sure there was no merge capability (but maybe I was just too junior to know stuff like that, it was definitely not something you would do as a matter of routine, like we do now with git). Someone leaving for vacation and forgetting to unlock a file could be a huge pain in the ass.
After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.
Rational Clearcase is a disaster but you could do merges.
The fun/frustrating part was getting branching and "views" correct.
I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.
I've never actually set up CVS myself, so I suppose I can't accurately say how easy or not easy it is. The "nobody" might deserve an asterisk - I'm sure it's not literally not a single person, but it seems to be pretty rare in my impression. Nevertheless, AFAIK, the non-distributed nature seems pretty limiting.
In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?
I don’t think so, it’s been a while, but I think there were tools for copying history between cvs servers. I don’t remember CVS being particularly difficult to manage.
I’ve used Clearcasse in 2006 and it had all the branches that we needed. We didn’t use locks because that was annoying. It was at Deutsche Bank and we were 200.
When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).
I once worked at a corp that sold software. For the exact same software product, maybe 10-20% of customers received so much benefit we could have charged 2-3x the price and they would still be very happy. On the other end, for maybe 10-30+%, the exact same product ranged from painful to a disaster. Often because they would try to force it to work the way their pre-existing paradigms worked instead of taking the time to learn how it actually worked.
> source control was clunky and a pain in the ass to set up
I concur. Creating a CVS repository was not harder than creating a git repository now `cvs -d $dir init`, and using, typical for the time, shared network drive, gives server experience without setting up a specialized server. It was just not included with Windows/typical Windows dev tools, so you had to know it existed, and knowledge was slower and harder to get.
> In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.
COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!
But we were using Subversion's lock system and had a proper CI/CD with Jenkins.
> It took a small team two years to migrate the entire codebase to git with proper CI
I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.
Would be great to work in such a company as a Linux guru.
There are so many entangled services and machines that you feel like an Indiana Jones. You ssh into a machine and feel century-old dust beneath your footsteps. And you never know what will you find. Maybe a service which holds the company together. Maybe a CPU eating hog which didn't do anything useful last 3 years.
I don't enjoy writing new code much. But in such an environment even with my limited skills I can do decent improvements, especially from security point of view. Feels great
Well it sounded like there are exactly 0 Linux machines running there.. it's all windows .net / c# and a bunch of native windows apps, as I understood the article.
But maybe you can replace your statement with "windows guru" and SSH with "remote desktop" and perhaps that would be fun
well, with linux at least I get get the source code of the kernel and of the MySQL when given a database server which hasn't been restarted for 7 years
When I started my recent job the team kept referring to a box running "weird linux".
After getting on-boarded and accessing the box it turned out to be running a very old version of OpenBSD.
To this day I'm curious who had the wherewithall to install OpenBSD in prod but was seemingly ignorant of the 1yr support cycle.
I started reading and was hoping someone had installed one of the Linux distros that were popular at the turn of the millennium, like Mandrake, Slackware, etc. and it was still trucking along 25 years later.
In 2010 or 2011, deviantArt was running on several hundred slackware servers. One of my projects at that job was converting everything to run on Debian with config management in Puppet to replace all of the old manually configured services.
Thats a name I haven't heard for a while - I was obsessed with scouring deviantart for cool wallpapers during my very early Linux days of desktop tinkering (mostly fluxbox and gkrellm tweaking).
Oldest boxes running are RHEL5 and Win Server 2008.
I was trying to explain to the IT Director that both were released when I was in elementary school so I'd have to brush up on them.
I'm guilty of something like that. At my very first job I used OpenBSD to manage the corporate firewall and proxy. 6 months later quit to attend college, and I have no idea what they did to that box lol.
Sounds like me, I left a number of openbsd machines at a previous shop I worked at. I kept them updated, but based on the bewildering mishmash of linux distros and versions, there were a surprising number of sco boxes hanging around as well, their general philosophy was to set a box up for a task then never update it afterwards. So I expect all my obsd boxes are still there at exactly the same version I left them at.
Back about 15 years ago, I worked for a web hosting company that provided some sysadmin consultation services. Customer paid us, and I would take a look.
I had one customer who came back with the same request, slightly differently worded, every single month, and every single month I'd say the same thing. They had this site they were running that was essentially a Yellow Pages type site. They had a large set of companies with contact details, each with multiple business categories associated with it. You'd choose a category, and they'd return a list of matching companies.
The problem was the site was really slow. I took a quick look around, and saw that all the time was lost querying the database. Taking a quick look at the schema I discovered that their approach to categorisation was to have a TEXT column, with semicolon separated 4 character strings in it. Each 4 character string mapped to a business category.
So when someone wanted to load up, say, all pest control companies, it would check the category mapping table, get the 4 character string, and then go to the companies table and do:
SELECT * FROM companies WHERE categories LIKE "%PEST%"
So on each page load of the main page type the site was there to provide, it did a full text search over the category field for every single record in the company table.
I guess that's probably okay for the developer without real world scale data, and real world traffic counts to worry about. But they had lots of data in the database, and that category field could have dozens of categories against a company. As soon as they had more than about 4-5 simultaneous customers performance started tanking.
I could never get them to accept that they needed to rethink the database schema. One month they were bleating about how is it possible that Google can manage to do such a search across a much larger amount of data, much faster. They really didn't like my answer that amounted to "By having a sane database schema". All they were willing to do was pay over the odds for our most powerful server at the time, which had enough capacity to hold the entire database in memory.
I stumbled across that comment a few years back and it changed the way I handle tags and categories so just sharing it here. If anyone has an equivalent for Sqlite, I’d love to hear it!
That's what my suggestion came down to. A two column table with company id and category id. They had ids already. They could index off category and get the results in split seconds
Probably some of the worst code I ever worked on was a 12k+ line single file Perl script for dealing with Human Genome Project data, at Bristol-Myers Squibb, in the late 1990s.
The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo.
By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.
As an outsider of any reasonably large application can you really ever expect to grasp 200 random lines of code in any language?
Maintenance programming is all about understanding the software‘s context and implicit design or a model thereof — even if it is a 20 year old patch work.
Young developers tend to be amazed when I find the source of an obscure bug in our 1M lines of Perl code application as a senior. But the thing is, I just happen to „get the application“ and developed intuition about its inner working. It’s rarely of knowing Perl particularly well. The thing could have been written in TCL or even Brainfuck — with a working model of your software in your head (not with the memory of the total code base, mind you) you will eventually find the problem in a piece of software written in any language.
If you get a typically-structured Java/Spring application then often yes. By typically structured I mean domain objects, DTOs, service layer, controller layer, view layer. The idiomatic use of dependency injection frees the mind of the reader from understanding all the dependencies in code, just use an @Autowired class you want and someone part has the knowledge to configure it and does so.
We should use the Antarctic highlands as a prison colony for people found guilty of writing Stringly Typed Code. Siberia isn’t awful enough for them. I thought people who stuffed multiple independent values into a single database column were the worst and then I saw what people can accomplish without even touching a database.
Ha - we did that too at BMS. We were paying Oracle by the column or something like that, so people would shove entire CSV rows into a single value (because corporate said everything HAD to be in Oracle) and then parse them application-side.
We got rid of most of the DBAs because they got overfond of saying 'no' and they became a major source of friction for doing reasonable architecture. People started figuring out ways to work around the DBAs like stuffing more shit into existing columns or adding secondary tables that established new relationships between the sacrosanct tables.
A lot of the worst sins happened on Oracle databases, mostly because they were the most popular at the time, and they were the only real DB cult, so the cottage industry of specializations that told other developers not to worry their pretty little heads was especially bad.
I have to work with a DBA who has decided that nothing new gets developed using Postgres and is confident that our use cases are best served with a document db… all without knowing any of our use cases, requirements or constraints.
Now I just don’t involve him in anything unless forced.
A friend once picked up a tiny contract from Uncle Sam, requiring some changes to a Perl script that stashed values as "this/that/thenext" in one column of an Oracle table. She had not previously dealt with Perl, so I explained to her about "split" and "join".
I don't know why the developers had chosen that method.
No we should reserve the antarctic highlands for type system fetischists who write abstract protocol adaptor factory repositories for every damn simple piece of functionality.
That sounds more like an OOP thing than a type system thing - functional languages make excellent use of type systems without mention of anything remotely sounding like a “Design Pattern (tm)”
People balk at the idea of licensure, but the bar is so low that all you have to do is ask "what is an array?", and you'll filter out vast swaths of people like this.
Admittedly I've somehow only worked in perl but the worst code I tried for fix felt similar. They know about arrays but every map and grep used perks default $_ and there was enough nesting the function was near 1k lines if I remember right.
If I were being charitable (sometimes I try), I'd guess he was concerned that an array of scalars is going to have a high per scalar size overhead and perhaps worse cache locality, whereas the non-unicode perl string implementation at the time probably just stored the string as a large char* with a length prefix.
In a similar vein, an alternate explanation may just be that a lot of bioinformatics algorithms are described as string algorithms. It would be a pity if that was just taken too literally, but I've seen a lot of bioinformatics perl that works on strings in this way.
While trying to be charitable though, it's almost impossible to come up with a rationalization for the three character offsets string. Sometimes I think when people really go wrong is when they hear of some problem or are trying to be too clever and end up going beyond their abilities and stray into farce.
Being wrong with malicious intent and being wrong because you have an interstate-number IQ are indistinguishable; the outcome is still terrible. I don't care who's playing 4D chess, the pieces are still all over the floor and I have to clean them up.
I am somewhat surprised that a programmer who was unaware of arrays in Perl managed to have tests. But then again, he managed to implement their own version of arrays, maybe he came up with the concept of software testing by himself :-P
It sounds more like the refactoring started with creating tests for the old code base. We do this with our legacy code base, too, sometimes. Of course, it’s always the high risk code of an application that‘s never been accompanied by automatic tests.
My first day at my first full-time programming gig, I was asked to look at some reporting job that had been failing to run for months. I logged in, found the error logs, found that it needed a bit more memory assigned to the script (just a tweak in the php.ini) and let the team lead know it should run fine that night. He was shocked, "Dude, if you just fixed that report you probably just got a promotion, no one has been able to figure that out for months." He was joking about the promotion, but my boss was just as shocked. I'd realize later that most the other people on the dev team didn't like linux and wanted to rewrite everything in .NET and move everything to Windows so no one even tried with anything related to any of the linux machines.
I know things have gotten somewhat better, but the amount of wasted time and latency of using RDP and Windows UI for development, testing and production maintenance is insane. Throw in some security requirements of RDP into host 1 to the RDP jump to host 2 and companies are just wasting money on latency. There is, often, not an appreciation of the administrative costs of the delivery. Not necessarily system admin costs, but developer and QA time associated with delivering and ongoing maintenance.
Oh yeah, been there. I personally find Windows to be a terrible server OS. I'd rather be stuck with a fleet of AIX machines or something equally uncommon (but still nix-ish) than any Windows system.
>Throw in some security requirements of RDP into host 1 to the RDP jump to host 2 and companies are just wasting money on latency.
So much this at my job, login to my laptop, login to vpn using password and 2-factor, check out my admin credentials using my login and 2-factor, login to the jump box using my admin creds and a different 2-factor, finally login to the system I need to be on using my admin creds. Multiply the last step by however many systems I need to connect to. Also, the clipboard and screen resolution are going to get messed up along the way no matter how much you mess with the settings.
The reverse also applies, where people that like Linux just totally ignore Windows maintenance. They’ll look down their noses at it and say that it’s insecure when they spend their time patching Linux and leaving Windows to fend for itself against attackers.
Someone must have made that "smart" decision to use servers with a proprietary OS, so that someone surely has paid the extra cost for MS support. No need to worry then.
I am not talking about Windows updates here. But even if I was, there are things wrong with it. One example, that frequently bites me on my gaming OS:
Windows 10 by itself decides: "Hey, I'm going to replace your installed graphics card driver with some other version, that I will find myself online!" Then next time I start this OS, I get crashing graphics card driver and need to go and look on AMD website for the correct drivers, installing them over whatever crap Windows installed. This has happened at least 3 times already and it is pissing me off. It would probably be better to deactivate automatic Windows updates completely and then try to have a manual selection of installed components. But even then I could not be sure, that it will not try to replace my installed drivers with crap.
I think AMD has that too in its app, but if your graphics card driver is crashing in windows, your screen is freezing. Windows might try to restart the driver or so, getting the screen working briefly, but then it crashes again and ultimately it fails completely, you can only press the power button on your machine then.
sure it might be a mess, but at least it's purpose built. I love that kind of performance gains. Honestly most companies die before purpose built code like that becomes a problem.
Jira's now discontinued server version had a sequence table to stop you sharding it. It also made disaster recovery from a hard shutdown awful. I have nothing good to say about Atlassian.
Atlassian, and JIRA specifically, are responsible for so much wasted time and capital expenditure. If I could get ahold of metrics like "hours spent building, using, and maintaining JIRA" versus "value obtained from JIRA" for each of the companies I've worked at, I'm pretty sure I could generate a report so scathing that nobody would ever use it again.
Gatekeeping your work organization system from the people working on, and often MANAGING it is such a huge friction point I'm amazed they ever got clientele. I'm an Admin on a project right now and I can't change our status types without going through our assigned Atlassian rep.
And don't even get me started on the dumpster fire that is BitBucket. Ever tried to use the API? it's somehow even more worthless than the UI.
> Now the story I heard at the time was that once upon a time SQL Server didn't support auto-incrementing ids. This was the accepted, correct answer.
At a company that I used to work, they heard the same rumor, so instead of using identity columns or sequences, they kept a table with a number of ids "available" (one row per id).
Whenever unique id was needed, the table would be locked, an id selected and marked as used. If there were no ids available, more ids would be added and then one used.
A scheduled job would remove ids marked as used from time to time. Note that there was a single "sequence table", that was shared among all of the entities.
That was not even the weirdest part. That id was unique, but NOT the primary key of the entity, only part of it.
The structure of the database was fairly hierarchical, so you had for example a table CUSTOMER in 1-to-many relation with a USER table, with a 1-to-many relation with an ADDRESS table.
while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
> while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
Maybe they wanted to avoid maintaining a separate CUSTOMER_ADDRESS 1-to-many table or maybe it was done to make easy reverse lookups.
A couple of weeks ago I had a flat on the way to the airport. It was a total blowout, and my car doesn't include a spare. We were already over budget on our trip, so I had the car towed to the closest tire shop and had them put on the cheapest tire that could get me to the airport. I know I'll need to replace other tires, as it's an AWD, and I know it's not a tire I really want. I made a calculated choice to make that a problem for future me due to the time crunch I was under.
Supposedly there are reasons (for example, efficiency by avoiding the extra weight) and they give you a kit that's essentially a "fix a flat" product, but in the 2 times I've had a flat in a car outfitted like this, what they provided wasn't useful, usually due to the type of damage to the tire.
it sucks, yes, but when coined, the term referred to a platform extracting value from its users by degrading their experience, forcing them to upgrade to a premium plan. your car not having a spare isn't an example of that!
Many of us grew up with cars having spare tyres as a matter of course. Now car manufacturers are extracting value from their customers by degrading our experience (of safety and expedience of repair), forcing us to pay extra for a spare tyre (if even available), or even upgrade to a better car.
Ordinary cars not having a spare tyre as standard equipment any more is a perfect example of enshittification.
A lot of "why TF would anyone you do this?" questions can be answered with variations like this. Borrowing from economics, the discount rate defines how much more valuable something is RIGHT NOW vs at some point in the future. If the system is down and we're losing money by the minute until it is up; if the company will be broke next week unless we make customer X happy; if you have to get to the airport in 2 hours - it is completely rational to make the quick fix. The costs of not doing so outweigh the costs of doing it again properly.
It really becomes a problem when the same short term calculation becomes the standard - you can quickly rack up costs that future you or future company will never pay down.
I worked at a startup where we had to make a lot of compromises. The (non technical) founders could not get enough funding at the beginning, so they had to constantly raise small amounts to survive the next few months.
We had to grow fast and prove that we had a market fit. This meant making compromises, but the technical debt was actually under control. We knew what to improve and how to do it once we had enough resources.
Eventually we proved ourselves and got decent funding and could hire more developers. We were looking forward to improving our system.
Unfortunately the new guys (many fresh out of college, lead by a few "FAANG"-ish seniors) decided that the technical debt was a proof of our incompetence and convinced the leadership that they needed to take over the technical development and rewrite everything. The result was a year where no major features were added and we lost our momentum. The funding dried up and the company had to let go 2/3 of the employees.
The worst part was that the new systems were more complicated to extend due to their clever "scalable design" (distributed monolith) with more micro services than developers. When the rewrite was done and it was time to add the new features, they had lost the trust of the leadership and just threw code at it in desperation to get things done as quickly as possible. I pithy the developers that inherited the result... By that time the rock stars had moved on and are inflicting damage elsewhere after touring conferences where they presented their greatness.
One of the most important skills a person can have is to assess a situation, understand the actual context and constraints and build something based on this information. It sounds to me your team worked this way. You were playing within the given constraints.
Exactly the opposite of the one trick pony’s who just mindlessly apply what they learned or think is cool with a total disregard for context.
I'm glad OP was able to maintain a good sense of humor about it. Such discoveries as the nested classes of empty methods have sent me into teeth-gnashing fury followed by darkness and despair. One such experience is why I would rather change careers if my only option were to build on the Salesforce platform, for instance.
Early on in my career, at a large company, I encountered someone who took “codebase” a little too literally. At the time every department had their own developers, sometimes just employees who had an aptitude for computers.
This one guy established himself by making an Access database for their core business, and when the web became a thing, built a customer site. But not on it—in it. He simply served ASP pages directly from the database, inserting dynamic content in queries. When I was asked to help improve their terrible design, I was forced to untangle that unholy mess of queries, ASP (new to me) and HTML. It was easiest to write all the HTML and insert their ASP right before I sent the files back (because I wasn’t given access to their DB/web server). Thinking “I could do better than this” got me into programming.
He was a Microsoft-everything head. Finally went too far when he presented a new web interface starring a Clippy-like parrot using Microsoft’s DirectX avatar API. The executives were unimpressed and then I noted that 20% of our customers couldn’t use his site. (I probably still had a “best viewed with IE” badge on the main site, lol)
This codebase sounds like a haunted graveyard[1], where everyone just fixes their local corner of things and avoid the risk of untangling the existing mess.
Not needing to conform to some company-wide standard is probably really pleasant while it lasted, but every such effort adds to the haunted graveyard, and the lack of consistency will eventually come back to bite whoever is still around.
Once not long enough, I'd worked on 4 projects which was literally copied from the first made and changed parts of the customers and internal users according to each use of it. So, the main problem is bugs found in one project was found on the other 3 and I'd to fix the same bug!
Codebases like this or from the OP is cool to learn how to not do certain things.
That is why people these days tend to use a single JSON blob instead of multiple columns. And because it is so popular, SQLITE and other DBs are building better and better JSON support into the DB.
I wonder if better support of EAV tables would solve this issue better.
If one could do "SELECT price,color,year FROM cars! WHERE status='sold'" and the "!" would indicate that cars is an EAV table ...
entity attribute value
1 price 20750
1 color red
1 year 2010
1 status sold
... and the result of the query would be ...
20750 red 2010
That would solve most of the use cases where I have seen people use JSON instead.
587 comments
[ 3.9 ms ] story [ 355 ms ] threadAlso, be aware of, but don't get hung up on, engineering best practices. Chances are that someone is going to see your POC and put it in front of a client or promise it to someone. It will enter service being less than perfect, if you are lucky.
When you pull this off people will tell stories about you and brag about you when you are not around. None of them will know anything about the code engineering.
True!
> None of them will know anything about the code engineering.
Not true. In environments where it's very easy to fuck up (hyperconnected codebases, lots of legacy tech debt, fragile systems generally), programmers who create tools/libraries that let other engineers get work done without blowing their feet off or spending huge amounts of time on silly things do get praised. Maybe the praise comes from different people than in your example, but it definitely is rewarded.
Two caveats: this only happens in certain environments; some gigs are just culturally unable to care about quality tools (death march feature factories and the like). Also, you actually have to save time and reduce defect rates with your engineering work; starting an architecture-astronaut rewrite that isn't actually useful for other programmers' day-to-day work doesn't count. Making "one tool to rule them all" that requires a Ph. D or pairing session with the author doesn't count either.
Might help to have enough experience that you have (well-placed) confidence in your gut feel for how much engineering a particular thing needs.
(More common is to not have an engineer mode, or to not have enough experience to do it well.)
If you don't have the kludge mode, try some things where kludges are outright necessary. One time I recall this was when writing Emacs extensions. For example, it used to be that you'd hit a point where normal programmatic navigation/parsing of the buffer was just too slow, and a well-placed regexp operation that worked 99.99% of the time (and the other 0.01% of the time it's OK enough, and the user understands) made the rest of the code viable.
Another example is personal open source projects, when you really want to experiment with an usual implementation approach, and you give yourself permission.
(Maybe don't do this when the open source is done only for resume purposes, where you're trying to demonstrate you follow all the current fashionable conventions. You're almost guaranteed that someday a new-grad on a public forum or not-very-technical hiring manager will stumble across that bit of code, and fixate on the one thing they think they recognize as a bad practice. Documenting the kludge and rationale, in, say, a code comment, is great practice, but, again, it will also draw the attention of the least-skilled. Much like, if you're relating an anecdote in a job interview, and implied are 20 things you did right and 1 thing you did brilliantly, and you note as an aside one mistake you made, most of that will whoosh over the head of the dimmest FAANG interviewer, but they'll latch onto that mistake you spelled out, as a wise insight they have, and that's what's going in their report. :)
Then, armed with an experienced multi-skilled brain, when your startup's imminent MVP launch is facing crazy constraints, you triage what bits need to be rock-solid so they'll absolutely work and not fail, what bits need creative kludging so you can hit your launch window for other key requirements, and what bits you need to mitigate any compromises. Ideally, you have the wisdom to know which is which, and can activate different skills for each kind of work.
That Turbo Encabulator needs to be well-machined, but the new sensor you just decided it needs for a test doesn't need a week to be drilled into the engine block, nor a mount to be designed and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or maybe wad of chewing gum will do.
P.S. you opened a parenthesis and forgot to close it so now everything I read all day is part of your anecdote)
I have a zsh alias `alias changes='git add . && git commit -am "Changes" && git push'` that I use for my commits in the repo.
This all may feel silly, but it's what I needed to get back to that time of playing. Where I never worried about the code. But where I also didn't feel I was wasting my time working on code no one could see. I'd definitely recommend trying something like that if you are struggling with it.
[1]: https://github.com/jimmyhmiller/PlayGround
i considered a few times to make an alias like this but i feel like git is so precious to get right that i would rather type out each command just to "feel" what i'm doing and be active in it.
There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated?
But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do.
Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff.
These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...
Internet Archive has a crawl from today but no earlier; which doesn't mean it can't be earlier of course. My guess is it was written recently though.
Have you had the pleasure of blowing young minds by revealing that production-grade databases come with fully fledged authnz systems that you can just...use right out of the box?
https://www.postgresql.org/docs/current/ddl-rowsecurity.html
Those features aren't used often in modern app development where one app owns the database and any external access is routed through an API. They were much more commonly used in old school apps enterprise apps where many different teams and apps would all directly access a single db.
Row level security has been wonderful for multi tenancy in my experience though. I would highly recommend it.
https://supabase.com/docs/guides/database/postgres/row-level...
However I would still advise people to use a third normal form - they help you, constraints help you, and often other sets of tooling have poor support for constraints on JSON. Scanning and updating every value because you need to update some subset sucks.
You first point is super valid though - understanding the domain is very useful and you can get easily 10x the performance by designing with proper types involved, but importantly don't just build out the model before devs and customers have a use for anything, this is a classic mistake in my eyes (and then skipping cleanup when that is basically unused.)
If you want to figure out your data model in depth beforehand there's nothing wrong with that... but you will still make tons of mistakes mistakes, lack of planning will require last minute fixes, and the evolution of the product will have your original planning gather dust.
Mirrors my experience exactly. Querying json can get complex to get info from the db. SQLite is kind of forgiving because sequences of queries (I mean query, modify in appliation code that fully supports json ie js, then query again) are less painful meaning it's less moprtant to do everytning in the database for performance reasons. But if you're trying to do everything in 1 query, I think you pay for it at application-writing time over and over.
Really!? Are you building applications by chance or something else? Are you doing raw sql mostly or an ORM/ORM-like library? This surprises me because my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks. SQLite's fluid type system haa been a nice middle ground for me personally. For reference my application layer is kysely/typescript.
Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
Practical example: I recently wrote my own SMTP server (bad idea!), mostly to be able to control spam (even worse idea! don't listen to me!). Initially, I thought I would be really interested in remote IPs, reverse DNS domains, and whatever was claimed in the (E)HLO.
So, I designed my initial database around those concepts. Turns out, after like half a million session records: I'm much more interested in things like the Azure tenant ID, the Google 'groups' ID, the HTML body tag fingerprint, and other data points.
Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent, but fortunately modern C# makes that easy as well.
And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
I do the same thing with serde_json in Rust for a desktop app sqlitedb and it works great so +1 on that technique.
In Rust you can also tell serde to ignore unknown fields and use individual view structs to deserialize part of the JSON instead of the whole thing and use string references to make it zero copy.
That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Also I don't understand why you're afraid of migrations, especially since you're the only developer on your own SMTP server.
Did you miss that he’s using sqlite? The dev experience with a sqlitedb is way better than running yet another service, especially for personal projects.
Sqlite is used just as much as an application file format as it is a relational database.
My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""
So, care to take another guess? And, while we're here, does MongoDB run fully in-process these days? And/or allow easy pagination by ROWID?
Which feels off by six generations or so of memes. It feels prior to "memes" existing in the first place (at least this modern iteration, pics with captions in them; not Dawkins' original verion). I'd guess it is, ironically, chronologically closer to, well, your username here.
What gain would MongoDB offer here?
You certainly would lose a lot of things, like a well supported path to linking with to the database engine, and a straightforward way to start to introduce relational tables as the project matures. Nothing completely insurmountable, of course, but carry a lot of extra effort for what benefit?
Our solution for a similar situation involving semi-structured data (in postgres) was to double it up: put all the json we send/receive with a vendor into a json field, then anything we actually need to work on gets extracted into regular table/columns. We get all the safety/performance guarantees the database would normally give us, plus historical data for debugging or to extract into a new column if we now need it. The one thing we had to monitor in code reviews was to never use the json field directly for functionality.
If some vendor is giving you a list of categories you don't care about, there's no need to make a vendor categories table and a many-to-many link table until you actually need them.
The point is that putting data properly in the database lets you use database features on it and get database performance.
Recently I’ve been tempted to make an SMTP server that translates emails into a web-hook. Please tell me more horror stories so that I might be convinced not do it.
> And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
That's a migration.
> Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent
That's an effect of not migrating - having to process null and absent fields instead of just null fields. After doing more of these, you'll run into the same thing that made people stop using NoSQL databases: with no schema, your code has to parse all previous versions of the data format and they probably aren't even well-documented. While an RDBMS can just set the new column to null in existing rows.
For the data schema, we're using Protobufs with buf validate. This works surprisingly well, you can use the same types in the backend API and on the frontend. We even have a cron job that reads all the data periodically and verifies that the JSON blobs conform to the schema. Our next intern is going to write a PostgreSQL extension to do that on commit/update :)
One real advantage of this approach is that it's easy to do stuff like "search everywhere".
Classic fine-grained schemas are not that much different. A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.
In addition, our frontend apps also support offline mode. They can get all the relevant objects, and then operate on them locally. So our API was designed from the start to deal with conflicts.
All well and good but you do need to handle failures elegantly in code. The nice thing about flat DB tables and SQL is you don't really have to care if your goal is to update a single column authoritatively. The state of the other values in the table are, often, immaterial. It gets even more complicated reconciling deeply nested conflicts in open schemas.
Not knocking your approach, it's just a trade-off I guess.
Perhaps that is true... I'm a Rails dev and ActiveRecord definitely tracks mutations on the object and selectively updates.
I would assume this is consistent with Django and I'd assume a policy is probably required to be specified by SQLAlchemy.
An example would be updating Kubernetes resources, the admission controller will verify the correctness of the change you are trying to make and reject the patch if it's non-comformant. Nested values have... value... in terms of contextualizing and isolating/localizing data leaves but at the end of the day aren't you still dealing with strict schemas and, when those schemas change you have to reconcile schema migrations?
Certainly. But you have to do that with SQLite anyway, because (by default, without strict mode [1] enabled) it won't stop you from putting the wrong type of data into a column.
[1]: https://www.sqlite.org/stricttables.html
The actual "data" is a mess: different data types for the same JSON field. Imagine storing "price" as both a float and a string, depending on whatever buggy code happened to do the inserts.
It worked enough for a prototype and that was enough for management to believe the project was a success. I can't wait until we actually try and deploy it.
I think you can only judge that by knowing the context, like the domain and the experience of the designer/dev within that domain.
I looked at a DB once and thought "why are you spending effort to create these datatypes that use less storage, I'm used to just using an int and moving on."
Then I looked at the volumes of transactions they were dealing with and I understood why.
I guess I just answered my own question though. Supposing there's a system which is slow and connected with very slow connectivity and still sending lots of data around, I guess there's your answer. An embedded system on the Mars Rover or something.
Pairing your approach with a „version“ field where you can check which version of a schema this rows data is saved with would actually allow you to be incredibly flexible with saving your data while also being able to be (somewhat) sure that your fields schema matches what you’re expecting.
I think not doing database migrations only makes sense when you can make do without version numbers (or if you can't do atomic migrations due to performance constraints, but that's only a problem for a very small number of projects).
Is that "not" at the front supposed to be there?
This is not my experience, it only happens rarely. I’d like to see an analysis of what causes schema changes that require nontrivial migrations.
On a relatively neat and well modelled DB, large migrations are usually when relationships change. E.g. One to many becomes a many to many.
Really the biggest hurdle is managing the change control to ensure it aligns with you application. But that's a big problem with NoSQL DB deployments too.
At this point I don't even want to hear what kind of crazy magic and "weird default and fallback" behavior the schema less NoSQL crowd employs. My pessimistic take is they just expose the DB onto GraphQL and make it front ends problem.
I'm gonna take a wild guess here that you have never worked in unfamiliar domains (like lets say deep cargo shiping or subpremium loans) where your so called subject matter experts provided by client werent the sharpest people you could hope for and actually did not understand what they where doing for most of the time?
Because I on the other hand am very familiar with such projects and doing schema overhaul third time in a row for production system is bread and butter for me.
Schemaless systems is the only reason I'm still developer and not lumberjack.
Same for the database that was about 8-9 years old at my last company. Migrations are fine. It's the relationship-changing that is painful.
But I do try and get a schema that fits the business if I can.
That is how I read it.
Another possible reason you don't see those code bases anymore is the fact that such teams/companies don't have a competitive comp, so there are mostly junior devs or people who can't get a job at a more competent team that get hired in those places
But of course, the obvious solution is to have one table with just ROWID and data, and another table with the friendly IDs! If you time the insertions really well, then the ROWIDs in both tables with match and voilà.
https://stackoverflow.com/questions/41928749/pros-cons-of-re...
https://www.postgresql.org/docs/current/sql-createsequence.h...
In the article, what he complains about is not about what a sequence is, but about implementing it manually with a table that is read, incremented and then saved. This us more expensive, and depending on how it was implemented you need to take care of the whole data flow so you are unable to allocate the same id twice. That's what he considers odd
Really hope they had a “get key” stored procedure to handle that.
In reality, that wasn't too unusual to see because frameworks would use that technique because it's a lowest common denominator across RDMS.
I've worked with globally unique (to our application) integer keys, and per table integer sequences (which obviously aren't globally unique), but I don't recall seeing anyone use a global sequence but purposefully reuse elements of the sequence before.
In this scenario the database is choosing the id during a transaction and so you'll only find out the value if the transaction successfully commits and the response makes it back to you.
Generally you'd want the client to provide the idempotency key such that whatever failure occurs you know the value and can provide it on a retry such that the server can use it to prevent double entry / etc
How do you pronounce that?
Was it like the word munch in “munching on some snacks”?
Or like the name of the painter Edward Munch? https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 (note: this link is in Norwegian)
My favorite factoid for others was that when I was there, we used split-horizon DNS to squat on an in-use domain name for tons of internal services, including Github. I kept wondering what would happen if the owner realized & set up his own services to catch people who weren't on the VPN.
I've seen this play out. Usually the many columns is because everyone misuses the table and eventually their special little business scenario or "filter" needs to be a column. Bonus points is whoever has to reference this table, they have to copy over whatever the hell your PK seems to be, and the cycle repeats, this time a bit worse.
Last place I did a brief project in had this. Queue 1000 tables spread across 25 schemas, each table having wide PKs, 20 redundant indexes on each table, and despite all this the database performs poorly. No one can tell you what each table represents, the table names are meaningless and the same data is everywhere. In order to get anything done you have to ask a small cabal of priests that knows the processes that write between these tables. After about 10 years, a partial rewrite happens and you now have 1/3rd of the data on each side with plenty of duplicate and overlap because hey.
I feel torn, I really wanna name&shame this company as a warning to all developers thinking about working there.
My thoughts are that, hyper-specialization, and grind breed this type of data structure. But so many companies are forced to choose, and generally tend to sacrifice on the database side of things. Then you end up with this type of unruly structure.
Database theory and practice should be a MUST on all software development courseware.
That's almost exactly the opposite of my experience, but then I've worked on smaller teams with long-term team members, so perhaps that's why.
Our tables are wide because law require we store the data for 10 years, and it's easier and faster to store things in a single row, rather than spread out over several dozen detail tables.
We don't denormalize in the sense of storing only invoice items, repeating the head data on each item. We do denormalize in the sense of storing full name and address for each party rather than storing that in a detail table. So minimal, if any, duplication of data. If we duplicate it's almost always due to the law, ie data might change in one table but shouldn't be changed in the other to preserve history.
For child tables we always have a unique ID, sequence or autoinc, and the FK to the parent, but we include the root-level ID as well. This way we can select all the child rows in a single select regardless of level, again without tons of joins.
This way our core data is just a handful of tables, and there's not much doubt about what goes where.
- there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it)
- it’s back to school again and everything goes. You can implement features in any way you want because the constraints the system imposes. Granted, sometimes it’s painful to add functionality
- there’s usually no room to subjective topics like clean code and architecture. The most important thing with these systems is correctness (and this is usually an objective topic)
- nobody can blame you for something that doesn’t work. It’s always the fault of the legacy system
I wouldn’t recommend working on such systems to junior engineers, though.
I don’t really like to work on “perfect” codebases where everyone follows the same pattern, with linters, where if something breaks is because of your shitty code (because the codebase is “clean”). It’s very frustrating and limiting.
I've created proof-of-concepts that worked perfectly but would make you cry if you looked at how they worked. Eventually they became that mess. Everything is a self-contained unit so it doesn't mess anything else up. Of course, there is always time to keep adding new stuff but never to refactor it into what it should be.
I prefer the way with linters and tests, it at least lessens the chances of whatever is put in being broken (or breaking something else). (Then again, somebody putting "return true" in a test _will_ surprise you sooner or later)
And there is nothing worse than a crappy codebase the company won't let you improve.
For many people, their first job in software engineering is the worst codebase they will deal with professionally for this reason. The first job hires lot of people with little/no experience. As soon as someone gains some experience than can move on to better paying jobs, where there are better developers with better standards.
Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. That made porting code fun.
The standard does not have a boolean type. It's a postgres extension that the other open source databases adopted (because, yeah, it's obvious). But the proprietary ones insist on not having.
The official recommendation is using byte on MS SQL and char(1) on Oracle. Both are ridiculous.
In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.
Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).
Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.
This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.
Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.
It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.
Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.
I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe
Tools do not fix bad design.
CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.
Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.
I presume there were other reasons for resigning, or you just like massive refactoring projects.
After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.
There are few efforts as truly thankless in many organizations as fixing things.
The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.
IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).
I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).
I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)
Oddly, the project was one of the best I've ever worked on.
Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.
Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.
I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.
[0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.
The fun/frustrating part was getting branching and "views" correct.
I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.
In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?
When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).
I concur. Creating a CVS repository was not harder than creating a git repository now `cvs -d $dir init`, and using, typical for the time, shared network drive, gives server experience without setting up a specialized server. It was just not included with Windows/typical Windows dev tools, so you had to know it existed, and knowledge was slower and harder to get.
> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.
COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!
But we were using Subversion's lock system and had a proper CI/CD with Jenkins.
So you won, your project was worse!
I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.
The Rube-Goldberg VCS-mutex is hilarious though.
There are so many entangled services and machines that you feel like an Indiana Jones. You ssh into a machine and feel century-old dust beneath your footsteps. And you never know what will you find. Maybe a service which holds the company together. Maybe a CPU eating hog which didn't do anything useful last 3 years.
I don't enjoy writing new code much. But in such an environment even with my limited skills I can do decent improvements, especially from security point of view. Feels great
But maybe you can replace your statement with "windows guru" and SSH with "remote desktop" and perhaps that would be fun
"Generation Z", I think.
I had one customer who came back with the same request, slightly differently worded, every single month, and every single month I'd say the same thing. They had this site they were running that was essentially a Yellow Pages type site. They had a large set of companies with contact details, each with multiple business categories associated with it. You'd choose a category, and they'd return a list of matching companies.
The problem was the site was really slow. I took a quick look around, and saw that all the time was lost querying the database. Taking a quick look at the schema I discovered that their approach to categorisation was to have a TEXT column, with semicolon separated 4 character strings in it. Each 4 character string mapped to a business category.
So when someone wanted to load up, say, all pest control companies, it would check the category mapping table, get the 4 character string, and then go to the companies table and do:
So on each page load of the main page type the site was there to provide, it did a full text search over the category field for every single record in the company table.I guess that's probably okay for the developer without real world scale data, and real world traffic counts to worry about. But they had lots of data in the database, and that category field could have dozens of categories against a company. As soon as they had more than about 4-5 simultaneous customers performance started tanking.
I could never get them to accept that they needed to rethink the database schema. One month they were bleating about how is it possible that Google can manage to do such a search across a much larger amount of data, much faster. They really didn't like my answer that amounted to "By having a sane database schema". All they were willing to do was pay over the odds for our most powerful server at the time, which had enough capacity to hold the entire database in memory.
I stumbled across that comment a few years back and it changed the way I handle tags and categories so just sharing it here. If anyone has an equivalent for Sqlite, I’d love to hear it!
The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo.
By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.
Maintenance programming is all about understanding the software‘s context and implicit design or a model thereof — even if it is a 20 year old patch work.
Young developers tend to be amazed when I find the source of an obscure bug in our 1M lines of Perl code application as a senior. But the thing is, I just happen to „get the application“ and developed intuition about its inner working. It’s rarely of knowing Perl particularly well. The thing could have been written in TCL or even Brainfuck — with a working model of your software in your head (not with the memory of the total code base, mind you) you will eventually find the problem in a piece of software written in any language.
So you invented a “no-sql” document database in Oracle.
A lot of the worst sins happened on Oracle databases, mostly because they were the most popular at the time, and they were the only real DB cult, so the cottage industry of specializations that told other developers not to worry their pretty little heads was especially bad.
Now I just don’t involve him in anything unless forced.
I don't know why the developers had chosen that method.
All his sins can be forgiven because he made it so easy to refactor by writing comprehensive tests.
In a similar vein, an alternate explanation may just be that a lot of bioinformatics algorithms are described as string algorithms. It would be a pity if that was just taken too literally, but I've seen a lot of bioinformatics perl that works on strings in this way.
While trying to be charitable though, it's almost impossible to come up with a rationalization for the three character offsets string. Sometimes I think when people really go wrong is when they hear of some problem or are trying to be too clever and end up going beyond their abilities and stray into farce.
Never attribute to competence that which is adequately explained by incompetence.
I am somewhat surprised that a programmer who was unaware of arrays in Perl managed to have tests. But then again, he managed to implement their own version of arrays, maybe he came up with the concept of software testing by himself :-P
So much this at my job, login to my laptop, login to vpn using password and 2-factor, check out my admin credentials using my login and 2-factor, login to the jump box using my admin creds and a different 2-factor, finally login to the system I need to be on using my admin creds. Multiply the last step by however many systems I need to connect to. Also, the clipboard and screen resolution are going to get messed up along the way no matter how much you mess with the settings.
Windows 10 by itself decides: "Hey, I'm going to replace your installed graphics card driver with some other version, that I will find myself online!" Then next time I start this OS, I get crashing graphics card driver and need to go and look on AMD website for the correct drivers, installing them over whatever crap Windows installed. This has happened at least 3 times already and it is pissing me off. It would probably be better to deactivate automatic Windows updates completely and then try to have a manual selection of installed components. But even then I could not be sure, that it will not try to replace my installed drivers with crap.
That's why nvidia has an app for that.
https://www.mit.edu/~xela/tao.html
Gatekeeping your work organization system from the people working on, and often MANAGING it is such a huge friction point I'm amazed they ever got clientele. I'm an Admin on a project right now and I can't change our status types without going through our assigned Atlassian rep.
And don't even get me started on the dumpster fire that is BitBucket. Ever tried to use the API? it's somehow even more worthless than the UI.
At a company that I used to work, they heard the same rumor, so instead of using identity columns or sequences, they kept a table with a number of ids "available" (one row per id). Whenever unique id was needed, the table would be locked, an id selected and marked as used. If there were no ids available, more ids would be added and then one used. A scheduled job would remove ids marked as used from time to time. Note that there was a single "sequence table", that was shared among all of the entities.
That was not even the weirdest part. That id was unique, but NOT the primary key of the entity, only part of it.
The structure of the database was fairly hierarchical, so you had for example a table CUSTOMER in 1-to-many relation with a USER table, with a 1-to-many relation with an ADDRESS table.
while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
Maybe they wanted to avoid maintaining a separate CUSTOMER_ADDRESS 1-to-many table or maybe it was done to make easy reverse lookups.
Programming is a lot like this.
IMO its a classic example of real world enshittification
it sucks, yes, but when coined, the term referred to a platform extracting value from its users by degrading their experience, forcing them to upgrade to a premium plan. your car not having a spare isn't an example of that!
https://en.wikipedia.org/wiki/Enshittification
Ordinary cars not having a spare tyre as standard equipment any more is a perfect example of enshittification.
It really becomes a problem when the same short term calculation becomes the standard - you can quickly rack up costs that future you or future company will never pay down.
We had to grow fast and prove that we had a market fit. This meant making compromises, but the technical debt was actually under control. We knew what to improve and how to do it once we had enough resources.
Eventually we proved ourselves and got decent funding and could hire more developers. We were looking forward to improving our system.
Unfortunately the new guys (many fresh out of college, lead by a few "FAANG"-ish seniors) decided that the technical debt was a proof of our incompetence and convinced the leadership that they needed to take over the technical development and rewrite everything. The result was a year where no major features were added and we lost our momentum. The funding dried up and the company had to let go 2/3 of the employees.
The worst part was that the new systems were more complicated to extend due to their clever "scalable design" (distributed monolith) with more micro services than developers. When the rewrite was done and it was time to add the new features, they had lost the trust of the leadership and just threw code at it in desperation to get things done as quickly as possible. I pithy the developers that inherited the result... By that time the rock stars had moved on and are inflicting damage elsewhere after touring conferences where they presented their greatness.
Exactly the opposite of the one trick pony’s who just mindlessly apply what they learned or think is cool with a total disregard for context.
This one guy established himself by making an Access database for their core business, and when the web became a thing, built a customer site. But not on it—in it. He simply served ASP pages directly from the database, inserting dynamic content in queries. When I was asked to help improve their terrible design, I was forced to untangle that unholy mess of queries, ASP (new to me) and HTML. It was easiest to write all the HTML and insert their ASP right before I sent the files back (because I wasn’t given access to their DB/web server). Thinking “I could do better than this” got me into programming.
He was a Microsoft-everything head. Finally went too far when he presented a new web interface starring a Clippy-like parrot using Microsoft’s DirectX avatar API. The executives were unimpressed and then I noted that 20% of our customers couldn’t use his site. (I probably still had a “best viewed with IE” badge on the main site, lol)
Not needing to conform to some company-wide standard is probably really pleasant while it lasted, but every such effort adds to the haunted graveyard, and the lack of consistency will eventually come back to bite whoever is still around.
[1] https://www.usenix.org/sites/default/files/conference/protec...
Codebases like this or from the OP is cool to learn how to not do certain things.
I wonder if better support of EAV tables would solve this issue better.
If one could do "SELECT price,color,year FROM cars! WHERE status='sold'" and the "!" would indicate that cars is an EAV table ...
... and the result of the query would be ... That would solve most of the use cases where I have seen people use JSON instead.