I love this! It would be great to have follow-ups and series of books about how to make things worse, as a way to learn to make things better.
Maybe it could be like O’Reilly, except the covers could have shittily-drawn fantasy animals, e.g. a 7-year-old’s drawing of a unicorn with a head on each side of its body both talking on their AirPods giving away their money to scammers, making PowerPoint slides, eating too much, doing hard drugs, live-streaming on Facebook, and standing on the railroad tracks with a train in the distance.
If you are ever going to get to grips with optimizing something, why not do the opposite first or as a foil?
How often do you really fenangle your database (or other system) properly? Are your performance enhancements really based on science or cargo culting or something else?
The Defence of Duffer's Drift is an early example of this genre.
In the first story in the book you learn how to do platoon level tactics badly( and lose most of your men )
In each subsequent story, a few tactical parameters are tweaked
and the story is recounted again, with improved results.
The same approach is used in newer tactical books like Musicians of Mars 2.
The first Team Badger story in Musicians is very similar to the first story in Duffer's Drift, allowing for changes in location, equipment and techniques..
"The lingering sulfur odor of gunpowder, mixed in the haze of the dust and
smoke of battle, dominated the senses of CPT Fred Morris, commander of
Team Badger. He sat amid the wreckage of what used to be some of the
world’s finest combat machines, M1A2 Abrams tanks and M2A3 Bradley
fighting vehicles.
Now most of his company’s tanks and Brads were mere
smoking hulks, emanating the distinct smell of burning electronics to add
another element to the senses of the dismal scene before him.
He sat, quietly reflecting on the sequence of events that led to the
destruction of his beloved team. His mind raced as he recounted how the
enemy force, so much more formidable than he had expected, employed
assets, capabilities, and tactics that he had not taken into account. At the
same time, he wondered why he was unable to bring to bear his own
extensive combat power to earn decisive victory. He had felt so ready, so
confident, and believed that his team had prepared so diligently.
Yet, still
they had been defeated so soundly. How could that be?
LTC Joe Milner, the task force commander, had given Team Badger the task
to defend the center of the task force main battle area (MBA) in an area
defense to destroy the attacking enemy motorized rifle brigade. Specifically,
CPT Morris and Team Badger were to defend against the enemy’s main
effort motorized rifle battalion in Battle Position (BP) Badger, along the
enemy’s most likely avenue of approach.
That motorized rifle battalion
plowed through Team Badger as if it were not even there."
I really want to see some operational setups be available as a sort of playground for observability tooling.
A decent sized SaaS-like with usage simulation, and a postgres/rabbit setup that is merely fine. A place to check if your debugging tooling/strategy is upt to snuff.
And forget indexes, multiple tables, transactions, entity relationships, and referential integrity. Use a single table as a KVS NoSQL SQL for all data just like TRIRIGA's early versions did.
I wrote an identity/auth service that had the backend like that intentionally. I wanted to be able to adapt it to pretty much any backend that was desired. Each identity had 4 entries with predictable keys, and the data was json that was optionally encrypted through the app, if using a dbms that didn't provide encryption at rest.
The data access interface was simple enough, that it allowed interface layering allowing for different backends, encryption, etc. to be very flexible.
I had adapters for SQLite (default), MS-SQL, PostgreSQL, Redis, Cassandra and AWS DynamoDB. It was deployed for apps that went from embedded, to integrated with our applications, and external applications on internal and external infrastructures.
The only disappointing thing is that I wasn't allowed to open-source/release the application. The concern is it would be a security risk if anyone knew how the application worked. I still disagree with this decision. It was a bit simpler, and more flexible than "Identity Server" in use, but it served the needs of a lot of applications where an existing identity provider (IS, Octa, AD Auth, etc) wasn't already in place. Most of the apps I worked on at that company had used JWT allowing for somewhat flexible configurations of RSA signed providers... integration usually consisted of a bridge service for authentication going from the client's provider to an app targeted JWT. This kept integrations separate.
I really like the idea of finding what dials can be turned and then doing so and seeing how much headroom you normally have and how far away from things breaking you are.
Plus, those sorts of artificial limitations can be really helpful in finding where your systems bottleneck and what are the main optimizations that you should do (for example, I've seen N+1 problem sneak past because at the time the performance was good enough but once there was enough data it crumbled).
Great read, and an interesting warning sign about overly-controllable configuration files. If there's no conceivable good reason to to much of this, then the ability to do so becomes bad design that fails to protect your users.
A practical use of this would be in helping to determine where memory utilization typically is vs. where it really needs to be and which knobs are tuned where/how.
If you wanted to run PostgreSQL in a lower memory device, such as a lower end RPi, then you might want specific features tuned down based on your real workloads. Especially if you are considering say SQLite, but aren't sure how it will perform vs. putting say half the system's resources to PG.
21 comments
[ 3.3 ms ] story [ 36.5 ms ] threadMaybe it could be like O’Reilly, except the covers could have shittily-drawn fantasy animals, e.g. a 7-year-old’s drawing of a unicorn with a head on each side of its body both talking on their AirPods giving away their money to scammers, making PowerPoint slides, eating too much, doing hard drugs, live-streaming on Facebook, and standing on the railroad tracks with a train in the distance.
If you are ever going to get to grips with optimizing something, why not do the opposite first or as a foil?
How often do you really fenangle your database (or other system) properly? Are your performance enhancements really based on science or cargo culting or something else?
In the first story in the book you learn how to do platoon level tactics badly( and lose most of your men )
In each subsequent story, a few tactical parameters are tweaked and the story is recounted again, with improved results.
The same approach is used in newer tactical books like Musicians of Mars 2.
The first Team Badger story in Musicians is very similar to the first story in Duffer's Drift, allowing for changes in location, equipment and techniques..
Warning: PDFs below:
The Defence of Duffer's Drift https://www.armyupress.army.mil/Portals/7/combat-studies-ins...
Musicians of Mars 2 https://api.army.mil/e2/c/downloads/2023/01/19/5a01ae1c/16-1...
"The lingering sulfur odor of gunpowder, mixed in the haze of the dust and smoke of battle, dominated the senses of CPT Fred Morris, commander of Team Badger. He sat amid the wreckage of what used to be some of the world’s finest combat machines, M1A2 Abrams tanks and M2A3 Bradley fighting vehicles.
Now most of his company’s tanks and Brads were mere smoking hulks, emanating the distinct smell of burning electronics to add another element to the senses of the dismal scene before him.
He sat, quietly reflecting on the sequence of events that led to the destruction of his beloved team. His mind raced as he recounted how the enemy force, so much more formidable than he had expected, employed assets, capabilities, and tactics that he had not taken into account. At the same time, he wondered why he was unable to bring to bear his own extensive combat power to earn decisive victory. He had felt so ready, so confident, and believed that his team had prepared so diligently.
Yet, still they had been defeated so soundly. How could that be? LTC Joe Milner, the task force commander, had given Team Badger the task to defend the center of the task force main battle area (MBA) in an area defense to destroy the attacking enemy motorized rifle brigade. Specifically, CPT Morris and Team Badger were to defend against the enemy’s main effort motorized rifle battalion in Battle Position (BP) Badger, along the enemy’s most likely avenue of approach.
That motorized rifle battalion plowed through Team Badger as if it were not even there."
A decent sized SaaS-like with usage simulation, and a postgres/rabbit setup that is merely fine. A place to check if your debugging tooling/strategy is upt to snuff.
The data access interface was simple enough, that it allowed interface layering allowing for different backends, encryption, etc. to be very flexible.
I had adapters for SQLite (default), MS-SQL, PostgreSQL, Redis, Cassandra and AWS DynamoDB. It was deployed for apps that went from embedded, to integrated with our applications, and external applications on internal and external infrastructures.
The only disappointing thing is that I wasn't allowed to open-source/release the application. The concern is it would be a security risk if anyone knew how the application worked. I still disagree with this decision. It was a bit simpler, and more flexible than "Identity Server" in use, but it served the needs of a lot of applications where an existing identity provider (IS, Octa, AD Auth, etc) wasn't already in place. Most of the apps I worked on at that company had used JWT allowing for somewhat flexible configurations of RSA signed providers... integration usually consisted of a bridge service for authentication going from the client's provider to an app targeted JWT. This kept integrations separate.
Sorry for veering off a bit.
Plus, those sorts of artificial limitations can be really helpful in finding where your systems bottleneck and what are the main optimizations that you should do (for example, I've seen N+1 problem sneak past because at the time the performance was good enough but once there was enough data it crumbled).
If you wanted to run PostgreSQL in a lower memory device, such as a lower end RPi, then you might want specific features tuned down based on your real workloads. Especially if you are considering say SQLite, but aren't sure how it will perform vs. putting say half the system's resources to PG.