Lets not kid ourselves, a few million rows is not small either and if not done properly can quickly become a system's bottleneck. But it's not humongous either. Depending on ones perspective it could be seen as medium or large but certainly not small.
I think it's safe to call anything that fits within the memory of a mid range notebook computer "small". I guess it depends on how wide those tables are.
Anybody who has a "bottleneck" caused by a table with a few million rows is functionally incompetent. From any half-modern database's perspective that's "tiny". Any reasonable production machine would be able to fit (at least the important parts of) a million row table in memory. Even if this is your hobby project running on a RaspberryPi Zero, you'd have at least the index in ram.
Either you don't care about that query's runtime (a perfectly valid approach if your use case is "this creates a report after COB on Friday that needs to have completed before 9am Monday morning"), or you need someone who knows what they're doing take over from you (also a valid approach if this is in your tech demo or POC, and you're the Technical Founder who's now out of their depth and needs a proper engineer to build a scalable product for you).
Rows with an integer id and a few integer or bit flag columns or maybe a short short fixed width char column or two and a few well though out indexes, is a way different beast compared to something with dozens of big varchar() or blob columns (or worse still, json or xml text columns with un-indexable key/values inside that you need to parse out before you can run selects on them...)
Ah, I think I understand now. It's not the amount of data (rows), it's how that data is structured (columns). I thought you were saying that number of columns was more important than number of rows, which clashed with my naive intuition.
Modern RDBMS's can support JSON and XML as native data types, i.e. you absolutely can index within them and they have facilities to speed up the parsing.
This thought has a very narrow perspective. During the space program, a few million rows was very large. Today, you have better tools so a few million rows is small to you. But someone else who works on Fugaku has tools that you don't have and will find that your "large" amount of data is small to them -- they also get to use the electrical equivalent of 20,000 homes to process that data.
Most people today don't have the computing tools that you do. Yes, a consumer laptop running Numpy can process it quickly. But they have Excel, not Numpy...and Excel cannot process millions of rows. So in the context of the tools they have, it is a large table.
IMHO when discussing "Big Data" you have to go up PB (in total) and some large numbers (in daily) to reach "Big Data". If the data can be stored inside a commercial Xeon box then it's not even "Medium Data".
to me, Big Data is more of a concept than just a description of the amount of data. So ontop of the storage, it's also the analysis and usage of that data.
So Big Data services could be just as useful to a small company with a mere 100GB of data once they learn how to squeeze the juice from the berries they've collected.
But, if it is 100 GB of data, Big Data tools are the wrong tools, because they are inefficient at that size. Most Big Data come with lot's of restricitions and complications which are fine if they are the only alternative. But don't ever think about using Cassandra instead of Postgres if you have less than 10 TiB. Even over 10 TiB it is not clear when to switch to something more scalable.
Again, this comment is focused on the size of the data on storage and how much data there is. To me, this is just a single aspect of "Big Data". To me, "Big Data" is also what information can be gleaned from that data that is being stored whether it be Postgres, NoSQL, Cassandra, etc. That part of it is only relevant in conversations about "how much data" is available to the "Big Data" processes. People paying for "Big Data" are excited about "how much data" they have other than a bragging point. What they are paying for is the information that can be garned for having that data.
Incidentally this is why you should be extremely skeptical of predictions by pure academics about the attainability of certain performance _at a given price_. It was the simultaneous increase in performance, and collapse in cost per transistor that only business at scale could deliver… that made Roy Lightfoot and his American contemporaries look foolish just a couple of decades later.
I vividly remember the moment I was reading about the IBM mainframes used by NASA during the 1960s, trying to convert the words of memory from the amount of bits the system used, to something more modern to give me some perspective.
Naturally, I fired up [Microsoft] Calculator, and arrived at something like 4kb. Later, I had to force quit a stalled process and opened Task Manager, to find that I had forgot to close Calculator.
There it was, idling at 20mb. Initially, I thought perhaps it was slowly leaking memory, so I restarted it. Just the same.
Now, I can already anticipate someone motivating why a calculator needs to exceed the computing power of calculating ~5000 space programs. To that I say, Windows 95 runs at 4mb of memory. Microsoft Calculator is not more complex than Windows 95. At least, I deeply hope that it isn't.
I think a downvote on this is unfair. Excessive memory usage starts in the data model. And yea, a difference of a few bytes there can quickly stack up (sorry for pun).
PDP-6's Type 360 CRT had a reachable resolution of 1024x1024 pixels, with 3 bit per pixel for intensity level. I was a vector based screen so was operating a bit differently compared to modern framebuffer based screens. Works for text based applications like hn for example, but fails for video or picture heavy stuff like what we have in social networks, youtube, etc.
I agree though that framebuffers put a hard lower limit onto how much RAM you have to use if you want to write a graphical application or even entire OS that saves resources.
The display is a refrigerator-sized unit with a big circular CRT in the middle. You give it vectors and characters, and it draws them with 1024x1024 resolution. It also had a light pen that you could use for interaction with the display.
It is much more programmable than I expected. Even for the slow speeds of the day, it looks like animations and display updates could happen locally and that the systems could be made to be fairly responsive.
Joining displays together, scaling modes, pixel, line and text drawing and conditional subroutines, and all in 30 ish pages.
When did any PC display get hardware line drawing? It would be fun to implement Logo that interacted with the light pen for creating generative L-systems.
From your original comment it sounded like you may have had first hand experience. There are a handful of 70+ year old folks on hn. Good to hear that folks are looking at all periods of computing.
Pointers are usually stored in 64-bits on 64-bit systems, but some of the bits may be unused depending on hardware. Each CPU/platform has its own limit for the size of the virtual and physical address space. But the standard size of pointers in memory stays the same.
Indeed! Apple decides the amount of RAM they put in iPhones not based on projected application RAM usage, but on camera pixel count and display resolution.
The TI89 is a ridiculous outlier in terms of capability to compute resources though. Every other piece of software I can think of is a pathetic joke in comparison.
"To that I say, Windows 95 runs at 4mb of memory."
My second PC was a 80486/25MHz with 4Mb of RAM - four 1MB DIMMs. I skipped 80386!
Anyway, that box ran Win 95 eventually after DOS 6 and Win 3.1, then 3.11 W4WG. I also tried out OS/2 on it - 24 1.44Mb floppies I think. It had a new MB with a DX2/66 and loads of RAM (32Mb) later on and ran Win98.
I grew up shortly after that and Linux is my weapon errr OS of choice. Looks quite cool these days and I get less problems with Teams than my MS sporting colleagues. When I need to get something installed I ask pacman or yay to install it and then crack on, whilst my colleagues play with Google and dodgy downloads.
Anyway, on Win9x, calc ran quite happily on a box with 4Mb RAM. Word 2.0 worked too ...
I watched a presentation on the Apollo Guidance Computer recently. I'd seen one before but this one had a lot more information.
The ROM on the device was just a series of magnets and wires. Magnetic core memory without the extra wires to write the values. Instead the values were 'written' by having someone thread the wires through the right magnets. Thousands of times. It was a very surreal "wait, that's how they did that? ...well of course, how else would they do that?" moment.
I liked the explanation Dave from Dave's Garage gave in his "Why does rebooting fix everything" video. The hypothetical example he gave of a Canadian word like "colour" versus "color" that u may eventually cause a memory leak. US programmers wouldn't think of it as a bug since they'd only assume color existed not colour.
That 20 MB of space, would mostly be for writing graphics code. All the images, resources, hidden stuff, animations, code to run those animations, etc.
Its not that the core program has increased in size or has lost performance, but that the code required to build a good program based on that core has increased exponentially.
Also, as a developer, I usually don't care much about memory until I complete the program, check if its consuming abnormally high memory and then sit about to optimize. So, the developer for calculator.exe saw that the program consumes about 0.125% of commonly available memory size (16 GB), and probably concluded that its good enough for a program that is opened and closed, maybe 10 times a day.
Its the same with processing power. Unless you are writing highly resource hungry systems, losing a few thousand cycles here and there are not an issue.
Compounding this problem is the depth of the tech stack and the typical number of stacks used for an application have increased tremendously.
I’m still floored when I use the MacBook calculator to calculate something like 873*523 and it takes 3 seconds to get an answer. Not sure why. There’s no animation that I can see and the it has the foreground.
I guess we can chalk it up to OS prioritization or something similar. With no other programs running, calculator on my Windows does everything instantly. With Chrome or Firefox, it lags. I would also guess that the OS considers the calculator as non critical or something.
This has been bugging the shiznit out of me. However, I dont think it is taking that long to compute. I think it is just not displaying the answer until a redraw like moving window. Whatever the cause, it's absolutely a joke that it is still not fixed.
I more ask, how much money and effort would you spend for a calculator program which saved that $0.16c worth of RAM[1] which you didn't even notice was being used until something else drew it to your attention? If someone built one, would you buy it? I can only suspect not, since there are a lot of calculator apps out there which you could have found and installed for free.
You probably used more paper in your lifetime of printing and book reading and newspaper buying and phonebook and bank statement receiving than Euclid, Pythagoras, Archimedes, Euler, Gauss and Newton combined, is that a problem worth solving?
[1] first price for memory I found on Crucial.com was 8GB for $67.99. Other prices are available.
It's using more than ripcord, the discord client, even when using voice chat. Compare that to the official app that makes you have to worry about ram prices and actually takes a good chunk of a modern common pcs memory.
Discord is using 157MB of 24GB on my desktop right now. 0.6%
On my laptop, 114MB of 8GB. 1.3%.
I even used Windows Calculator to work out these percentages because the effort to make the "calculator" button on the keyboard load another calculator, and find another one to load, and learn to use it, to save 19MB of RAM for a few seconds is not worth it.
Interesting that the author concludes his otherwise interesting observations by discussing the advanced processing power of the iPhone. I don’t dispute the numbers or the results, but when I want to do actual work, I choose my aged Linux laptop every time. The iPhone feels more like a thin client to someone else’s power.
That's true, but there have been several experiments with expanding the phone to a full desktop when you plug it into a monitor with the necessary USB ports. Apple won't ever support it, but several Android phones, Windows phones and Ubuntu phones do. The problem is usually software, either because of third party software not running or clunky first party support for adding a screen.
With the power of a modern phone, all you really need is a monitor with a USB hub to turn it into a fully fledged PC. Sure, you won't be doing too much in the way of debugging code or editing video, but for most people it should be enough. It's not like the 4K screen on your phone has that fewer pixels than the 4K monitor you hook it up to.
From what I can tell, you can run a usable desktop environment if you plug in a modern high-end Android device to a suitable monitor. Good enough for browsing the web, editing basic spreadsheets and doing homework, anyway.
IMO this could be the ultimate tool for companies with loads of employees that don't do too complex work and get fancy company phones already. Even if you don't use the power of the phone directly, you could use a thin client app to save costs there. I think eventually someone will be able to make a business case out of it, but so far all attempts have failed.
Sadly, my phone doesn't do DisplayPort out, so when I tested a standard laptop dock all it did was add a mouse cursor and a notification about the physical keyboard to my phone. I bought a midrange phone, though so that's kind of to be expected.
Unless your laptop has 128GB of ram, it's not even a million 7090s.
CPU time is essentially free now. When you price e.g. cloud instances you can nearly ignore everything except RAM and non-volatile storage. If you want more than 16GB of ram you need to go for the "mobile workstation" class devices.
For the curious, I found this manual[1] about programming the IBM 7090 (the mainframe mentioned in the article).
The machine language and corresponding assembly code is still recognizable by today's standards. It looks pretty similar to the Z-80, 6502 and 8086 assembly that I learned back in the day.
To me, it feels different. It feels like "data processing" rather than like "computing". I can't precisely define the difference, but it feels different.
Maybe it's because it's from a typewriter, rather than a typeset manual. Maybe it's because things are in octal, instead of in assembly opcodes.
My take is a bit different: the early microprocessors were a big step backward in functionality. The 7090 was a 36-bit machine, while the Z-80 and 6502 were 8 bit. These microprocessors didn't support multiplication or division. The 7090 supported not only integer multiplication and division, but floating point. Even the 8086 didn't have floating point (without the 8087). The 7090 also had channels, basically a built-in DMA engine for I/O.
But overall I agree with you that the assembly code is fairly similar as far as the basic structure. Some of the early computers have pretty strange assembly code that requires a big mental shift. For example the Honeywell 1800 with two program counters that can go forward or backward, or the IBM 1401 with its word marks and arbitrary-length words.
Early microprocessors like the Z80 and the 6502 were a lot weaker than even minicomputer CPU's, they were more like glorified calculator chips. The Intel 80386 and Motorola 68020 were notable for being among the first really popular micros to reach true parity with the minicomputer featureset.
The two program counters were first introduced in the National Bureau of Standards' SWAC computer. The idea was that you would use the second program counter for a subroutine call, and then switch back to the first program counter to return to your original code. I don't think anyone really used the backward counting on the program counter; as far as I can tell, that was a side-effect of all the registers being able to auto-increment or auto-decrement.
I entered grad when labs were freed from the tyranny of central university mainframe into local computers. The mainframe inputted punchcards and output to lineprinter. The lab computer ran UNIX and had CRT terminal input/output.
I don't remember why I was there, particularly as a physics major, but I remember a single look through while half-opening the door to a 4th? floor lab in Evans (UCB) in 1978 or so, understanding it to be the Unix lab. Not a brush-with-greatness moment per se, but "how might life be different if I had gone through the door?". I might have been lookiing for one Dr. McKracken or someone.
Missed the colloquium two years later where the talk afterward was: Everyone will have a VAX on their desk.
Now you get to enjoy the tyranny of central cloud management and deployment infrastructure with cloud shell and browser clients. :)
First UNIX was Xenix on a PC tower that the teacher would carry around for the OS classes, where we would take turns after preparing the class material on a MS-DOS PC.
Later when I got to university they were on DG/UX with the same tyranny of a central server to telnet into.
I remember in 2013 I spent $3000 on the latest everything and built myself a new PC. My wife walked in just after I got it running and looked at me; I had graphs and spreadsheets all over the screen.
"I can tell you're not working. What on earth are you doing?"
"Calculating how far back in time I would have to go until my PC was the fastest computer on Earth. 1992 is the answer. It would be faster than DARPA's $90m supercomputer that filled a warehouse. If I go back to 1984 my PC is actually faster than every computing device on Earth, combined."
> Most modern computer cores can operate at a sustained rate of 3 billion instructions per second, with much faster peak speeds.
3 billion instructions per second is an IPC less than 1. You can often expect IPCs closer to 2 even on older Skylake derived cores, so just shy of 10 billion instructions per second. Very modern top end cores do even better.
> But I wonder, are we using all that computation effectively to make as much difference as our forebears did after the leap from pencil and paper to the 7090?
This sentiment seems very much like an Eternal September reference. I think widely available is great and the use cases should naturally dilute in terms of importance. It would be weird if people today only bought computers when trying to solve problems of the Apollo scale. I like that many users are effectively using technology as a toy, it’s become so much more that many back then would have imagined. Of course, some bad with the good but a net gain IMO.
Consider Electron: grotesquely inefficient in terms of CPU usage, because it's optimized for minimizing developer time instead.
We aren't solving the same problems our forebears did. People weren't designing native apps with pencil and paper when somebody figured out how to bundle up the Chromium Embedded Framework, but they were burning tons of time duplicating effort on multiple native apps because OS platform vendors are manically determined to lock us in and stop us from writing portable code.
Typescript + Next.js + Electron would be much, much faster for me to develop for than GTK + C, and even than Python + PyQT.
I wish there existed a modern cross-platform open source reactive GUI framework which was less of a resource hog than Electron, that would be great! Qt specifically is not much more economical, though the redistributable is smaller. Java is also rather nice for cross-platform GUIs, and also rather a hog.
Sadly, Electron is not so outstanding wrt resource consumption :-/
> I wish there existed a modern cross-platform open source reactive GUI framework which was less of a resource hog than Electron
Flutter (Desktop) comes pretty close to that. Its really fast to write and your code works on IOS/Android/MacOS/Windows/Linux unless you need Platform specific API's, and even that is possible.
> Is developer time actually being conserved though?
in the case of Electron, probably.
it is comparatively difficult to release a cross platform graphical application in any other way.
in the case of other "high level" languages like Python, almost certainly not.
I remember reading something on Dropbox' engineer or developer blog about their conversion from Python 2 to Python 3 and how well it went for their desktop client, which had over one million lines of code and I had to stop and reread that a few times to make sure I read correctly.
How "high level" is Python if this many lines of code are required to create the Dropbox client? maybe Python is the epitome of a high level language, I don't know, but if it is, I don't see how one million lines of Python saves anyone any time over something low level like C or even Assembly, really. the promised tradeoff of execution speed for programmer productivity isn't there, at all, and almost no one sees it.
even if the majority of those lines of code are in libraries, why do those libraries have so many lines of code? what can they possibly be doing to need that many lines of code? it really does boggle my mind.
entire operating systems require fewer lines of code than the Dropbox client, and are written in languages that are supposedly much less expressive per line of code. so where is this supposed developer productivity when it takes over one million lines of code to write the Dropbox client?
I'm not trying to pick on Python, specifically, mind you; this is just an example I was thinking about recently. I'm quite sure that there are other tradeoffs also not delivering what they promised at all in all other languages, and I'm 110% certain no one is noticing those, either.
> it is comparatively difficult to release a cross platform graphical application in any other way.
wxWidgets has been enabling cross-platform native graphics since the 1990s. Though the overall development experience is rather MFC-like, and not to many people's taste.
> what can they possibly* be doing to need that many lines of code? it really does boggle my mind.*
Unless you've spent lots of time (half a decade or more) at each layer of "full stack", it is really hard to comprehend just how mind-bogglingly fractal systems requirements get the more users you expose them to. Anytime you go b2c, or even b2b with a large enough user base, be prepared to say, "huh, I didn't see that coming" once a week or more frequently.
There are entire problem domains that lay submerged until you reach certain scales. Some code bases reflect the encapsulation of those submarine domains; some don't, most assuredly, but until the software becomes unused, at my clients I never ask the question the way you ask. Instead, I ask them to walk me through the code at successively finer-grained conceptual blocks until I reach the level where I can start attacking the problem I've been brought in to help address.
>I don't see how one million lines of Python saves anyone any time over something low level like C or even Assembly
I do somehow agree with you, but going from Assembly to C was a huge step, from C to Java/C# a huge step due to GC and better typing.
Python indeed does not feel much higher level to me than e.g. C#, the only reason I've ever really seen why people save lines of code is the lack of braces, which doesn't really matter IMO, and because there are so many libraries. A big reason why there are so many libraries? I guess because people think it's easy, nothing innate in the language, certainly not that it's easy to package stuff.
The Dropbox client is busy trying to steal your screenshots and flash drive contents, and promote Dropbox Pro every time your storage goes above 90% full, or even randomly otherwise. Though I can't see how that would take 100k lines of code out of the million total.
Developer time is not infinite, so a specific ecosystem optimizing for it doesn't come to me as weird.
You might begrudge the resources used at scale, but if these programs weren't built in the first place it would be net negative. I also don't see Slack coded in GTK for instance, if it wasn't electron it would have been some Java framework, and I'm not sure it would have been better.
Comparing VSCode with Eclipse (which was optimized compared to other frameworks....), I'm pretty happy with VSCode and think we came a long way.
Even a Java based gui written using the Swing toolkit (which I believe is considered ancient tech that most have moved on from although I haven't built a gui in java in ages so don't take my word for it) would be an improvement on these electron apps just because it will prevent the cancer that is Javascript, node, npm and the entire JS ecosystem from invading the desktop. We deal with JS/Node on the frontend because we have no choice, that tech invading the server and the desktop is just sad to watch.
I found that we have the choice and successfully migrated an enormous bloatware of thousands of binding setinterval functions (Angular 1) full of node dependencies, npm breakages, impossible to understand grunt builds into a simple, 10 files pure javascript + some jquery approximately well compatmentalised set of bespoke business services.
The only critic I ever faced with this was that it was not classical nor "the proper way", but the fact it loaded instantly and worked exactly the same way albeit infinitely faster managed to convince we may not need all that jazz.
You always have a choice but you'll face more resistance and worry not to go with everyone else towards suicide by a thousand dependencies.
> Even a Java based gui written using the Swing toolkit (which I believe is considered ancient tech that most have moved on from although I haven't built a gui in java in ages so don't take my word for it) would be an improvement on these electron apps just because it will prevent the cancer that is Javascript, node, npm and the entire JS ecosystem from invading the desktop.
I remember a very similar complaint being made about Eclipse, Java, and Swing as you make about Electron, JS, and Node as a cancer on the desktop.
From my Java days, backend work looked like creating plugins to auto-generate XML, which would be parsed by the build system to auto-generate classes...on a philosophy level, I can't remember people preaching simplicity and pragmatism outside of the Eclipse people (who had to deal with real hard complexity issues day in and day out...)
An alternative history where Java would have been huge on the desktop doesn't look to me like we'd have a heaven of well-though lightweight apps TBH.
I personnally think npm is something inevitable, and am not sure Maven for instance is fundamentally better in the grand scheme of things (I'd argue as a dev it's way more of a PITA without any specific upside, but it might be my own incompetence).
> it would have been some Java framework, and I'm not sure it would have been better.
Java Swing is much more responsive and light when compared to Electron. Java is no longer heavy since the first Core series processors of Intel, and it's visibly optimized to be lighter and faster every release.
> Comparing VSCode with Eclipse (which was optimized compared to other frameworks....), I'm pretty happy with VSCode and think we came a long way.
Eclipse of today can run circles around both the Eclipse of yesteryear and VScode in terms of performance, resource utilization and capabilities.
I'm using Eclipse for 20 years (and still using it), and just because it's old enough to get a driver's license, it doesn't mean it's obsolete or featureless.
If you want to try an example of a truly scalable and snappy GUI experience with lightweight Java check my node/tree editor out: http://move.rupy.se/file/logic.html
> because OS platform vendors are manically determined to lock us in and stop us from writing portable code
Sorry, but that’s both flat out not true, and a fundamental misunderstanding of what the role of a platform vendor is.
If Apple wanted to ban non-native apps on iOS it’s hard to see what would stop them. They’re clearly fine with them. Microsoft has even embraced cross platform development wholeheartedly with many Electron apps, and more to come. They’re even plumbing the chrome engine into Windows to make sharing it more efficient.
The role of a platform vendor though is to make the best platform for native development they can. Portability to other platforms, with a few exceptions such as POSIX, just isn’t their problem. There’s no particular reason it should be. It would just stifle innovation and lead to boring me-too systems that are stuck in the past. That’s simply not in the interests of users or vendors.
Depends who is "we". It does seem like a waste if a considerable amount of the user's resources is wasted for developer convenience, path dependency and politics.
A bit of inefficiency is alright, but things get ridiculous if modern web pages actually run less smooth than programs 40 years ago despite having quadrillion times the computing power available - because this power is wasted to write objects to JSON half a dozen times or simulate a network architecture between a virtual server and a virtual client which actually reside on the same machine - all because that's the only architecture modern web developers seem to understand.
IMHO "Electron is bloaty" has become a meme on its own which just gets repeated over and over without actually checking how much of it is true.
For instance when comparing VSCode to Visual Studio or Xcode, VSCode isn't universally slower, far from it. VSCode is much faster from start to first keystroke than both VS or Xcode. In other areas it's not so clear, some actions are slower in VSCode, some are faster (and before "but VSCode is a text editor, not an IDE", with the right plugins VSCode is as much an IDE as it needs to be).
So if even the Visual Studio or Xcode teams at Microsoft and Apple can't create applications that are universally better than an Electron application, why should 3rd party developers care about creating "native" applications?
The Windows and macOS platform teams don't seem to have quite understood yet that they are in competition with Electron, and in order to compete, they have to make it easier to create cross-platform applications than what Electron provides. Nobody will choose native macOS or Windows APIs just because they provide a "native look and feel" (which neither Microsoft nor Apple care about all that much in their own applications either).
My problem with electron is not that the apps are slow it is that I can (usually) guess if an app is electron by seeing which ones are using up an entire core when the app is just sitting doing nothing
That's not what I'm seeing with VSCode on macOS though: the highest priority task when sitting "mostly idle" 'Code Helper (Renderer)' at 0.2%, which is at 7th or 8th position (and for instance below an idle iTerm2).
I suspect that the applications that use up a core when idle would do this also if Electron is not used (e.g. it's a property of the development team, not of the framework).
Maybe, but it seems there’s something about development teams that choose to write native apps that avoid that because I’ve never guessed an app was electron and been wrong using this method.
VSCode is the outliner in Electron and not the norm. M$ had a whole team on VSCode dedicated just for performance. Outliner to Outliner you may want to compare VSCode to Sublime Editor.
Yes, almost as if performance doesn't happen on its own. It must be a stated goal and the whole team must work towards that goal (you really don't need a "dedicated performance team", with or without Electron).
Well yes the author quipped about Minecraft, but I can’t be alone in that my job as a PM consists of a lot of email and word processing, tasks that we’re solves decades ago. But somehow Outlook demands millennia worth of 7090’s processing power to show me my email.
It's such a lazy and useless conclusion, that overrides all the other cool comparisons the rest of the article provided. But I guess it drives engagement, QED...
It can also be seen as a sort of challenge: for the longest time what we could compute was limited by our hardware, and we ran close to its limits. Nowadays what we can compute is mostly limited by our imagination, and we run close to those limits. The gauntlet thrown down by that closing statement is to dream bigger.
I'm always amazed about how modern web sites will render different direction quotes, will render apostrophes as something other than ', yet they can't seem to get a carat (^) to work. What's the problem here? Is ieee.org unfamiliar with the carat? Or do they really refer to a "quadrillionth (10-15)" as negative five?
Sorry - I'm picking nits. But consider how many sites have examples of shell commands where the apostrophes and backticks are converted to something else, to where two dashes are converted to an emdash, and all I can think is how completely disconnected writers are from what appears on web sites.
When I was a mainframe engineer at IBM, I heard more than once that IBM would never fall behind because we had a whole pipeline of incredible machines in development, and they could just be brought forward if there was ever a really serious competitive threat.
What an horrible website. Click to read. I missed it at first and did not understand what was happening. Then as I start reading, I get a popup in the face. And I have an adblocker at that! I left at the pop-up.
I was confused by the quadrillion figure for a while, then I remembered that our quadrillions are bigger. (Does this qualify as an exception to the rule that everything is bigger in Texas?)
Well, we do have all of that computation power. The question for what do you want to use it? What kind of endeavors do we have for which we could use all of that power? The better question would be, what would the world look like today if we didn't have all of that computation power.
"are we using all that computation effectively to make as much difference as our forebears did[...]?"
With Bitcoin we are literally using this power to prove to each other we burned some megajoules which we could otherwise use to power some physical stuff.
The difference made so far: it is easier to pay for drugs in order to evade stupid anti-drug laws.
I'm genuinely amazed at the level of deep-seated hatred folks on HN have towards crypto.
Aside from the fact that your comment is completely OT, what could possibly motivate you, at every tiny opportunity to do so, to spew so much negativity on a topic that - in all likelihood - has exactly zero impact on your life?
As much as I try to imagine another scenario, I find myself forced to cycle back to a nocoiner type explanation, which if correct, would be very sad indeed.
A friend of mine first wanted to get into computers in the mainframe era. He took his first program to the counter and the operator told him to come back tomorrow. He returned the next day and there had been a syntax error in his code. This won't do, he said. So he gave up.
Move forward a few years and he got himself a C64. Now we can do something with a computer!
142 comments
[ 3.4 ms ] story [ 200 ms ] thread\s
Anything that fits in ram, at least in the context of "a table" in a RDBM probably counts as 'micro data'...
Either you don't care about that query's runtime (a perfectly valid approach if your use case is "this creates a report after COB on Friday that needs to have completed before 9am Monday morning"), or you need someone who knows what they're doing take over from you (also a valid approach if this is in your tech demo or POC, and you're the Technical Founder who's now out of their depth and needs a proper engineer to build a scalable product for you).
I don’t know about “big” or “small”, but the threshold for “done properly” has to be really low for that to become a bottleneck.
Most people today don't have the computing tools that you do. Yes, a consumer laptop running Numpy can process it quickly. But they have Excel, not Numpy...and Excel cannot process millions of rows. So in the context of the tools they have, it is a large table.
So Big Data services could be just as useful to a small company with a mere 100GB of data once they learn how to squeeze the juice from the berries they've collected.
Naturally, I fired up [Microsoft] Calculator, and arrived at something like 4kb. Later, I had to force quit a stalled process and opened Task Manager, to find that I had forgot to close Calculator.
There it was, idling at 20mb. Initially, I thought perhaps it was slowly leaking memory, so I restarted it. Just the same.
Now, I can already anticipate someone motivating why a calculator needs to exceed the computing power of calculating ~5000 space programs. To that I say, Windows 95 runs at 4mb of memory. Microsoft Calculator is not more complex than Windows 95. At least, I deeply hope that it isn't.
Resolution does nasty things to memory.
As does having every memory allocation be 64bits wide (instead of 8 or 16).
Though I agree with your sentiment.
I agree though that framebuffers put a hard lower limit onto how much RAM you have to use if you want to write a graphical application or even entire OS that saves resources.
The display is a refrigerator-sized unit with a big circular CRT in the middle. You give it vectors and characters, and it draws them with 1024x1024 resolution. It also had a light pen that you could use for interaction with the display.
Joining displays together, scaling modes, pixel, line and text drawing and conditional subroutines, and all in 30 ish pages.
When did any PC display get hardware line drawing? It would be fun to implement Logo that interacted with the light pen for creating generative L-systems.
*edit I found a video linked by Don Hopkins of Conway's Game of Life running on a 340 display https://www.youtube.com/watch?v=hB78NXH77s4
Indeed! Apple decides the amount of RAM they put in iPhones not based on projected application RAM usage, but on camera pixel count and display resolution.
https://www.macrumors.com/2021/09/15/how-much-ram-in-iphone-...
My second PC was a 80486/25MHz with 4Mb of RAM - four 1MB DIMMs. I skipped 80386!
Anyway, that box ran Win 95 eventually after DOS 6 and Win 3.1, then 3.11 W4WG. I also tried out OS/2 on it - 24 1.44Mb floppies I think. It had a new MB with a DX2/66 and loads of RAM (32Mb) later on and ran Win98.
I grew up shortly after that and Linux is my weapon errr OS of choice. Looks quite cool these days and I get less problems with Teams than my MS sporting colleagues. When I need to get something installed I ask pacman or yay to install it and then crack on, whilst my colleagues play with Google and dodgy downloads.
Anyway, on Win9x, calc ran quite happily on a box with 4Mb RAM. Word 2.0 worked too ...
The ROM on the device was just a series of magnets and wires. Magnetic core memory without the extra wires to write the values. Instead the values were 'written' by having someone thread the wires through the right magnets. Thousands of times. It was a very surreal "wait, that's how they did that? ...well of course, how else would they do that?" moment.
Timestamp 2:55 https://www.youtube.com/watch?v=9IPP39OF78M
IIRC, Windows 3.0 didn't run happily in any circumstances, that had to wait for 3.1.
Its not that the core program has increased in size or has lost performance, but that the code required to build a good program based on that core has increased exponentially.
Also, as a developer, I usually don't care much about memory until I complete the program, check if its consuming abnormally high memory and then sit about to optimize. So, the developer for calculator.exe saw that the program consumes about 0.125% of commonly available memory size (16 GB), and probably concluded that its good enough for a program that is opened and closed, maybe 10 times a day.
Its the same with processing power. Unless you are writing highly resource hungry systems, losing a few thousand cycles here and there are not an issue.
Compounding this problem is the depth of the tech stack and the typical number of stacks used for an application have increased tremendously.
You probably used more paper in your lifetime of printing and book reading and newspaper buying and phonebook and bank statement receiving than Euclid, Pythagoras, Archimedes, Euler, Gauss and Newton combined, is that a problem worth solving?
[1] first price for memory I found on Crucial.com was 8GB for $67.99. Other prices are available.
On my laptop, 114MB of 8GB. 1.3%.
I even used Windows Calculator to work out these percentages because the effort to make the "calculator" button on the keyboard load another calculator, and find another one to load, and learn to use it, to save 19MB of RAM for a few seconds is not worth it.
I think that’s why people are so amped about the M1 (and getting Linux into it)
And the form factor. It’s hard to do any serious work on a phone.
With the power of a modern phone, all you really need is a monitor with a USB hub to turn it into a fully fledged PC. Sure, you won't be doing too much in the way of debugging code or editing video, but for most people it should be enough. It's not like the 4K screen on your phone has that fewer pixels than the 4K monitor you hook it up to.
From what I can tell, you can run a usable desktop environment if you plug in a modern high-end Android device to a suitable monitor. Good enough for browsing the web, editing basic spreadsheets and doing homework, anyway.
IMO this could be the ultimate tool for companies with loads of employees that don't do too complex work and get fancy company phones already. Even if you don't use the power of the phone directly, you could use a thin client app to save costs there. I think eventually someone will be able to make a business case out of it, but so far all attempts have failed.
Sadly, my phone doesn't do DisplayPort out, so when I tested a standard laptop dock all it did was add a mouse cursor and a notification about the physical keyboard to my phone. I bought a midrange phone, though so that's kind of to be expected.
CPU time is essentially free now. When you price e.g. cloud instances you can nearly ignore everything except RAM and non-volatile storage. If you want more than 16GB of ram you need to go for the "mobile workstation" class devices.
CPU > Ram > Network > Disk(even the fastest).
That’s at least the most difference compared to a normal server or even a desktop. The markup on CPU cores in the cloud is insane.
The machine language and corresponding assembly code is still recognizable by today's standards. It looks pretty similar to the Z-80, 6502 and 8086 assembly that I learned back in the day.
[1]https://justine.lol/sectorlisp/ibm709.pdf
Maybe it's because it's from a typewriter, rather than a typeset manual. Maybe it's because things are in octal, instead of in assembly opcodes.
But overall I agree with you that the assembly code is fairly similar as far as the basic structure. Some of the early computers have pretty strange assembly code that requires a big mental shift. For example the Honeywell 1800 with two program counters that can go forward or backward, or the IBM 1401 with its word marks and arbitrary-length words.
For a lot of old IBM 7090 manuals, see: http://bitsavers.org/pdf/ibm/7090
Wow! What is the point of that? How is it used?
(For 2nd year, we got a room full of Mac 512ks...)
Missed the colloquium two years later where the talk afterward was: Everyone will have a VAX on their desk.
First UNIX was Xenix on a PC tower that the teacher would carry around for the OS classes, where we would take turns after preparing the class material on a MS-DOS PC.
Later when I got to university they were on DG/UX with the same tyranny of a central server to telnet into.
"I can tell you're not working. What on earth are you doing?"
"Calculating how far back in time I would have to go until my PC was the fastest computer on Earth. 1992 is the answer. It would be faster than DARPA's $90m supercomputer that filled a warehouse. If I go back to 1984 my PC is actually faster than every computing device on Earth, combined."
"I'm sorry I asked."
Not exactly the same position, but very similar in the ecosystem. Both of them will be blamed when the system is not available.
3 billion instructions per second is an IPC less than 1. You can often expect IPCs closer to 2 even on older Skylake derived cores, so just shy of 10 billion instructions per second. Very modern top end cores do even better.
This sentiment seems very much like an Eternal September reference. I think widely available is great and the use cases should naturally dilute in terms of importance. It would be weird if people today only bought computers when trying to solve problems of the Apollo scale. I like that many users are effectively using technology as a toy, it’s become so much more that many back then would have imagined. Of course, some bad with the good but a net gain IMO.
We aren't solving the same problems our forebears did. People weren't designing native apps with pencil and paper when somebody figured out how to bundle up the Chromium Embedded Framework, but they were burning tons of time duplicating effort on multiple native apps because OS platform vendors are manically determined to lock us in and stop us from writing portable code.
Typescript + Next.js + Electron would be much, much faster for me to develop for than GTK + C, and even than Python + PyQT.
I wish there existed a modern cross-platform open source reactive GUI framework which was less of a resource hog than Electron, that would be great! Qt specifically is not much more economical, though the redistributable is smaller. Java is also rather nice for cross-platform GUIs, and also rather a hog.
Sadly, Electron is not so outstanding wrt resource consumption :-/
Flutter (Desktop) comes pretty close to that. Its really fast to write and your code works on IOS/Android/MacOS/Windows/Linux unless you need Platform specific API's, and even that is possible.
in the case of Electron, probably. it is comparatively difficult to release a cross platform graphical application in any other way.
in the case of other "high level" languages like Python, almost certainly not.
I remember reading something on Dropbox' engineer or developer blog about their conversion from Python 2 to Python 3 and how well it went for their desktop client, which had over one million lines of code and I had to stop and reread that a few times to make sure I read correctly.
How "high level" is Python if this many lines of code are required to create the Dropbox client? maybe Python is the epitome of a high level language, I don't know, but if it is, I don't see how one million lines of Python saves anyone any time over something low level like C or even Assembly, really. the promised tradeoff of execution speed for programmer productivity isn't there, at all, and almost no one sees it.
even if the majority of those lines of code are in libraries, why do those libraries have so many lines of code? what can they possibly be doing to need that many lines of code? it really does boggle my mind.
entire operating systems require fewer lines of code than the Dropbox client, and are written in languages that are supposedly much less expressive per line of code. so where is this supposed developer productivity when it takes over one million lines of code to write the Dropbox client?
I'm not trying to pick on Python, specifically, mind you; this is just an example I was thinking about recently. I'm quite sure that there are other tradeoffs also not delivering what they promised at all in all other languages, and I'm 110% certain no one is noticing those, either.
Do you have some baseline to which we can compare? How many lines "should" it take?
By no means do I wish to awaken the ghost of that infamous HN comment on how easy/why someone else had not done those bells and whistles before.
I ran `git clone` on the repos of both rsync[0] and the official Dropbox SDK for Python[1]. I then ran sloccount on both of them.
---
rsync came in at a total of 51,410 physical source lines of code.
The Dropbox SDK came in at a total of 74,140 physical source lines of code.
---
(This data generated using David A. Wheeler's 'SLOCCount'.)
[0] https://git.samba.org/rsync.git
[1] https://github.com/dropbox/dropbox-sdk-python
wxWidgets has been enabling cross-platform native graphics since the 1990s. Though the overall development experience is rather MFC-like, and not to many people's taste.
Unless you've spent lots of time (half a decade or more) at each layer of "full stack", it is really hard to comprehend just how mind-bogglingly fractal systems requirements get the more users you expose them to. Anytime you go b2c, or even b2b with a large enough user base, be prepared to say, "huh, I didn't see that coming" once a week or more frequently.
There are entire problem domains that lay submerged until you reach certain scales. Some code bases reflect the encapsulation of those submarine domains; some don't, most assuredly, but until the software becomes unused, at my clients I never ask the question the way you ask. Instead, I ask them to walk me through the code at successively finer-grained conceptual blocks until I reach the level where I can start attacking the problem I've been brought in to help address.
I do somehow agree with you, but going from Assembly to C was a huge step, from C to Java/C# a huge step due to GC and better typing.
Python indeed does not feel much higher level to me than e.g. C#, the only reason I've ever really seen why people save lines of code is the lack of braces, which doesn't really matter IMO, and because there are so many libraries. A big reason why there are so many libraries? I guess because people think it's easy, nothing innate in the language, certainly not that it's easy to package stuff.
You might begrudge the resources used at scale, but if these programs weren't built in the first place it would be net negative. I also don't see Slack coded in GTK for instance, if it wasn't electron it would have been some Java framework, and I'm not sure it would have been better.
Comparing VSCode with Eclipse (which was optimized compared to other frameworks....), I'm pretty happy with VSCode and think we came a long way.
The only critic I ever faced with this was that it was not classical nor "the proper way", but the fact it loaded instantly and worked exactly the same way albeit infinitely faster managed to convince we may not need all that jazz.
You always have a choice but you'll face more resistance and worry not to go with everyone else towards suicide by a thousand dependencies.
I remember a very similar complaint being made about Eclipse, Java, and Swing as you make about Electron, JS, and Node as a cancer on the desktop.
An alternative history where Java would have been huge on the desktop doesn't look to me like we'd have a heaven of well-though lightweight apps TBH.
I personnally think npm is something inevitable, and am not sure Maven for instance is fundamentally better in the grand scheme of things (I'd argue as a dev it's way more of a PITA without any specific upside, but it might be my own incompetence).
I don't think so. Different programs might have been built in a more efficient framework instead.
Java Swing is much more responsive and light when compared to Electron. Java is no longer heavy since the first Core series processors of Intel, and it's visibly optimized to be lighter and faster every release.
> Comparing VSCode with Eclipse (which was optimized compared to other frameworks....), I'm pretty happy with VSCode and think we came a long way.
Eclipse of today can run circles around both the Eclipse of yesteryear and VScode in terms of performance, resource utilization and capabilities.
I'm using Eclipse for 20 years (and still using it), and just because it's old enough to get a driver's license, it doesn't mean it's obsolete or featureless.
Sorry, but that’s both flat out not true, and a fundamental misunderstanding of what the role of a platform vendor is.
If Apple wanted to ban non-native apps on iOS it’s hard to see what would stop them. They’re clearly fine with them. Microsoft has even embraced cross platform development wholeheartedly with many Electron apps, and more to come. They’re even plumbing the chrome engine into Windows to make sharing it more efficient.
The role of a platform vendor though is to make the best platform for native development they can. Portability to other platforms, with a few exceptions such as POSIX, just isn’t their problem. There’s no particular reason it should be. It would just stifle innovation and lead to boring me-too systems that are stuck in the past. That’s simply not in the interests of users or vendors.
Even electron apps aren't new, we already had that disease once with Active Desktop and packaged web sites.
There is some hope that like 20 years ago eventually the bubble will burst.
A bit of inefficiency is alright, but things get ridiculous if modern web pages actually run less smooth than programs 40 years ago despite having quadrillion times the computing power available - because this power is wasted to write objects to JSON half a dozen times or simulate a network architecture between a virtual server and a virtual client which actually reside on the same machine - all because that's the only architecture modern web developers seem to understand.
For instance when comparing VSCode to Visual Studio or Xcode, VSCode isn't universally slower, far from it. VSCode is much faster from start to first keystroke than both VS or Xcode. In other areas it's not so clear, some actions are slower in VSCode, some are faster (and before "but VSCode is a text editor, not an IDE", with the right plugins VSCode is as much an IDE as it needs to be).
So if even the Visual Studio or Xcode teams at Microsoft and Apple can't create applications that are universally better than an Electron application, why should 3rd party developers care about creating "native" applications?
The Windows and macOS platform teams don't seem to have quite understood yet that they are in competition with Electron, and in order to compete, they have to make it easier to create cross-platform applications than what Electron provides. Nobody will choose native macOS or Windows APIs just because they provide a "native look and feel" (which neither Microsoft nor Apple care about all that much in their own applications either).
I suspect that the applications that use up a core when idle would do this also if Electron is not used (e.g. it's a property of the development team, not of the framework).
This is a worthy end goal.
Sorry - I'm picking nits. But consider how many sites have examples of shell commands where the apostrophes and backticks are converted to something else, to where two dashes are converted to an emdash, and all I can think is how completely disconnected writers are from what appears on web sites.
Apologies for the tangent.
(Source: I used to work at the magazine and am painfully aware of previous such mistakes.)
It's hard to say if the success of "mainframe" before has become the reason why IBM has been falling behind in the cloud competition.
With Bitcoin we are literally using this power to prove to each other we burned some megajoules which we could otherwise use to power some physical stuff.
The difference made so far: it is easier to pay for drugs in order to evade stupid anti-drug laws.
Aside from the fact that your comment is completely OT, what could possibly motivate you, at every tiny opportunity to do so, to spew so much negativity on a topic that - in all likelihood - has exactly zero impact on your life?
As much as I try to imagine another scenario, I find myself forced to cycle back to a nocoiner type explanation, which if correct, would be very sad indeed.
[1] https://ccaf.io/cbeci/index
Move forward a few years and he got himself a C64. Now we can do something with a computer!
The C64 is the most important change in history, it will be celebrated as long as we still can afford 15W.