One time I reverse engineered the flight controls for a octocopter (like a quadcopter, but 8 rotors) so I could access the debug port and stream telemetry to a ground station while issuing GPS waypoints.
Probably the only thing worse than reading C++ is when it's poorly documented and in German. :P
A raid controller cleared the mbr on a windows NT domain controller, the backup did not work so I had to use Norton Disk Editor to write a new mbr, took some time since i had to scan the disc for star/end of partitions.
I discovered that our company's old ASP code has not a memory leak, but a connection leak, and after 2 days of experimenting, I managed to get our sites to load in under 1.2 seconds by recycling app pools every 15 minutes.
I taught myself how DFS works (distributed file share - an extension of AD technology, it maps a set of shortcuts on domain controllers and other DFS root holders), and expanded the available DFS roots to lead to a reduction in site load times.
I hooked up an ultrasonic distance sensor and a motor shield to an arduino, and put the assembly on an RC car. I programmed it to slow down as it approaches walls, and to back up if something is too close. I want to build robots to fill some need, but I'm still looking for that need.
I think it was around 1992/93, I was deep into graphics programming for games. The conventional way was to first draw the background and then draw everything on top of it (i.e. sprites etc.). However, this is wasting a lot of bandwidth since you can end up writing to the same video memory location multiple times.
I came up with an algorithm treating the each scan line of the screen as a binary tree which allowed me to keep track of which part of each scan line was already written to, meaning, I was able to build the screen up from front to back and visit each video memory location only once. So, on a 320x200 screen only ever 64000 bytes would be written to memory. With all the clipping etc. this was quite a complex beast and fully written in 286 assembler. In the end I think it made the overall graphics rendering about 20% to 25% faster.
Edit: I don't have the code anymore. I lost all my "floppy disks" in a house move... :(
Without real experience in Javascript nor the framework, I created a Google Hangout face tracking app in five days. You can still try it here: http://cocacola-hangout.appspot.com/
I once worked on an embedded system with a Microchip PIC. As if working on the massively underpowered CPU wasn't difficult enough, the compiler (C18) was ridden with quirks and the c libraries that came with it were full of bugs. To make matters worse the debugger displayed wrong values for variables every now and then, so I couldn't trust the debugger either...
I ended up simulating the entire application on a PC, so I had some decent debugging capabilities.
I think my dad could corroborate that. He was very deep into PIC programming about 10-15 years ago and ran into similar issues, enough that he decided to just write everything for them in assembler (mostly POCSAG/pager related stuff). He then created one of the first full visual PIC emulators to help him debug code, although it was then quickly surpassed by things like PICsim. I think he would also place all of this under some of the most extreme tech problem solving he did before retiring.
I've had the same experiences with the C18 compiler. Using that compiler, building a simple 6-channel PWM application becomes quite difficult. For every interrupt, the compiler would copy the entire (call)-stack to a different memory location, then enter the interrupt routine and finally copy the entire stack back, resulting in latencies of ±100 cycles before and after the interrupt.
So, to get rid of the timing jitter, we had to fire the timers early, and then wait for the final timer counts in the ISR.
Later, we switched to the NXP LPC platform using only open source tools: GCC ARM Embedded toolchain, OpenOCD + gdb for debugging and vim, make as 'IDE'. What a relief.
Funny that you mention that, I am currently in the process of designing the "next" generation of the project in question. Luckily I was able to convince the product owner to switch from PIC18 to NXP ARM chips.
I have loads of experience with the LPC series and I only realized how nice they are to work with until this project came by. Also, GCC + OpenOCD + gdb is a very nice toolchain to work with, although the first versions of OpenOCD were a bit of a pain to get (and keep!) running.
This isn't an objectively difficult one, but it's certainly one with a fun payoff.
I hacked together a system of file system listeners and named pipes so that I can inject code into a sandboxed app on OSX and have it communicate with the non-sandboxed process that put it there.
While working on IPTV product for Geodesic, we came across this peculiar problem our video data was shifted by 3 bytes from what we expected, throwing our decoder into fits. It took us three months of debugging, and delay of product launch , to fix the problem. A renegade "&" in C++ code, instead of value of pointer we were accessing its address . Why it caused only 3 bytes is beyond me, I was just very very happy to have fixed the issue. ( It was back in 2009 so some details are little hazy )
- Deploying a financial app through 4 bastion hosts by keeping Russian doll ssh tunnels up (clients outsource IT bouncing across the world to get to the right boxes)
- porting a 8 MLoC fortran nuclear reactor simulator from UNIX to windows
- generating PowerShell on Linux to be run on a windows box by reverse engineering the MS Api.
- silencing dialog boxes by DLLs which patch and proxy system DLLs
- making Java JRE run from a CD-ROM with the right JNI dll/so and a custom installer I wrote, before the advent of installanywhere (talking Java 1.1 days)
>- generating PowerShell on Linux to be run on a windows box by reverse engineering the MS Api.
Sorry but my god, WHY? was it morbid curiosity, because I can see why you would like to do it then or do you actually prefer the power shell syntax? (sorry if this comes off as biased, and it may very well be but I haven't actually heard of many people that can stand powershell)
I'm definitely a *nix guy (run Linux and OS X, only, for the last 5 years or so) nowadays, but I've had to use PowerShell to do some admin stuff at work... and it's not that bad. Sure beats the crap out of Batch files anyway, though it's IMO not as good at gluing disperate processes together as Sh is. That however seemed more like a function of Windows' apps themselves more than PowerShell, though.
> better than bash in terms of functionality and elegance.
How so? I haven't used it much, but I've not noticed any elegance to it? is it like lisp and if I use it for a while it will hit me what the power of it is?
Much of the functionality stems from being able to use .NET objects in the scripts, and that the result of commands are objects rather than text. For example you can pass the output of ls to a foreach loop that outputs either the absolute or relative path of each file, or various other file properties like size or modification date.
You do have to be careful in some cases because UNIX style syntax is supported but the result may be different because of the underlying implementation. Like if you output text to a file using redirection, by default the output is UTF-16. It can make you crazy if you're not aware of things like that.
A few years ago I had to create a nearest neighbour lookup algorithm that had to perform 2D searches in a microsecond with 16 million points (k-d trees and the like didn't cut it).
I spent a lot of time reading books and papers on computational geometry. I had an idea that involved a few minutes of precomputing things, and eventually came across a useful algorithm in a paper that let me implement this as I envisaged. In the end, everything worked perfectly. It was very satisfying.
The paper was only an algorithm for part of this — I can't remember the paper, but it gave an algorithm to find all of the closest points to a line.
How my algorithm worked was to break the search space into a 2D array of much smaller squares. In the initialisation phase I put the points in each square that are closest to any search point in that square (some points were in multiple squares). Therefore, when a search was run, the square the search point falls in is looked up, and the list of twenty or so points was looked through for the closest point in a slightly optimised way (no square roots here).
Going from real development tools to basic web development. Essentially from Delphi to web development, passing by QT/C++ development and with some excursions to android and other things. Journey destination : got really, really, really tired of web development and ran to something without UIs (beyond cli).
Manually making sure your frontend/backend matches. I'll repeat. Manually. Weakly typed. Constant bugs. Constantly seeing parts of your implementation getting deprecated, going unsupported, and of course not matching clients' "hot new thing" (WHY do you need a UI designer, which is a title given to every fuckwit that has ever started photoshop, successfully or -usually- not on an internal order management system ?). Impossible to generate tests that actually catch those bugs. Inconsistent language implementations, to say nothing of dom. Zero support for any kind of legacy thing. Layout engine inferior to what was available before I was born on machines that get their ass kicked by my watch (talking about the NeXT platform).
HTML/Javascripts idea of "cross-platform" does not mean even the fucking basic integer datatype is actually cross platform. HTML UI's don't reflow at all. There are UI engines that reflow pretty badly (VB). There are UI engines that reflow kinda bad (android/delphi, with delphi slightly better). There are UI engines that actually reflow (QT/GTK), I wouldn't say they do it really well, but acceptably. Web is undoubtedly much worse than VB/Delphi in reflowing. Much worse.
Getting 4 buttons stuck to the lower right of the component currently on display, which will push out other things is in-fucking-possible. You see theses written, hundreds of pages, on -get this- getting 2 columns of text displaying side-by-side. Editing a string correctly, knowing it might contain non-latin characters, is impossible. Translating a webpage dynamically from one language to another, impossible (you can change it on the server, which of course means that any strings generated in javascript need a different solution -aargh-).
Getting a redraw loop going for a game is so ridiculously difficult and slow it's painful to write about it. Give me a fucking canvas that calls me when it needs to draw itself, with OPTIONAL double-buffering. Doesn't exist (no, HTML5 canvas is not this, is slow and resource hungry, and this will never change due to how it's designed).
I was happy for 6 months with a small side-scroller implemented on a microcontroller that, together with the LCD, ran for about 6 months on an old CR2025 battery. This thing provides 225 mAh over it's entire lifetime, keeping a game going for 6 months.
The state of the art HTML5 side scroller at http://playbiolab.com/ drains my phone battery in 42 minutes flat. Granted, the music's better. The screen size isn't. That battery is a rechargeable 2300 mAh.
Getting anything remotely resembling productivity going on these platforms (I refuse to refer to web as a single platform) was so hard that I just gave up and ran the other way. And now, of course, every UI toolkit is "legacy" at best, just plain unmaintained at worst, and a non-starter for any project you might want to do.
I've been doing web dev for over 15 years now (granted, most of that as a hobby) and I've never really felt this kind of frustration. Weak typing and most of the other things you mentioned have never really bothered me. That's not to say that there aren't challenges, but that's what keeps it fun for me :)
Have you done development outside of web development ? 15 years would make your start time 1999. Did you ever do a large program with a swing gui ? A Delphi program ?
What do you think ? How do your experiences on the web compare to the others ?
Never done any Delphi and only a couple fairly small Swing/GTK apps. Most of my other development experience has been backend systems. I'll agree Web development has a lot of rough edges, but the main reason I love it so much is that it's the easiest way to get code I write in front of others.
In my case I solved a problem in genomics that people had been trying to solve for around 20 years and which would have saved the human genome project billions of dollars - the only problem is I did it 10 years too late :(
Thanks. I have always wondered why nobody else came up with the same solution or even something similar. The idea used nothing that had not been around since the 1970s so it should have been invented by someone else long before I had the idea.
For the non-molecular biologists I should explain how it would have saved billions. The human genome was effectively sequenced by breaking it millions of random fragments of around 1000 bases (letters) and the sequence (order) of the bases in each fragment determined. In order to be able to put the fragments back together in the right order each base was sequenced 10-15 times in different fragments. My idea allowed you to avoid all this redundancy meaning you had to only sequence each base once or twice. I did some simulations based on the actual costs of the human genome project and it would have saved 80% of the costs and finished it 3 years earlier.
I got handed a custom mp3 encoder and asked if I could figure out why the output was too quiet. At the time I had essentially no DSP experience.
It seems the gain had been reduced to cover up another problem: a tonal hissing sound. Once I learned how polyphase filter banks work, I tracked the problem down to a premature optimization, namely the replacement of an integer divide by 2^n with a right shift.
Such a shift of a negative 2's compliment integer rounds toward negative infinity instead of zero. This caused a slight DC bias within each sub band filter. In all but the lowest band, this DC bias gets shifted up to a non-zero frequency.
I call the optimization above premature because fixing it only added one cycle per operation. Granted, this was a real-time MP3 encoder on an ARM7, but the cycles were there.
This was a small consumer electronics company no longer in existence. The encoder was written by a quite clever dutch hacker, whom I very much enjoyed working with.
At a different gig I built a linux x264 batch transcode cluster that accepted, among other formats, Apple ProRes. (Quicktime under wine using xvfb, through avisynth, piping a yuv4mpeg stream out of the wine domain).
At a yet another gig I developed a method to differentiate male and female insects at 30m using a beam-steered high speed low res camera.
I took horrible basically no documentation globally scoped spaghetti code in perl, with a proprietary db based on flat files and forking web scraper from the 90s, which used n ancient form of ipc that predated posix and mostly didn't run, and rewrote, debugged, unit tested and updated it so it runs, faster and better than before. I built a new front end using dancer (perl framework kinda like Sinatra) with a nginx reverse proxy to replace the cgi scripts originally used.
I took horrible basically no documentation globally scoped spaghetti code in perl, with a proprietary db based on flat files and forking web scraper from the 90s, which used n ancient form of ipc that predated posix and mostly didn't run, and rewrote, debugged, unit tested and updated it so it runs, faster and better than before. I built a new front end using dancer (perl framework kinda like Sinatra) with a nginx reverse proxy to replace the cgi scripts originally used.
I used to work for a massive media company for a short stent and I kept trying to explain to them why DRM will always be broken but it just never took. I think it's hard for non-technical people to think about the 100+ ways media leaks on its way to the human's ears or eyes. :)
I've done quite a bit of stuff on the hardware/software boundary where things get hairy. I think the nastiest thing I've debugged there was a machine that HP [1] was making in about 1994 which had some native HP bus and an EISA bus hanging off it for expansion cards from PCs.
I was working for a company that made NICs which went into the EISA bus and we were seeing data corruption in this machine.
After a long, cold night in the Apollo works myself and an engineer from HP tracked it down to a timing problem on the EISA bus where the 16 bits being sent were arriving in two 8 bit chunks slightly delayed. Our NIC was spinning on a 16 bit word looking for a change in the top 8 bits as a single that the data had arrived. We'd then read all 16 bits, but 8 bits weren't ready yet.
Luckily, HP made lots of test equipment and getting a logic analyzer with 32 inputs, an in circuit emulator for the CPU, and some logic probes was easy...
When doing computer vision the first thing being done is usually camera calibration. The most common calibration technique is to make images of a physical object with know spatial properties (grid patterns are often used) and to extract 2D point positions from the resulting images (e.g. using a corner detector). Given the match between the extracted 2D points and the corresponding locations on the calibration grid it is possible to determine camera and lens parameters such focal length and distortion.
An in-house camera calibration application at my company solved the correspondence problem by using an algorithm that looked at properties that are not invariant under projection (i.e. angles and distances). This made the calibration process extremely fragile. The algorithm often failed to detect the calibration grid even when the image was crystal clear, which made calibrating a camera a very frustrating endeavour.
Since it was an in-house application there was never much priority to fix it. Eventually I got so annoyed though that I wrote a whole new real-time algorithm for the detection of arbitrarily-sized grids from a set of 2D points. The algorithm is capable of dealing with a significant amount of outliers (even when between the valid grid points), it can handle missing grid points and it is not affected by perspective or non-linear lens distortion. In the end it took me longer that I had hoped, but the resulting algorithm is one that still makes me proud.
A professional grade USB audio driver for Windows, to stream multichannel audio at low latencies to connected USB hardware. Also firmware for said hardware, with DSP processing (equalizers, filters, mixing, interpolation, metering). Completed with zero prior experience, and a little outside help.
187 comments
[ 4.2 ms ] story [ 525 ms ] threadProbably the only thing worse than reading C++ is when it's poorly documented and in German. :P
I taught myself how DFS works (distributed file share - an extension of AD technology, it maps a set of shortcuts on domain controllers and other DFS root holders), and expanded the available DFS roots to lead to a reduction in site load times.
I hooked up an ultrasonic distance sensor and a motor shield to an arduino, and put the assembly on an RC car. I programmed it to slow down as it approaches walls, and to back up if something is too close. I want to build robots to fill some need, but I'm still looking for that need.
I came up with an algorithm treating the each scan line of the screen as a binary tree which allowed me to keep track of which part of each scan line was already written to, meaning, I was able to build the screen up from front to back and visit each video memory location only once. So, on a 320x200 screen only ever 64000 bytes would be written to memory. With all the clipping etc. this was quite a complex beast and fully written in 286 assembler. In the end I think it made the overall graphics rendering about 20% to 25% faster.
Edit: I don't have the code anymore. I lost all my "floppy disks" in a house move... :(
Later, we switched to the NXP LPC platform using only open source tools: GCC ARM Embedded toolchain, OpenOCD + gdb for debugging and vim, make as 'IDE'. What a relief.
I have loads of experience with the LPC series and I only realized how nice they are to work with until this project came by. Also, GCC + OpenOCD + gdb is a very nice toolchain to work with, although the first versions of OpenOCD were a bit of a pain to get (and keep!) running.
I hacked together a system of file system listeners and named pipes so that I can inject code into a sandboxed app on OSX and have it communicate with the non-sandboxed process that put it there.
- Deploying a financial app through 4 bastion hosts by keeping Russian doll ssh tunnels up (clients outsource IT bouncing across the world to get to the right boxes)
- porting a 8 MLoC fortran nuclear reactor simulator from UNIX to windows
- generating PowerShell on Linux to be run on a windows box by reverse engineering the MS Api.
- silencing dialog boxes by DLLs which patch and proxy system DLLs
- making Java JRE run from a CD-ROM with the right JNI dll/so and a custom installer I wrote, before the advent of installanywhere (talking Java 1.1 days)
Sorry but my god, WHY? was it morbid curiosity, because I can see why you would like to do it then or do you actually prefer the power shell syntax? (sorry if this comes off as biased, and it may very well be but I haven't actually heard of many people that can stand powershell)
It's ugly in its own unique way, but it's better than bash in terms of functionality and elegance.
How so? I haven't used it much, but I've not noticed any elegance to it? is it like lisp and if I use it for a while it will hit me what the power of it is?
You do have to be careful in some cases because UNIX style syntax is supported but the result may be different because of the underlying implementation. Like if you output text to a file using redirection, by default the output is UTF-16. It can make you crazy if you're not aware of things like that.
I spent a lot of time reading books and papers on computational geometry. I had an idea that involved a few minutes of precomputing things, and eventually came across a useful algorithm in a paper that let me implement this as I envisaged. In the end, everything worked perfectly. It was very satisfying.
How my algorithm worked was to break the search space into a 2D array of much smaller squares. In the initialisation phase I put the points in each square that are closest to any search point in that square (some points were in multiple squares). Therefore, when a search was run, the square the search point falls in is looked up, and the list of twenty or so points was looked through for the closest point in a slightly optimised way (no square roots here).
Manually making sure your frontend/backend matches. I'll repeat. Manually. Weakly typed. Constant bugs. Constantly seeing parts of your implementation getting deprecated, going unsupported, and of course not matching clients' "hot new thing" (WHY do you need a UI designer, which is a title given to every fuckwit that has ever started photoshop, successfully or -usually- not on an internal order management system ?). Impossible to generate tests that actually catch those bugs. Inconsistent language implementations, to say nothing of dom. Zero support for any kind of legacy thing. Layout engine inferior to what was available before I was born on machines that get their ass kicked by my watch (talking about the NeXT platform).
HTML/Javascripts idea of "cross-platform" does not mean even the fucking basic integer datatype is actually cross platform. HTML UI's don't reflow at all. There are UI engines that reflow pretty badly (VB). There are UI engines that reflow kinda bad (android/delphi, with delphi slightly better). There are UI engines that actually reflow (QT/GTK), I wouldn't say they do it really well, but acceptably. Web is undoubtedly much worse than VB/Delphi in reflowing. Much worse.
Getting 4 buttons stuck to the lower right of the component currently on display, which will push out other things is in-fucking-possible. You see theses written, hundreds of pages, on -get this- getting 2 columns of text displaying side-by-side. Editing a string correctly, knowing it might contain non-latin characters, is impossible. Translating a webpage dynamically from one language to another, impossible (you can change it on the server, which of course means that any strings generated in javascript need a different solution -aargh-).
Getting a redraw loop going for a game is so ridiculously difficult and slow it's painful to write about it. Give me a fucking canvas that calls me when it needs to draw itself, with OPTIONAL double-buffering. Doesn't exist (no, HTML5 canvas is not this, is slow and resource hungry, and this will never change due to how it's designed).
I was happy for 6 months with a small side-scroller implemented on a microcontroller that, together with the LCD, ran for about 6 months on an old CR2025 battery. This thing provides 225 mAh over it's entire lifetime, keeping a game going for 6 months.
The state of the art HTML5 side scroller at http://playbiolab.com/ drains my phone battery in 42 minutes flat. Granted, the music's better. The screen size isn't. That battery is a rechargeable 2300 mAh.
Getting anything remotely resembling productivity going on these platforms (I refuse to refer to web as a single platform) was so hard that I just gave up and ran the other way. And now, of course, every UI toolkit is "legacy" at best, just plain unmaintained at worst, and a non-starter for any project you might want to do.
That is actually considered progress.
How did webdev ever get this fucked up ?
What do you think ? How do your experiences on the web compare to the others ?
In my case I solved a problem in genomics that people had been trying to solve for around 20 years and which would have saved the human genome project billions of dollars - the only problem is I did it 10 years too late :(
Edit. If anyone is interested I published the method in BMC Genomics a few years ago http://www.biomedcentral.com/1471-2164/10/344
For the non-molecular biologists I should explain how it would have saved billions. The human genome was effectively sequenced by breaking it millions of random fragments of around 1000 bases (letters) and the sequence (order) of the bases in each fragment determined. In order to be able to put the fragments back together in the right order each base was sequenced 10-15 times in different fragments. My idea allowed you to avoid all this redundancy meaning you had to only sequence each base once or twice. I did some simulations based on the actual costs of the human genome project and it would have saved 80% of the costs and finished it 3 years earlier.
It seems the gain had been reduced to cover up another problem: a tonal hissing sound. Once I learned how polyphase filter banks work, I tracked the problem down to a premature optimization, namely the replacement of an integer divide by 2^n with a right shift.
Such a shift of a negative 2's compliment integer rounds toward negative infinity instead of zero. This caused a slight DC bias within each sub band filter. In all but the lowest band, this DC bias gets shifted up to a non-zero frequency.
I call the optimization above premature because fixing it only added one cycle per operation. Granted, this was a real-time MP3 encoder on an ARM7, but the cycles were there.
At a yet another gig I developed a method to differentiate male and female insects at 30m using a beam-steered high speed low res camera.
https://www.youtube.com/watch?v=eYXPqrXZ1eU
I used to work for a massive media company for a short stent and I kept trying to explain to them why DRM will always be broken but it just never took. I think it's hard for non-technical people to think about the 100+ ways media leaks on its way to the human's ears or eyes. :)
Most of them were reinventing the wheel. :-(
I think the chemists have a saying: "With a few hard and dedicated work weeks in the lab, you might save hours in the library!"
I was working for a company that made NICs which went into the EISA bus and we were seeing data corruption in this machine.
After a long, cold night in the Apollo works myself and an engineer from HP tracked it down to a timing problem on the EISA bus where the 16 bits being sent were arriving in two 8 bit chunks slightly delayed. Our NIC was spinning on a 16 bit word looking for a change in the top 8 bits as a single that the data had arrived. We'd then read all 16 bits, but 8 bits weren't ready yet.
Luckily, HP made lots of test equipment and getting a logic analyzer with 32 inputs, an in circuit emulator for the CPU, and some logic probes was easy...
[1] I think it was an HP 9000 http://en.wikipedia.org/wiki/HP_9000 being made at the old Apollo works outside Boston.
An in-house camera calibration application at my company solved the correspondence problem by using an algorithm that looked at properties that are not invariant under projection (i.e. angles and distances). This made the calibration process extremely fragile. The algorithm often failed to detect the calibration grid even when the image was crystal clear, which made calibrating a camera a very frustrating endeavour.
Since it was an in-house application there was never much priority to fix it. Eventually I got so annoyed though that I wrote a whole new real-time algorithm for the detection of arbitrarily-sized grids from a set of 2D points. The algorithm is capable of dealing with a significant amount of outliers (even when between the valid grid points), it can handle missing grid points and it is not affected by perspective or non-linear lens distortion. In the end it took me longer that I had hoped, but the resulting algorithm is one that still makes me proud.
Very proud of the result :-) https://www.youtube.com/watch?v=SmMHsIETBwA