Pretty awesome, I for one don't think the constant stream of "x, rewritten completely in javascript" is tiring (this is not meant to be sarcastic).
[EDIT] - This post doesn't seem to say enough when I look back at it. Wanted to add this:
Seeing posts like this really excites me (and inspires me) about the future of web programming, and programming in general. Anyone that's excited about programming has to get excited about abstraction, and it doesn't get much more abstract that cross-coded/implemented virtualized systems like this. Even if you hate javascript.
Every step people take in blurring lines between systems like this should be exciting, given the large amount of abstraction that had to go into creating something like this.
This may as well be magic as far as I'm concerned. When I first saw http://bellard.org/jslinux/ I just about shat myself. This is definitely another level of awesome. Well done.
This really is extremely well done [and I'd like to see the unminified source]. The next feature I'd want to see is networking support. I love the idea to be able to run my entire dev setup in an emulated browser session [e.g. ruby, rails, redis]. That would be a game changer on how I'd develop.
Some things would do badly at that speed. But serving basic web queries would be fine, and give you a better feel for what is slow when milliseconds spent processing turn into actual seconds.
Just for reference's sake this does vary a fair bit by computer and browser - mine never drops below ~8kIPS and runs mostly at ~15kIPS (and up to ~23kIPS).
Whilst this is nothing like the several orders of magnitude required for more capable interactive behaviour, it does make my load times noticeably faster than those claimed on the page. :)
For reference, that's about as fast as a Motorola 68030 @50MHz [1] which was used in the AppleIIx, and while that was no slouch when it was released, you wouldn't want to run a modern development environment on such a machine either ;)
I think it would be really cool to link this up with some of the Chrome App API's (TCP/UDP networking, serial). Then you could distribute lightweight programs without bothering with Emscripten or Native Client.
Go to any machine with a browser, log in, and have your dev environment pop up in the same state it was when you left it somewhere else, yet all running locally rather over some VNC connection or potentially laggy ssh connection. Log out, and have the state persisted.
I'm presently working on networking (OpenCores ethmac) support for the jor1k[1] OpenRisc emulator. The plan is to push ethernet frames out over a web socket, and pipe them into a tun interface on the server via a node/python/C/whatever backend. From there, do whatever you want using your OS' standard networking tools (if you have specific needs for the server-side of this which go beyond using your os' standard tools, please let me know).
If you're interested in this, please do make plenty of noise at me to finish. I've got a flight back to the East coast from Portland on Sunday, so I'm hoping I can get most of the browser code finished by then. After that I'm going to be somewhat distracted through the end of January, as I'm moving from Charleston, SC to Christchurch, New Zealand. I could use a bit of motivation to get this done.
I am really looking forward. If you have any wishes for an image with some tools, please let me know. Lynx, elinks, dillo, ssh, netcat, telnet ... whatever you want. Then I have to think about a demo. Maybe we can allow very limited access to some websites.
I think the biggest thing I'm overlooking right now is randomized MAC generation. I need to take a look at whether or not the ethoc driver will attempt to read the MAC from hardware before loading its own. If so, I can randomize MACs in my ethmac.js, or even make a backend service assign them to guarantee uniqueness.
Otherwise I think it'd be great to get an image together with all of those tools, especially Lynx. It'd be great if I could make the HN submission to the demo page via the emulator itself. However for now if I can get an ICMP echo I'll be happy.
I think when it's ready I'm going to set up a full-access reCaptcha-protected demo on DigitalOcean or AWS and see what happens. If people don't abuse it and if the bill isn't too bad, I'll leave it up. If not, I'll worry about how to do a restricted demo. It would be nice to get a network-friendly game in there, though.
Another great demo that we'd have difficulty with is browser-in-browser. Chrome-in-chrome, firefox-in-firefox. That would, however, be the ultimate sandbox.
Anyway, get in touch with me via e-mail or better yet, Skype (see profile). We can coordinate better there.
Wait how does this differ from JSLinux? Aren't they doing the same thing?
Also, this isn't just an x86 emulator, right? If it can boot linux, it's emulating the entire PC platform, not just x86. Very impressive of course. But I am interested in a comparison vs jslinux (also not open source).
I had a quick look at the minified source. its pretty hard to decipher (method names have been renamed too), unlike traditional minification which you can recover by just reformatting.
AFAIK most of minificators also shortened variable / function names - are there any that don't do this? (take a look at minified jquery.js for instance)
Css get minified by removing whitespace because the keywords are important.. Javascript can have some magic done to it to reduce character count because only the consistency of the keywords matters.
I just ran Bellard's JSLinux (#) on my retina iPad with iOS 7. It works! Booting took 15 sec. Claims 20 bogoMips. And I can type the commands in it and execute them:
First I tap to the edit box on the right, the keyboard rolls in, then I tap in the "console." The commands work. I just don't know how I'd type ctrl-Keys without the physical keyboard.
Copy.sh ends with unimplemented GP handler however for anything I've tried.
#) JSLinux: the PC emulator is written in Javascript with the emulated hardware: 32 bit x86 compatible CPU,
8259 Programmble Interrupt Controller,
8254 Programmble Interrupt Timer,
16450 UART,
Real Time Clock,
IDE interface and hard disk. http://bellard.org/jslinux/tech.html
In a case like this, the more important licensing question is that of the x86 instruction set; I do not know what the strict licensing situation would be with that—it's quite possible (probable, actually) that this is technically illegal.
For the software part, why would you advocate GPLv3+ over permissive licenses like MIT/BSD/Apache?
Could you give some tips or links on how to go about emulating an architecture? I would like to know how to go about emulating an architecture much less complicated like PIC.
You might want to consider re-ordering the font list in the console though. Consolas isn't perfectly fixed-width apparently, and it made Rogue rather puzzling to play.
FF 24 is the latest stable. FF's docs say they added this function in FF 4.0, which was several years ago. So it's a bit more complicated than you make it out to be.
EDIT: I thought FF24 wasn't working, turns out I was confused between my Linux desktop's and Mac laptop's FF versions. Sorry!
Yea, Chrome DevTools has a pretty print option too, but if they ran it through something like closure compiler it renames all the vars. Would like to browse source on github or comparable -- also see the history :)
Nothing in the web platform lets you use raw (or TCP/UDP/etc) sockets. At most you could do VPN-style networking and use a bridge to talk to the rest of the internet; that bridge needs someone to host it.
Interesting bug: I can't seem to type '-', '=', '+', or '_' into the Linux image. Missing '-' in particular makes it hard to run commands with options.
Elaborate workaround:
/root% eval eq$(dmesg | grep 'e820 update' | sed 's/.*) \(.\).*/\1\1/')
/root% echo $eq
=
/root% eval dash$eq$(uname bad 2>&1 | grep Usage: | sed 's/.*\[\(.\).*/\1/')
/root% echo $dash
-
/root% uname ${dash}a
Linux (none) 2.6.34.14 #44 Tue Oct 15 20:50:15 CEST 2013 i586 GNU/Linux
The first command grabs an '=' from dmesg and sets "eq" to it (without typing '='), and then the second command grabs a '-' from the usage message of uname and sets "dash" to that (without typing '-'). The last shows how to use ${dash} in a command.
This is the most soul-crushing category of bug, the kind which keeps me up all night, drinking.
Build a skyscraper, and lock the keys inside for the ribbon cutting. Sorry pal you lost your funding
Land on Mars, see an alien, but your camera is out of batteries and everybody back on Earth thinks you're a quack.
Successfully perform open-heart surgery on a desert island, but muck up the stitches and cause a scar. Your patient will hold it against you the rest of his life (which you saved).
The soul-crushing bugs are the ones where you run a program, and nothing actually goes actively wrong, but the program overall just doesn't behave quite like it does on the real hardware.
Simple defects where a key is being ignored on certain systems are usually quite easy to fix ;)
Luckily, he was able to come up with a workaround.
Here's a soul-crushing situation that doesn't have a workaround, as far as I can tell.
You can play the original Deus Ex on OnLive. By default, it sets the fov to a very low number. It's only possible to change it in the console (there is no UI for it). But the console is disabled. However, you can get around that by pressing T, which creates a mini console with "say " prepoulated. Press delete to erase say, and type "fov 90". So close...
But they disabled the Enter key from working.
So you can get full access to the console, type the command that would fix the FOV, but you're unable to execute it because Enter key does not get sent to the game. You're completely helpless.
OnLive runs on OSX. gdb (i think) also runs on OSX. It should be possible to isolate the callback used to handle keyboard input and call it with c=10. This would take anywhere from hours to weeks of debugging though.
No, that won't work. Nothing you do client-side will help.
OnLive Client <-all input is transmitted-> OnLive Servers <-filter out Enter-> Deus Ex instance in a VM.
The enter key is filtered out on their servers. Unless you hack into their servers and change that, there is nothing you can do on the client-side. That's what makes you completely helpless.
Ah, my apologies, I thought the game was running locally. (This sort of assholeness on the part of servers is basically why 'cloud computing' is such a bad idea.)
I disagree that cloud computing is a bad idea because of this. It has advantages and disadvantages. Not being able to hack around to make changes you wish is a disadvantage, but having access to something you would not have otherwise is a much bigger advantage.
Also, any plans to implement GPF handling? I tried booting a test environment I have on it (which runs Python as part of the GRUB2 bootloader), but it failed primarily due to lack of support for trapping GPFs. (There are a few other issues I'd have to work around to make it work, but that's a major one that I can't easily work around.)
I found that bug too. First thing I did was think, "a system restore is a page refresh away and I've always wanted to test the effects of this", then I attempted to type "rm -r /*".
This is beyond awesome! I'd love to read about your thought process while writing it, I guess a lot of people (including myself) would learn a lot from it. How long did it take to write?
I got Freedos working about 2 years ago, but then stopped working on it. I picked it up again earlier this year. Approximately 3 months of full time work.
I'm considering writing some articles, just need to find some free time.
g3: Probably you have seen my emulator.
http://s-macke.github.io/jor1k/
It does more or less the same, but emulates a different CPU.
Your emulator is impressive and especially fast. I think it took a long time for you to optimize it.
I tried to start TinyCore, but it stops after decompressing the kernel. So still some work to do ;)
I saw the source and I have some tips for you:
Avoid the UInt32 Arrays and the >>> operator.
They could be transformed by the JIT compiler into doubles and slow everything down. This does not happen with the Int32 Arrays and the >> operator.
There is a plug-in called JIT-Inspector which give you this information. Unfortunately it does no longer work in Firefox 24. Firefox 22 was the last working version.
Additionally worker threads gave me an incredible speed boost in Firefox. The whole GUI stuff is then separated. Especially the costly canvas update.
Do you have any documentation to support the stuff about UInt32? The old Firefox JIT had some limitations like this but I'm not aware of anything like that in the current one (which has been shipping for multiple versions). Is it a v8 limitation similar to the 31-bit 'small integer' limitation?
Indeed at the moment I have nothing to support it. Half a year ago I had dozens of examples. Now it is more or less experience. In v8 I had two weeks ago some strange deoptimizations behavior in my code. It was almost gone after I exchanged two >>> operations. So in my experience it is still better to avoid such cases. No one knows really how the JIT compiler are working. What makes it worse. The behavior is changing every 6 weeks. And don't forget: Out there is more than Firefox and Chrome ;)
Have you consider reporting strange deoptimization behaviour to V8 team? Performance issues are better investigated than worked around.
V8 has special support for uint32 values in optimized frames to avoid converting them to doubles when they flow into truncating operations. If that support somehow became broken that would be unfortunate. (and it's actually only >>> 0 that is problematic from the optimization point of view. as long as you shift by non zero result is within Int32 range).
The problem is that I can't really explain it. I can hardly say: Hey, if I change this line (from 3000 lines) from A to B I get a 80% reduced chance that my program is deoptimized after 30 seconds and slows down by a factor of five.
No one will take the time to look at it.
What I want is information. They have to provide a tool that let me see what happens. Especially if a variable or operation is treated as double or int. The JIT-inspector for Firefox (is)/was a great tool for this. Unfortunately it does not support worker threads.
No, I think you are making a wrong assumption here. A bug report like that, that pinpoints a line in question and provides observable metrics is the best kind of performance issue that you can file against V8.
Back when I was on the V8 team I would immediately set aside some time to investigate and triage/fix/explain it because stable performance is as important for any VM as correctness.
So don't hesitate to file bugs like that against JavaScript VMs be it V8 or SpiderMonkey. You might be surprised with VM engineers' eagerness to investigate and address performance problems.
> What I want is information.
There are ways to trace optimizations and deoptimizations in V8. Unfortunately nothing as simple to use as JIT-inspector. Most of what V8 has right now is geared towards VM engineers.
I have a tool called IRHydra[1] that can display some information that V8 dumps in a more or less digestible form. It's most useful for performance investigations in the console but to a certain degree works for browser as well (biggest problem for it "multithreading": because V8 dumps part of the information into stdout and as a result it can arrive arbitrary mixed if you have many workers).
Hi s-macke. I've read the source code of your emulator before, good job.
I'm aware of the problem with big integers. I'll look into this more closely, thanks for your help.
Regarding workers: I need some restructuring to get this working. I'll also need to figure out the fastest way to exchange the canvas buffer between the worker and the browser.
I did the copying by a for loop copying In32 Arrays which is an acceptable solution. It slows down the worker thread by 1-2% by using 10fps and a screen size of 640x400x32. I think the reason why I didn't used more advanced ArrayBuffer functions to do this was the IE. Probably this has changed.
The color transformations and so on can be done in the master GUI-thread.
There are two sensible approaches: interpretation or dynamic translation.
If you want to write one as a learning experience for how the platform you're emulating works and how an emulator could work in principle, you probably want to write an interpreter: find out how the platform boots, build something to start reading and interpreting instructions, start adding emulated hardware devices, and you'll get an increasingly functional interpreter.
For a start, you could ignore emulation of devices, and just write a CPU core that interprets instructions.
If you want to learn how modern (but still non-hardware-assisted) emulation works, read about how qemu's dynamic code translator and other JIT compilers work, and build a CPU emulator based on that.
I'm not the OP but check out the book: "The Elements of Computing Systems: Building a Modern Computer from First Principles" It's quite comprehensive, not exactly about writing an emulator, but it should give you what you need.
I really like your tutorial. I been trying to learn assembly and your tutorial is really helping me put the big picture together. I hope you continue to work on making it better.
193 comments
[ 307 ms ] story [ 3025 ms ] thread[EDIT] - This post doesn't seem to say enough when I look back at it. Wanted to add this:
Seeing posts like this really excites me (and inspires me) about the future of web programming, and programming in general. Anyone that's excited about programming has to get excited about abstraction, and it doesn't get much more abstract that cross-coded/implemented virtualized systems like this. Even if you hate javascript.
Every step people take in blurring lines between systems like this should be exciting, given the large amount of abstraction that had to go into creating something like this.
That's not meant to take away from this project - it truly is amazing what can be interpreted in the browser now.
Whilst this is nothing like the several orders of magnitude required for more capable interactive behaviour, it does make my load times noticeably faster than those claimed on the page. :)
(I'm Chrome on Linux, i7 4th Gen, incidentally)
(This value was the average while booting Linux)
8 - 23 MIPS ain't so bad, right?
[1] http://en.wikipedia.org/wiki/Motorola_68030
Why? I'm honestly curious since I can't imagine any big advantages over more "classic" setups ...
It is a "social" development platform supporting rails, mysql etc.
If you're interested in this, please do make plenty of noise at me to finish. I've got a flight back to the East coast from Portland on Sunday, so I'm hoping I can get most of the browser code finished by then. After that I'm going to be somewhat distracted through the end of January, as I'm moving from Charleston, SC to Christchurch, New Zealand. I could use a bit of motivation to get this done.
1: http://s-macke.github.io/jor1k/
Otherwise I think it'd be great to get an image together with all of those tools, especially Lynx. It'd be great if I could make the HN submission to the demo page via the emulator itself. However for now if I can get an ICMP echo I'll be happy.
I think when it's ready I'm going to set up a full-access reCaptcha-protected demo on DigitalOcean or AWS and see what happens. If people don't abuse it and if the bill isn't too bad, I'll leave it up. If not, I'll worry about how to do a restricted demo. It would be nice to get a network-friendly game in there, though.
Another great demo that we'd have difficulty with is browser-in-browser. Chrome-in-chrome, firefox-in-firefox. That would, however, be the ultimate sandbox.
Anyway, get in touch with me via e-mail or better yet, Skype (see profile). We can coordinate better there.
Ruby's interestingness to performance ratio is not high.
http://www.zombo.com
Also, this isn't just an x86 emulator, right? If it can boot linux, it's emulating the entire PC platform, not just x86. Very impressive of course. But I am interested in a comparison vs jslinux (also not open source).
I had a quick look at the minified source. its pretty hard to decipher (method names have been renamed too), unlike traditional minification which you can recover by just reformatting.
Otherwise yeah, you can just run it through a whitespace-remover. There are also often options on e.g. the YUI compressor: http://refresh-sf.com/yui/ and Google's: http://closure-compiler.appspot.com/home
First I tap to the edit box on the right, the keyboard rolls in, then I tap in the "console." The commands work. I just don't know how I'd type ctrl-Keys without the physical keyboard.
Copy.sh ends with unimplemented GP handler however for anything I've tried.
#) JSLinux: the PC emulator is written in Javascript with the emulated hardware: 32 bit x86 compatible CPU, 8259 Programmble Interrupt Controller, 8254 Programmble Interrupt Timer, 16450 UART, Real Time Clock, IDE interface and hard disk. http://bellard.org/jslinux/tech.html
For the software part, why would you advocate GPLv3+ over permissive licenses like MIT/BSD/Apache?
... Since you're using seabios, I guess its bringup is relatively close to qemu's?
You might want to consider re-ordering the font list in the console though. Consolas isn't perfectly fixed-width apparently, and it made Rogue rather puzzling to play.
I assumed something named consolas would be a font suitable for consoles.
Which font do you like to use?
I'm on Linux, so that might also impact something.
Perhaps the line drawing characters are extra-small or something? Fire up Rogue and try it out...
ReferenceError: requestAnimationFrame is not defined http://copy.sh/v24/v86_all.js Line: 239
EDIT: I thought FF24 wasn't working, turns out I was confused between my Linux desktop's and Mac laptop's FF versions. Sorry!
Firefox 4 added mozRequestAnimationFrame.
Firefox 23 added the unprefixed version.
It's documented in the browser compatibility table at https://developer.mozilla.org/en-US/docs/Web/API/window.requ... (note that "moz" next to the 4.0).
http://freebsd.1045724.n5.nabble.com/Problem-with-FireFox-24...
this is great work. would love to see source un-minified.
2. Ship a COPYING file, add a header to your files: https://github.com/osterman/copyright-header
Interesting bug: I can't seem to type '-', '=', '+', or '_' into the Linux image. Missing '-' in particular makes it hard to run commands with options.
Elaborate workaround:
The first command grabs an '=' from dmesg and sets "eq" to it (without typing '='), and then the second command grabs a '-' from the usage message of uname and sets "dash" to that (without typing '-'). The last shows how to use ${dash} in a command.Build a skyscraper, and lock the keys inside for the ribbon cutting. Sorry pal you lost your funding
Land on Mars, see an alien, but your camera is out of batteries and everybody back on Earth thinks you're a quack.
Successfully perform open-heart surgery on a desert island, but muck up the stitches and cause a scar. Your patient will hold it against you the rest of his life (which you saved).
Simple defects where a key is being ignored on certain systems are usually quite easy to fix ;)
Here's a soul-crushing situation that doesn't have a workaround, as far as I can tell.
You can play the original Deus Ex on OnLive. By default, it sets the fov to a very low number. It's only possible to change it in the console (there is no UI for it). But the console is disabled. However, you can get around that by pressing T, which creates a mini console with "say " prepoulated. Press delete to erase say, and type "fov 90". So close...
But they disabled the Enter key from working.
So you can get full access to the console, type the command that would fix the FOV, but you're unable to execute it because Enter key does not get sent to the game. You're completely helpless.
OnLive Client <-all input is transmitted-> OnLive Servers <-filter out Enter-> Deus Ex instance in a VM.
The enter key is filtered out on their servers. Unless you hack into their servers and change that, there is nothing you can do on the client-side. That's what makes you completely helpless.
Also, any plans to implement GPF handling? I tried booting a test environment I have on it (which runs Python as part of the GRUB2 bootloader), but it failed primarily due to lack of support for trapping GPFs. (There are a few other issues I'd have to work around to make it work, but that's a major one that I can't easily work around.)
Regarding GPFs: This is one of the few things that are not completely implemented yet, I'm working on figuring this out.
eg your $eq code can be written like this:
I'm considering writing some articles, just need to find some free time.
The trick is in writing a fast and accurate one.
Also, swapping disks during the install might be an issue, I don't see a button...
Your emulator is impressive and especially fast. I think it took a long time for you to optimize it. I tried to start TinyCore, but it stops after decompressing the kernel. So still some work to do ;)
V8 has special support for uint32 values in optimized frames to avoid converting them to doubles when they flow into truncating operations. If that support somehow became broken that would be unfortunate. (and it's actually only >>> 0 that is problematic from the optimization point of view. as long as you shift by non zero result is within Int32 range).
What I want is information. They have to provide a tool that let me see what happens. Especially if a variable or operation is treated as double or int. The JIT-inspector for Firefox (is)/was a great tool for this. Unfortunately it does not support worker threads.
And yes, I think it was a ">>> 0" operation.
No, I think you are making a wrong assumption here. A bug report like that, that pinpoints a line in question and provides observable metrics is the best kind of performance issue that you can file against V8.
Back when I was on the V8 team I would immediately set aside some time to investigate and triage/fix/explain it because stable performance is as important for any VM as correctness.
So don't hesitate to file bugs like that against JavaScript VMs be it V8 or SpiderMonkey. You might be surprised with VM engineers' eagerness to investigate and address performance problems.
> What I want is information.
There are ways to trace optimizations and deoptimizations in V8. Unfortunately nothing as simple to use as JIT-inspector. Most of what V8 has right now is geared towards VM engineers.
I have a tool called IRHydra[1] that can display some information that V8 dumps in a more or less digestible form. It's most useful for performance investigations in the console but to a certain degree works for browser as well (biggest problem for it "multithreading": because V8 dumps part of the information into stdout and as a result it can arrive arbitrary mixed if you have many workers).
[1] http://web.irhydra.googlecode.com/git/hydra.html
I'm aware of the problem with big integers. I'll look into this more closely, thanks for your help.
Regarding workers: I need some restructuring to get this working. I'll also need to figure out the fastest way to exchange the canvas buffer between the worker and the browser.
If you want to write one as a learning experience for how the platform you're emulating works and how an emulator could work in principle, you probably want to write an interpreter: find out how the platform boots, build something to start reading and interpreting instructions, start adding emulated hardware devices, and you'll get an increasingly functional interpreter.
For a start, you could ignore emulation of devices, and just write a CPU core that interprets instructions.
If you want to learn how modern (but still non-hardware-assisted) emulation works, read about how qemu's dynamic code translator and other JIT compilers work, and build a CPU emulator based on that.
http://blog.alexanderdickson.com/javascript-nes-emulator-par...
Part 2 will be posted when it's done. :)
If you want to get something done much quicker, have a go at emulating the Chip-8 VM.
I also have a post, but it's less hand-holding than the first.
http://blog.alexanderdickson.com/javascript-chip-8-emulator
Good luck!