You have mainly two options: do everything from scratch or reuse. Starting from scratch is almost impossible to do alone, just Chrome was developed by different teams with different expertises, some of them acquired from other companies.
The second option is reusing a web component or browser and modify it for your purposes. Building on top of Chrome or Firefox (or their variants) is a good choice. I think Chrome is better engineered than Firefox and the code is clear. Obviously this is just an opinion.
Now the question is what do you want to add to such browser? Is this something you can add via an extension or is something that requires a new direction?
>Chrome was developed by different teams with different expertises, some of them acquired from other companies.
To expand on this, Chrome and Safari use webkit as their rendering/layout engine which is a fork of KDE's KHTML. IE came from NCSA Mosaic-> Spyglass -> Microsoft IE.
To be clear on my suggestions below, let's clarify what a browser actually is. The browser manages things like the navigation bar, bookmarks, download/mime handling, etc. This is known as the "chrome" or GUI. The layout engine handles HTML parsing, rendering, possibly is coupled with a Javascript engine like V8.
If you are trying to write a browser, the answer is to use webkit2 in whatever language you like. It's fast, light, and portable.
If you are trying to write a layout engine (rendering engine), do as another poster suggested and write one to target HTML 3.2.
Or a good study if the the goal language is different. Studying a clean codebase is where I'd start, there is likely a lot of tribal knowledge in HTML rendering (especially layout) that isn't documented anywhere but in code.
If you are seeking tutorials, or UI libraries etc, sorry, I can't help you with that as I am unaware of those things. However, if you are asking from which aspect to start building one from scratch, I would say start with layouting. Try implementing a tiny subset of div related things like position and display and see if this is what you want to spend your time working on. Even if you can crack this thing easily, remember, this is just the tip of the iceberg. I am not discouraging you, but know that you would have to digest the entire DOM and HTML spec, not to mention have a JavaScript engine and implement all the security related intricacies, at the least, for your software to become a web browser.
I'm curious: is there a ready-to-use set of tests for a layout engine that checks conformance with the specs? Something that you could, without much effort, run against a toy layout engine you'd write?
I know Webkit/Blink have humongous test suites, but these probably aren't easy to adapt to a toy layout engine.
There are also all those other browser benchmarks (e.g. http://peacekeeper.futuremark.com/), but those include things other than the layout engine, such as javascript performance, webGL, etc. You could try poking around a few of them and only looking at the CSS and DOM sections though (http://web.basemark.com/).
Equally - the car we drive today is a result of 130 years of human work and innovation. But after all that time we're seeing a real shake with the electric car. Things change when people try. I say good luck to the girl/guy - perhaps they'll create something incredible :-)
Even the electric car didn't start from scratch - it'll use the same component manufacturers as the rest of the car industry - e.g. tyres, brakes, suspension, interior, safety systems.
As somebody already wrote "Starting from scratch is almost impossible to do alone". However you could set a first goal of doing some basic rendering of plain HTML, even text based. That should make you familiar with the hurdles of a rendering engine. Then apply some CSS. Even writing a browser for HTML 3.2 from scratch and no JavaScript should be an exercise to be proud of https://www.w3.org/TR/REC-html32 (1997). Then start looking at modern browsers using the links others provided. You should have the experience to understand all the moving parts by then.
This is similar to writing the very first web application without using any framework, maybe even interfacing directly with the web server. I remember decoding CGI arguments in C in 1994 before discovering Perl's CGI.pm. Then you understand the magic inside frameworks and don't get burned by surprises.
I'm working on one right now. I'm using the electron shell because im not worried about the backend source (chromium) I'm just working on a new ui and a reader feature.
Email at matthew349hall@hotmail.com if you want to get a beta
1) Write socket code to retrieve the page from a server
2) Get it to draw the text in a window.
3) Write code to look for tags and start diving up the retrieved file into a proper DOM tree. Perhaps start by creating an object for each paragraph. Then maybe look for <b> tags, then <div> tags
4) Change the rendering code to draw the screen for the data structure you created above.
5) Iterate a lot. Add support for parsing more and more tags into the data model. Start adding annotations for various attributes.
6) Start improving the layout code. Draw things in <b> tags in bold. Draw things in <div> tags under each other and <span> tags separately.
7) Add limited CSS support. Allow just width and height and border sections. make the code read the css file and attach the attributes to the correct sections of the DOM document you are creating.
8) Improve the layout code to look at the width and height tags on each element, and where tags are nested to propogate the width and height information up and down the tree as needed. Draw borders if the css tells you to. Look at font weight etc.
9) Iterate. Add one feature at a time. Repeat for a year until you have an browser that can render basic pages.
10) Iterate some more. Read specs. Rewrite your document to DOM tree parse a number of times perhaps using a more formal grammar. You'll probably be on about year 8 by now.
11) Add javascript support... :P
Perhaps my point is though that this seems something where its very easy to start small, just render the text of a page, and incrementally add features and improve the layout. It will take a long time though as there are so many features to add.
The above list is a good start. I'd add: 2a) make clicking on links work, and 3a) display images. Those two features will make it much more fun.
"How to make a browser" is something I've thought about for a while, ever since someone suggested it as an interview question. (I think it would be an awful question.) It really depends on what you're trying to do. If you want to understand how browsers work and build a toy browser, follow jbb555's list. If you want to display web pages, use WebKit. If you want to build a real, commercial-scale browser, I recommend a large team, since there are a lot of components, each of which is insanely complex. (Everything from CSS support to security to plugins to JavaScript to bookmarks to a debugger to all the new network protocols.)
One more thing to add to jbb555's list: 0) "telnet news.ycombinator.com 80, GET /" - playing around from the command line can give you a good introduction to what really happens when you access a web page. (Edit: HN returns an error page for a plain GET. google.com returns insane code. apple.com looks like a better place to start.)
Is it still even doable for a single person to write a new rendering engine from scratch? I'm getting the impression that right now it's all about Webkit, Gecko and IE and given how complex things got, I imagine that it would cost an enormous sum of money to write something that, say, can display top 10 Alexa web pages. Things just got too complex.
Perhaps it would be better to have some effort in documenting Webkit code so that more hackers could actually read it and hack on it?
It shouldn't be too much for a single person to take on if they're truly dedicated to the task. A small team is still probably more ideal, but the browser isn't overly complex, it simply has a lot of capability that it can pull off with its feature set. All of a browsers moving parts are designed to be quite generic, which allows quite complex interactions to arise out of a small handful of simpler components.
Things used to be a lot harder 5 years ago, but the renewed focus on standards means that many smaller browsers can focus on just that, and ignore a lot of the weird quirks found in major browsers safely. (The death of Internet Explorer is finally bringing about the necessary changes in websites that have clung to its quirks mode so blindly for so many years.)
I think starting with a manually written parser is a necessary first step if you're writing a web browser. It's tempting to want to pull in Webkit or V8 or Gecko or Chakra, but if you're writing a web browser, you need to understand what these tools are actually doing for you before you use them. Writing your own versions, even if that's time consuming, will teach you why these other frameworks have become so popular, and educate you about their shortcomings so you can hopefully code around those too. If you do it write, and your codebase is appropriately modular, it should be straightforward to modify the project down the road to use a different parser, or even an entirely different DOM tree implementation.
- Write the networking code to retrieve data from servers. I'd start with GET requests to keep it simple. This part is not that bad and its actually pretty fun. Doing it in Python is very simple and straightforward.
- Write browser GUI because its a native application. This depends on what language you use for the task. If Python, you can get away with tkinter (flame suit on) for a very simple prototype. I believe tkinter is single threaded and that may turn out to be an issue (don't remember).
- Write a simple rendering engine to display the text. No need to do this inside of the GUI code because the output can be read through the console. You are pretty much reading the raw html and extracting the bits you want.
- Plug in the rendering engine into a GUI widget. Since this is a (assumed) read-only browser, you can get away with a simple textbox of sorts. Anything else will require a custom widget.
At this point you have a very minimal browser. Adding CSS, javascript, etc requires that you include a pipeline to route data through the proper channels before it reaches the GUI. This is where the bulk of the work goes. Bring beer. :)
You can accept entered URLs and fetch & display web pages quite easily nowadays. You should, depending on the vertical, be very concerned about security.
What are your goals (to a greater degree than "web browswer")?
[Disclaimer: I'm not an expert on developing browsers] It depends on what exactly you'd want to implement, since a (modern) browser is really a huge piece of software with several aspects and components in it. Looking at it from one angle, you could split it into chrome (not Google Chrome, but the UI, like tab bar, bookmarks, menus, extensions management, etc.) and content (the rendering engine).
You'd most likely get bogged down if you start writing your own rendering engine (processing HTML, CSS and JS, and processing image formats, video and sound formats, even not including any plugins), unless it's a very limited scope toy browser that can handle some simple stuff. In this case, start with base HTML support, including handling incomplete tags and quirks like that have been carried over for decades (the amount of cruft in rendering engines for backward compatibility and handling undocumented but accepted quirks would itself take a long time to develop). Then try a little bit of CSS support. I'd assume JS support could be quite challenging since that moves out of the already challenging "paint the screen" part into an even more challenging part on interpreting code, running it, optimizing it, etc., which is a lot more complex when you consider there would be several JS files loaded on most websites.
Something a little more easier (only comparatively) would be developing the chrome around an existing rendering engine and seeing how to handle tabs, background loading, foreground loading, bookmarks, private/incognito browsing, etc., and if you feel adventurous enough, adopt a particular process or threading model for a multi-tab browser.
Another alternative here is to create a WebView application and see which parts you could improve. For example, Chrome on iOS and Firefox on iOS are just the chrome for the built-in rendering engine in iOS, but address many things like tabs, networking, incognito mode, bookmarks, sync and other features. You could try creating a similar one on iOS, latching on to the work done by FoxBrowser [1] or Firefox for iOS [2]. On Android, you could try building on top of Firefox for Android [3].
If you'd like to work on a web browser rendering engine that's a bit more closer to "from scratch", I'd recommend working with the Servo [4] team and seeing how you can help. It's a long drawn project that has already made huge strides, but you'd get an inside view of the complexity of developing a browser engine. What's more, you could also learn Rust while doing this. [5]
Do you want to build a web browser or a web browser engine?
If you just want to build a web browser, here's my request for a new browser for iOS since web browsing on iOS is a second or third rate experience:
- Built-in ad-blocking.
- A way to disable JavaScript immediately for any website.
- A desktop mode that actually works, so that YouTube, Reddit, Imgur and other sites don't redirect me to their crappy mobile site without asking.
- Launches a completely separate app for private browsing.
Right now I use Dolphin browser on iOS and despite being in desktop mode, I still have to refresh every YouTube page that I land on in order to get the desktop style native HTML5 video player that allows me to go full-screen. On Imgur, gifv files don't play - I have to change the extension to gif and then wait 60 seconds as it downloads. Lots of other sites have similar types of issues. Meanwhile, gifv files work fine in Brendan Eich's "Brave" browser. It could be as easy to fix as just changing the user-agent.
Dolphin has good ad-blocking, but there is no way to quickly turn off JavaScript for a given page or domain (ala Quick JavaScript Switcher for Chrome).
The thing that really annoys me though about Dolphin is that Private Browsing does not work at all. That's probably because they want you to buy Dolphin Zero for $2.99.
Brave has a desktop mode that mostly works and a built-in adblocker.
It does not have a `quick JS disable` button.
It also doesn't have a speed dial that I can manually add items to, which is one very important item that I forgot to mention earlier.
What I really want to do is make an open source browser that I can compile and put on my iOS devices myself too, without going to the app store. Then I could add all sorts of features and extensions without having to ask Apple.
For a web browser? You really want to pay 3 dollars for a web browser?
Also - Are you really surprised that people balk at paying for things that were once free? Really????
Furthermore, Dolphin has a "Private Mode" and it does not work at all. It's a lie. They are clearly trying to steer you towards their Zero browser. I don't give 3 dollars to companies who try to play me like that.
Nope. No, no way. Uh-uh. Forget it. For now, I just use Brave for private browsing and Dolphin for public browsing.
I actually want an open source one and I'm thinking about building it myself too. Looks like Brave is open source AND it's built with Swift which is exactly what I was going to do. I can just take that and rip out all the crap and add what I want.
I did this once back in 2003. It was text only rendered, with mouse support so you could click on the links instead of tabbing around like Links.
I built it off of the IE object model (Internet Explorer Browser Extensions? Can’t recall). I think I just used the object model for connectivity settings. If you could get to the net using IE, my browser would work as well.
I think I coded to HTML 1.1 specs, and had it working after 2 months (with a few minor rendering issues). It did not support scripting, CSS or really much of anything (although it could POST), but it was surprising how much of the web was accessible. And it was really fast. Not because of my skillz, but when you cut out the garbage, it’s amazing how fast a page will load.
Anyway, I would not recommend this approach, but I would offer you encouragement. HTML parsing isn’t really that big of a deal, and you could get something very minimal (text rendering) working in short order. It would at least give you a feeling of what you are up against.
Well, this is what I could do in that (or almost any) case:
1) Setting features, key points, points to avoid, and goals of the project.
2) Choosing the toolchain to use, for resolving the points at 1)
3) Coding the features with that toolchain
Step 2, may involve: to read a lot (doc, rfc's, specifications, using search engines with keywords, search for "problems", etc) to look at other people's source code of similar programs, to try other people's tools and libraries and evaluate and compare them, to make integration tests, and to make Proofs Of Concept prototypes.
The toolchain (programing language, user GUI, etc) comes with an ecosystem (I hope you don't aim at rewriting also your $lang libraries for validate XML or interact with devices), so you have to choose looking at both.
To separate steps 2 and 3, use iterations, while you have resources.
If you find an opensource product that full-fills all your definitions of point 1), except a few of them, you can evaluate to start by a patch and a pull-request to them, or a fork :P
I'd say, if you have the opportunity and want to go wild skip the DOM and create an equivalent model using OpenGL. All the other Web platform features like the distribution model, resource streaming etc can still be used. It's just that the DOM is the great bottleneck and most of the hours devoted to improve the Web are being wasted on sorting quirks related to the DOM. Maybe yours will be the starting point towards a DOM-free Web! :)
55 comments
[ 2.5 ms ] story [ 39.4 ms ] threadThe second option is reusing a web component or browser and modify it for your purposes. Building on top of Chrome or Firefox (or their variants) is a good choice. I think Chrome is better engineered than Firefox and the code is clear. Obviously this is just an opinion.
Now the question is what do you want to add to such browser? Is this something you can add via an extension or is something that requires a new direction?
To expand on this, Chrome and Safari use webkit as their rendering/layout engine which is a fork of KDE's KHTML. IE came from NCSA Mosaic-> Spyglass -> Microsoft IE.
To be clear on my suggestions below, let's clarify what a browser actually is. The browser manages things like the navigation bar, bookmarks, download/mime handling, etc. This is known as the "chrome" or GUI. The layout engine handles HTML parsing, rendering, possibly is coupled with a Javascript engine like V8.
If you are trying to write a browser, the answer is to use webkit2 in whatever language you like. It's fast, light, and portable. If you are trying to write a layout engine (rendering engine), do as another poster suggested and write one to target HTML 3.2.
I know Webkit/Blink have humongous test suites, but these probably aren't easy to adapt to a toy layout engine.
There are also all those other browser benchmarks (e.g. http://peacekeeper.futuremark.com/), but those include things other than the layout engine, such as javascript performance, webGL, etc. You could try poking around a few of them and only looking at the CSS and DOM sections though (http://web.basemark.com/).
http://testthewebforward.org/
Start out with a solid base: (for example)
- https://cefbuilds.com/
- https://crosswalk-project.org/
- http://electron.atom.io/
- http://nwjs.io/
or you try to help with existing projects: (for example)
- https://github.com/breach/thrust (started out with https://github.com/breach/breach_core a "javascript"-Browser (core is not js))
https://github.com/minbrowser/min
easy bugs to hunt: https://github.com/servo/servo/labels/E-easy
[1] https://www.udacity.com/course/programming-languages--cs262
How to Create your Own Internet Browser in 5 Minutes! - https://www.youtube.com/watch?v=Bn22JBRC1UI
Chrome's Engine http://www.chromium.org/blink
This is similar to writing the very first web application without using any framework, maybe even interfacing directly with the web server. I remember decoding CGI arguments in C in 1994 before discovering Perl's CGI.pm. Then you understand the magic inside frameworks and don't get burned by surprises.
The HTML-parser is probably the part it makes sense to do first.
What language and platform do you plan to use? I think a modern OO-language will be helpful.
1) Write socket code to retrieve the page from a server
2) Get it to draw the text in a window.
3) Write code to look for tags and start diving up the retrieved file into a proper DOM tree. Perhaps start by creating an object for each paragraph. Then maybe look for <b> tags, then <div> tags
4) Change the rendering code to draw the screen for the data structure you created above.
5) Iterate a lot. Add support for parsing more and more tags into the data model. Start adding annotations for various attributes.
6) Start improving the layout code. Draw things in <b> tags in bold. Draw things in <div> tags under each other and <span> tags separately.
7) Add limited CSS support. Allow just width and height and border sections. make the code read the css file and attach the attributes to the correct sections of the DOM document you are creating.
8) Improve the layout code to look at the width and height tags on each element, and where tags are nested to propogate the width and height information up and down the tree as needed. Draw borders if the css tells you to. Look at font weight etc.
9) Iterate. Add one feature at a time. Repeat for a year until you have an browser that can render basic pages.
10) Iterate some more. Read specs. Rewrite your document to DOM tree parse a number of times perhaps using a more formal grammar. You'll probably be on about year 8 by now.
11) Add javascript support... :P
Perhaps my point is though that this seems something where its very easy to start small, just render the text of a page, and incrementally add features and improve the layout. It will take a long time though as there are so many features to add.
"How to make a browser" is something I've thought about for a while, ever since someone suggested it as an interview question. (I think it would be an awful question.) It really depends on what you're trying to do. If you want to understand how browsers work and build a toy browser, follow jbb555's list. If you want to display web pages, use WebKit. If you want to build a real, commercial-scale browser, I recommend a large team, since there are a lot of components, each of which is insanely complex. (Everything from CSS support to security to plugins to JavaScript to bookmarks to a debugger to all the new network protocols.)
One more thing to add to jbb555's list: 0) "telnet news.ycombinator.com 80, GET /" - playing around from the command line can give you a good introduction to what really happens when you access a web page. (Edit: HN returns an error page for a plain GET. google.com returns insane code. apple.com looks like a better place to start.)
Perhaps it would be better to have some effort in documenting Webkit code so that more hackers could actually read it and hack on it?
Things used to be a lot harder 5 years ago, but the renewed focus on standards means that many smaller browsers can focus on just that, and ignore a lot of the weird quirks found in major browsers safely. (The death of Internet Explorer is finally bringing about the necessary changes in websites that have clung to its quirks mode so blindly for so many years.)
I think starting with a manually written parser is a necessary first step if you're writing a web browser. It's tempting to want to pull in Webkit or V8 or Gecko or Chakra, but if you're writing a web browser, you need to understand what these tools are actually doing for you before you use them. Writing your own versions, even if that's time consuming, will teach you why these other frameworks have become so popular, and educate you about their shortcomings so you can hopefully code around those too. If you do it write, and your codebase is appropriately modular, it should be straightforward to modify the project down the road to use a different parser, or even an entirely different DOM tree implementation.
- Write the networking code to retrieve data from servers. I'd start with GET requests to keep it simple. This part is not that bad and its actually pretty fun. Doing it in Python is very simple and straightforward.
- Write browser GUI because its a native application. This depends on what language you use for the task. If Python, you can get away with tkinter (flame suit on) for a very simple prototype. I believe tkinter is single threaded and that may turn out to be an issue (don't remember).
- Write a simple rendering engine to display the text. No need to do this inside of the GUI code because the output can be read through the console. You are pretty much reading the raw html and extracting the bits you want.
- Plug in the rendering engine into a GUI widget. Since this is a (assumed) read-only browser, you can get away with a simple textbox of sorts. Anything else will require a custom widget.
At this point you have a very minimal browser. Adding CSS, javascript, etc requires that you include a pipeline to route data through the proper channels before it reaches the GUI. This is where the bulk of the work goes. Bring beer. :)
What are your goals (to a greater degree than "web browswer")?
You'd most likely get bogged down if you start writing your own rendering engine (processing HTML, CSS and JS, and processing image formats, video and sound formats, even not including any plugins), unless it's a very limited scope toy browser that can handle some simple stuff. In this case, start with base HTML support, including handling incomplete tags and quirks like that have been carried over for decades (the amount of cruft in rendering engines for backward compatibility and handling undocumented but accepted quirks would itself take a long time to develop). Then try a little bit of CSS support. I'd assume JS support could be quite challenging since that moves out of the already challenging "paint the screen" part into an even more challenging part on interpreting code, running it, optimizing it, etc., which is a lot more complex when you consider there would be several JS files loaded on most websites.
Something a little more easier (only comparatively) would be developing the chrome around an existing rendering engine and seeing how to handle tabs, background loading, foreground loading, bookmarks, private/incognito browsing, etc., and if you feel adventurous enough, adopt a particular process or threading model for a multi-tab browser.
Another alternative here is to create a WebView application and see which parts you could improve. For example, Chrome on iOS and Firefox on iOS are just the chrome for the built-in rendering engine in iOS, but address many things like tabs, networking, incognito mode, bookmarks, sync and other features. You could try creating a similar one on iOS, latching on to the work done by FoxBrowser [1] or Firefox for iOS [2]. On Android, you could try building on top of Firefox for Android [3].
If you'd like to work on a web browser rendering engine that's a bit more closer to "from scratch", I'd recommend working with the Servo [4] team and seeing how you can help. It's a long drawn project that has already made huge strides, but you'd get an inside view of the complexity of developing a browser engine. What's more, you could also learn Rust while doing this. [5]
[1]: https://github.com/graetzer/Foxbrowser
[2]: https://github.com/mozilla/firefox-ios
[3]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_g...
[4]: https://servo.org/
[5]: https://www.rust-lang.org/
If you just want to build a web browser, here's my request for a new browser for iOS since web browsing on iOS is a second or third rate experience:
- Built-in ad-blocking.
- A way to disable JavaScript immediately for any website.
- A desktop mode that actually works, so that YouTube, Reddit, Imgur and other sites don't redirect me to their crappy mobile site without asking.
- Launches a completely separate app for private browsing.
Right now I use Dolphin browser on iOS and despite being in desktop mode, I still have to refresh every YouTube page that I land on in order to get the desktop style native HTML5 video player that allows me to go full-screen. On Imgur, gifv files don't play - I have to change the extension to gif and then wait 60 seconds as it downloads. Lots of other sites have similar types of issues. Meanwhile, gifv files work fine in Brendan Eich's "Brave" browser. It could be as easy to fix as just changing the user-agent.
Dolphin has good ad-blocking, but there is no way to quickly turn off JavaScript for a given page or domain (ala Quick JavaScript Switcher for Chrome).
The thing that really annoys me though about Dolphin is that Private Browsing does not work at all. That's probably because they want you to buy Dolphin Zero for $2.99.
It does not have a `quick JS disable` button.
It also doesn't have a speed dial that I can manually add items to, which is one very important item that I forgot to mention earlier.
What I really want to do is make an open source browser that I can compile and put on my iOS devices myself too, without going to the app store. Then I could add all sorts of features and extensions without having to ask Apple.
Also - Are you really surprised that people balk at paying for things that were once free? Really????
Furthermore, Dolphin has a "Private Mode" and it does not work at all. It's a lie. They are clearly trying to steer you towards their Zero browser. I don't give 3 dollars to companies who try to play me like that.
Nope. No, no way. Uh-uh. Forget it. For now, I just use Brave for private browsing and Dolphin for public browsing.
I actually want an open source one and I'm thinking about building it myself too. Looks like Brave is open source AND it's built with Swift which is exactly what I was going to do. I can just take that and rip out all the crap and add what I want.
I built it off of the IE object model (Internet Explorer Browser Extensions? Can’t recall). I think I just used the object model for connectivity settings. If you could get to the net using IE, my browser would work as well.
I think I coded to HTML 1.1 specs, and had it working after 2 months (with a few minor rendering issues). It did not support scripting, CSS or really much of anything (although it could POST), but it was surprising how much of the web was accessible. And it was really fast. Not because of my skillz, but when you cut out the garbage, it’s amazing how fast a page will load.
Anyway, I would not recommend this approach, but I would offer you encouragement. HTML parsing isn’t really that big of a deal, and you could get something very minimal (text rendering) working in short order. It would at least give you a feeling of what you are up against.
1) Setting features, key points, points to avoid, and goals of the project.
2) Choosing the toolchain to use, for resolving the points at 1)
3) Coding the features with that toolchain
Step 2, may involve: to read a lot (doc, rfc's, specifications, using search engines with keywords, search for "problems", etc) to look at other people's source code of similar programs, to try other people's tools and libraries and evaluate and compare them, to make integration tests, and to make Proofs Of Concept prototypes.
The toolchain (programing language, user GUI, etc) comes with an ecosystem (I hope you don't aim at rewriting also your $lang libraries for validate XML or interact with devices), so you have to choose looking at both.
To separate steps 2 and 3, use iterations, while you have resources.
If you find an opensource product that full-fills all your definitions of point 1), except a few of them, you can evaluate to start by a patch and a pull-request to them, or a fork :P