It looks like it doesn't, so existing open source golang url routers will perform much better, contrary to this post. The one in vuland is interesting in that it supports both regex AND tries, so you get the best of both.
Additionally, httprouter (golang) uses a trie and is fast as hell:
These links are pretty cool. Are there any tutorials or walkthroughs you would recommend? Like, Go has always interested me but I have no idea where to start to get an experience like say ExpressJS or even where to find templating or rendering.
When I was learning Go, Negroni was where I left off and felt like what you may be looking for as a next step: https://github.com/urfave/negroni (I"m sure someone more experienced can provide more context or other / better links -- thought this would be a better answer than none until then)
With go there is really less emphasis on taking advantage of a web framework like expressjs (look at go alternatives like gin) and building server functionality from the std library (since everything you need is there). I wrote a small helper library to help with template rendering if you're interested in looking at the source to see how you can take advantage of the std library for templating: https://github.com/dannav/migo
I have written a very basic introduction to writing webapps in Go without using a framework, as others have rightly pointed out, we really do not need to use a framework in Go for writing webapps, and Go is a great language.
httprouter is really awesome. I switched to it when Go's built-in router would cause panics during msan stress testing (back in Go 1.6), and really performs well.
Regexp definitely isn't something you'd want to be using if you're primary goal is speed. When running tight loops in string parsing I've found using string splitting and then cycling through the range of indices in a slice was several times faster than Regexp matching. Obviously performance difference will vary depending on the expression and application but that was enough to convince me to think twice about future usage of Regex - as to whether the problem needed Regex or if I was just using them lazily. The latter being a practice I'd slipped into after years of Perl hacking.
That depends entirely on the regex implementation. If the implementation uses a DFA to match multiple regexes simultaneously then the performance will be as good as a trie because a DFA is more or less a trie.
True. I was talking specifically about the same Regexp package as the one used in the topic project though.
I assumed that would have been obvious given the context however I apologise for not stating that in my comment and shall amend it appropriately. [edit: i can't add an amendment to my previous post now]
True. But nowadays most regex implementations are quite good (apparently go's is not - I haven't used it).
That said, most regex performance problems are PEBKAC. Writing a fast regex is hard and requires a pretty thorough understanding of parser theory. And many who use regexes don't understand that it's critical to precompile them for performance. You don't get a fast parser when you rebuild the DFA each time you use it.
Go regexps are slow (https://goo.gl/r0K2xw ), the problem is not regexps but Go's implementation of regexps. So let's not blame regexps when regexps aren't the problem. Because by that logic, people shouldn't use the sort package as well ...
> Regexes are the problem, because they're simply the wrong tool for the job.
for what job? Extracting route variables from paths ?they are the right tool for the job, only in the Go community they are deemed "wrong tool for the job". Your statement embodies everything that is wrong with the Go community. Instead of finding a solution to a problem you guys spend your time shifting the blame on "bad practices".
Wouldn't it make more sense to use something faster and simpler for most routing, and then an optional argument for regular expressions? Lots of web frameworks use that approach.
You're making a distinction where one doesn't need to be made. It doesn't matter if regexp is generally slow or if it's Go implementation specifically - if you're using Go and wanting something where performance is your primary goal then you're generally best to avoid using regexp.
> You're making a distinction where one doesn't need to be made. It doesn't matter if regexp is generally slow or if it's Go implementation specifically - if you're using Go and wanting something where performance is your primary goal then you're generally best to avoid using regexp.
Avoiding regexp doesn't fix Go's implementation of regexp. Making them faster does. Your argument is preposterous. If the Go team really cared about performances it would fix its regexp implementation.
I think you're missing the point of the discussion entirely. When you're more or less doing string splitting, using regex (regardless of performance) really is the wrong tool for the job. In this use case (url routing) a tree based data structure aka a trie or radix tree, are better suited.
> I think you're missing the point of the discussion entirely. When you're more or less doing string splitting, using regex (regardless of performance) really is the wrong tool for the job. In this use case (url routing) a tree based data structure aka a trie or radix tree, are better suited.
I'm not missing the point of the discussion. Using regex is not the wrong tool for the job. You deemed it the wrong tool for the job. And deeming it the wrong tool for the job doesn't fix Go regex being slower than in other languages. The 2 issues are not separate .People like you talk about performances as a goal while dismissing obviously performance issues in the standard library as "wrong tool for the job".
You're not going to convince anybody with this kind of argument, aside from gophers who already think like you do. I'm not one of them.
FWIW If the router was in C or Rust or .net (which has a Jit for their regex engine) I would still tell you Regex is the wrong tool for splitting a URL on '/'. How many people have to tell you facts for you to believe them? Forget your pointless anti-go bias. A regex, while perfectly good for certain types of pattern matching makes no sense here.
I often have taught the same lesson to my junior colleagues who use regex in Python or C++ code where splitting the string would be simpler, more maintainable, and faster.
> How many people have to tell you facts for you to believe them? Forget your pointless anti-go bias
Because a few people on HN is the consensus ? give me a break. You have your opinion, I've got mine, whatever you think you are you're in no way an authority on the matter. There are many ways to implement an http router, there are also many ways to implement regular expressions. A bad implementation isn't saved by deeming the use of regexp "wrong tool for the job".
> A regex, while perfectly good for certain types of pattern matching makes no sense here.
What make no sense is your petty comment.
> Forget your pointless anti-go bias
Pointing out facts is "anti-go bias". OK , how about you stopping drinking the "pro-go koolaid" ?
I think you're a bit delusional, I don't even write go (hence there literally being no koolaid for me to drink), but you're free to continue disagreeing, and continuing to be wrong :)
A person looking to use a HTTP router most likely isn't going to rewrite/fix the regexp package just so they can use this router when there are already other routers that are faster as is. Do you dispute that?
> If you use a broken hammer to attempt to insert a screw, you're crazy for using the hammer, not because it's broken.
Are you resorting to insults now ? or is the typical hate and mean spirit of the Go community ? a router isn't a hammer. And I could care less about your opinion.
The comment linked is about CSV parsing being slow (the linked GitHub issue in the comment shows that the `regexp` doesn't show up in the benchmarks at all). So I don't understand why you linked a 6-year-old thread that's been revived by unrelated topics?
Trie's are my favorite data structure. The implementation in the Linux kernel is a lock free RCU structure (http://lxr.free-electrons.com/source/lib/radix-tree.c). It is used for the page cache among other things.
They allow you to match the URL in a series of steps. Where each step can be static (ie "/foo") or dynamic (ie /images/).
You can then easily add features to this structure, like route middlewares at every level in the tree; just apply them while doing the routing.
This also allows you to match route variants easily, ie "foo" and "foo/" are the same, so are "/" and "/index.html". In the same way you can extract query parameters for middleware/handler use.
Also, tries means it's trivial to do detect routing ambiguities, so your routing table doesn't necessarily have to be built up front, or in any particular order. This is particularly powerful when you're developing an application with a team distributed both over space and time.
A trie is O(k), where k is the length of the key on inserts and deletes. Generally, any hashing algorithm is also O(k) with the additional overhead of the lookup in the table. This also ignores that most hashes have only amortized O(1) inserts. A trie usually also exhibits better memory locality and caching behavior. Also, a trie (like most tree data structures) can be implemented in a lock-free format fairly easily.
I don't usually say this but - isn't this a poor choice of name? Gorilla framework's Mux [0] has been around awhile and is quite popular for routing in Go.
What? No! There's no indication at all these two libraries are compatible - why would you want two completely disparate projects to give some indication they support the same interface?
Because many of the base infrastructure libraries are compatible as long as they used interfaces (grumble grumble os.File). I would be very surprised if this doesn't implement net/http/ServeHTTP
There are cases where it makes sense, like https://github.com/pkg/errors which is meant to be a drop-in replacement for the stdlib `errors` package containing a superset of its functionality.
Same here. I was working with gorilla/mux 2 min ago with some "legacy" code and saw this. Thought to myself "huh it might be some good new changes with gorilla/mux"
Do you know how it stacks up to httprouter for example? We can run benchmarks all day but it would be cool if you happened to have some real production statistics, by any change?
They're in the same ballpark, choosing one over the other was preference.
The benchmarks skew slightly in favour of httprouter, but only because pressly/chi embraces use of req.Context and this does a few ops. If you add req.Context use to httprouter they're basically the same.
If you're doing work with Context, you aren't going to be able to measure a difference that you'll care about.
If you're just serving static files, perhaps you'll care.
Interesting.. and btw, pressly/chi supports those routes easily, among many other combinations. Chi was designed to be very composeable with middlewares, subrouters and handlers. The idea is to consider the request passing through a "flow" of layers, handling and building the response along the way. This makes it easier to reason and organize each piece separately and then connect it all together.
What's the use case for all these routing libraries? Why not write a few lines of procedural code? It's a natural encoding of the routes "trie".
@methods('POST')
def set_int(request):
x = pop_int_component(request)
no_more_components(request)
do_set_the_int(x)
return Ok
def myapp(request):
x = pop_id_component(request)
if x == 'set_int':
return set_int(request)
else:
return NotFound
No type system hacks, extremely modular and simple to understand. Exceptions can replace basically all the boilerplate. The only drawback being there's no URL generation, but it's not like most sites have so many URLs that writing this by hand (directly in HTML templates or as "inverse functions") would be unmaintainable.
In this case, I don't see any typesystem hacks either, it uses the standard library types and as it looks, can just hook up to net/http/ServeFunc typed functions, which are basically what you'll use anyway if you use the standard library.
I don't quite understand what your irk is with this particular library (excepts that it's not the net/http default mux)
All these routing libraries are just wrong level of abstraction. In HTTP you have resources, and resources are not flat, they are hierarchical.
And hierarchy of resources defines some connection between then, and some common properties (data, access level etc). So any resource can respond to some HTTP method or can delegate to another, nested resource (if any) for any method.
So (in PHP, from our still proprietary framework):
// Index resource, just routing, no HTTP method handling.
class IndexResource extends Resource {
public function __construct(Application $app) {
// ...
}
// Route nested resources for any request
public function any(Request $request) {
// Static path:
$this->path(
CollectionResource::Path, new CollectionResource($this)
);
}
}
class CollectionResource extends Resource {
const Path = 'items';
// Parent resource type constraint, you can access
// parent data using IndexResource API:
public function __construct(IndexResource $parent) {
//...
}
// Route nested resources for any HTTP method:
public function any(Request $request) {
// Regexp pattern for URI segment:
$this->match(ItemResource::Pattern, new ItemResource($this));
}
// Or handle GET
public function GET(Request $request) {
// ...
}
// Or POST maybe
public function POST(Request $request) {
// ...
}
}
class ItemResource extends Resource {
const Pattern = '(\d+)';
public function __construct(CollectionResource $parent) {
// ...
}
public function any(Request $request, $prefix, $id) {
$this->item = Item::find($id);
}
public function GET(Request $request) {
return JSON::string($this->item);
}
}
It's not about routers + controllers etc, it's just resources + delegation to nested resources:
- recursive routes are trivial, so no problems with CMS-like applications;
- looks good with type systems;
- no long routing tables (in fact, no routing tables at all), so true modular apps;
- you can use anything (e.g. database) for resources lookup, good for CMSes again;
- it's absolutely RESTful.
Yes, automatic URI building is not so easy, but it's kinda possible, in some semi-automatic way.
I wonder why frameworks built this way are so uncommon (ours were inspired by Bullet BTW).
> In HTTP you have resources, and resources are not flat, they are hierarchical.
That isn't really true. There is nothing about HTTP, or even REST, that requires resources to be hierarchical. That said, it is quite common, and helpful for human beings, if they are hierarchical, so ...
> I wonder why frameworks built this way are so uncommon
Good question! The only one that springs to mind is Stapler:
There is Bullet http://bulletphp.com, that inspired us, but it is too simple and lacks some important (for us) features, e.g. recursion.
There are nested routes in Express and some Golang libraries (see gongular near on HN), but they are not resource oriented. And yes, there are resources in Rails, but it's sooo complex and heavy.
One downside I see to this: It's not possible to statically determine which resource will respond to which URL.
I suppose that is also a benefit (maybe some resource can divert to another resource under heavy load/when its DB is unavailable). But quite frequently I want to answer this question with just the source code.
> But quite frequently I want to answer this question with just the source code.
Exactly. And that is also a benefit if routes are fetched dynamically from database, e.g. for content application, so your database is really content repository without too many levels of abstraction.
As for source code, it's kinda silly to split your app to fixed «controllers» and «models», you should split it functionally for simple reuse. Most modern PHP frameworks are so unflexible that way (laravel is just terrible IMO).
+1 for the confusion with Gorilla's mux library. Have you benchmarked this against Gorilla's lib to show why we would want to use this (relatively unproven) library instead?
+1 for the confusion with Gorilla's mux library. Have you benchmarked this against Gorilla's lib to show why we would want to use this (relatively unproven) library instead?
If you're not going to explain in your readme or docs why anyone should use this over the Gorilla library you're kind of wasting everyone's time. Gorilla/Mux is very widely used and well-supported.
In what sense? Can it support many, many more routes? Or does it scale better in the sense that it causes zero or near zero GC pressure? Better than what?
Why does routing get so complicated? I use Web2py which doesn't even have a router and it's so much easier. Are regxes, variables, http methods all that necessary in the router? Just pass the info to a controller.
Parts of the readme are copied from [0] julienschmidt/httprouter...
"In contrast to the default mux of Go's net/http package, this router supports variables in the routing pattern and matches against the request method. It also scales better."
79 comments
[ 3.9 ms ] story [ 161 ms ] threadhttps://gdstechnology.blog.gov.uk/2013/12/05/building-a-new-...
The Vulcan proxy from Mailgun does the same thing:
http://vulcand.github.io/proxy.html#route
It looks like it doesn't, so existing open source golang url routers will perform much better, contrary to this post. The one in vuland is interesting in that it supports both regex AND tries, so you get the best of both.
Additionally, httprouter (golang) uses a trie and is fast as hell:
https://github.com/julienschmidt/httprouter
Edit: Adding another one (thanks buro9!) that is used in production by CloudFlare:
https://github.com/pressly/chi
Any suggestions are appreciated!
I have written a very basic introduction to writing webapps in Go without using a framework, as others have rightly pointed out, we really do not need to use a framework in Go for writing webapps, and Go is a great language.
If you don't like mine, https://github.com/astaxie/build-web-application-with-golang... is a nice start for new comers.
True, and anyone who knows that Russ Cox is a core member of the Go team will have a hard time suppressing a smirk when reading this :)
https://swtch.com/~rsc/regexp/
I assumed that would have been obvious given the context however I apologise for not stating that in my comment and shall amend it appropriately. [edit: i can't add an amendment to my previous post now]
That said, most regex performance problems are PEBKAC. Writing a fast regex is hard and requires a pretty thorough understanding of parser theory. And many who use regexes don't understand that it's critical to precompile them for performance. You don't get a fast parser when you rebuild the DFA each time you use it.
*edit: a word
for what job? Extracting route variables from paths ?they are the right tool for the job, only in the Go community they are deemed "wrong tool for the job". Your statement embodies everything that is wrong with the Go community. Instead of finding a solution to a problem you guys spend your time shifting the blame on "bad practices".
Avoiding regexp doesn't fix Go's implementation of regexp. Making them faster does. Your argument is preposterous. If the Go team really cared about performances it would fix its regexp implementation.
I'm not missing the point of the discussion. Using regex is not the wrong tool for the job. You deemed it the wrong tool for the job. And deeming it the wrong tool for the job doesn't fix Go regex being slower than in other languages. The 2 issues are not separate .People like you talk about performances as a goal while dismissing obviously performance issues in the standard library as "wrong tool for the job".
You're not going to convince anybody with this kind of argument, aside from gophers who already think like you do. I'm not one of them.
I often have taught the same lesson to my junior colleagues who use regex in Python or C++ code where splitting the string would be simpler, more maintainable, and faster.
Because a few people on HN is the consensus ? give me a break. You have your opinion, I've got mine, whatever you think you are you're in no way an authority on the matter. There are many ways to implement an http router, there are also many ways to implement regular expressions. A bad implementation isn't saved by deeming the use of regexp "wrong tool for the job".
> A regex, while perfectly good for certain types of pattern matching makes no sense here.
What make no sense is your petty comment.
> Forget your pointless anti-go bias
Pointing out facts is "anti-go bias". OK , how about you stopping drinking the "pro-go koolaid" ?
Are you resorting to insults now ? or is the typical hate and mean spirit of the Go community ? a router isn't a hammer. And I could care less about your opinion.
https://github.com/ShamariFeaster/Templar/blob/master/core/R...
You can then easily add features to this structure, like route middlewares at every level in the tree; just apply them while doing the routing.
This also allows you to match route variants easily, ie "foo" and "foo/" are the same, so are "/" and "/index.html". In the same way you can extract query parameters for middleware/handler use.
[0] https://github.com/gorilla/mux
If it was "github.com/foo/poppycock" I could just write import mux "github.com/foo/poppycock" and use it as mux.
Import name similarity is a terrible argument for picking a name.
Historically gophers haven't minded the occasional namespace collision: https://github.com/golang/go/issues/9
Where can I find the benchmarks?
The list of those using it in production includes:
- Pressly
- Cloudflare
- Heroku
- 99designs
- Origami
- IT Jobs Watch
- CrowdRiff
At Cloudflare it's our default router for internal APIs, which are all written in Go.
(and yes it uses a trie)
The benchmarks skew slightly in favour of httprouter, but only because pressly/chi embraces use of req.Context and this does a few ops. If you add req.Context use to httprouter they're basically the same.
If you're doing work with Context, you aren't going to be able to measure a difference that you'll care about.
If you're just serving static files, perhaps you'll care.
In this case, I don't see any typesystem hacks either, it uses the standard library types and as it looks, can just hook up to net/http/ServeFunc typed functions, which are basically what you'll use anyway if you use the standard library.
I don't quite understand what your irk is with this particular library (excepts that it's not the net/http default mux)
And hierarchy of resources defines some connection between then, and some common properties (data, access level etc). So any resource can respond to some HTTP method or can delegate to another, nested resource (if any) for any method.
So (in PHP, from our still proprietary framework):
It's not about routers + controllers etc, it's just resources + delegation to nested resources:- recursive routes are trivial, so no problems with CMS-like applications; - looks good with type systems; - no long routing tables (in fact, no routing tables at all), so true modular apps; - you can use anything (e.g. database) for resources lookup, good for CMSes again; - it's absolutely RESTful.
Yes, automatic URI building is not so easy, but it's kinda possible, in some semi-automatic way.
I wonder why frameworks built this way are so uncommon (ours were inspired by Bullet BTW).
That isn't really true. There is nothing about HTTP, or even REST, that requires resources to be hierarchical. That said, it is quite common, and helpful for human beings, if they are hierarchical, so ...
> I wonder why frameworks built this way are so uncommon
Good question! The only one that springs to mind is Stapler:
http://stapler.kohsuke.org/what-is.html
There are nested routes in Express and some Golang libraries (see gongular near on HN), but they are not resource oriented. And yes, there are resources in Rails, but it's sooo complex and heavy.
I suppose that is also a benefit (maybe some resource can divert to another resource under heavy load/when its DB is unavailable). But quite frequently I want to answer this question with just the source code.
Exactly. And that is also a benefit if routes are fetched dynamically from database, e.g. for content application, so your database is really content repository without too many levels of abstraction.
As for source code, it's kinda silly to split your app to fixed «controllers» and «models», you should split it functionally for simple reuse. Most modern PHP frameworks are so unflexible that way (laravel is just terrible IMO).
In what sense? Can it support many, many more routes? Or does it scale better in the sense that it causes zero or near zero GC pressure? Better than what?
https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface
"In contrast to the default mux of Go's net/http package, this router supports variables in the routing pattern and matches against the request method. It also scales better."
[0] https://github.com/julienschmidt/httprouter