Ask HN: Any C programming Postgres wizards willing to port 460 lines of code?
I wonder if there are any kind C programming wizards who know Postgres and might consider doing the open source port? It's beyond my Python programming skills and I dare not write crappy C code for fear of creating something nasty and insecure.
I can repay either by reciprocating with Python/web development/Linux/AWS knowledge, or if I have nothing of value to offer then I can offer thanks and praise.
The existing MySQL implementation is 460 lines of code.
There's a MySQL implementation here: https://github.com/libgit2/libgit2-backends/blob/master/mysql/mysql.c
There's a sqlite implementation too: https://github.com/libgit2/libgit2-backends/blob/master/sqlite/sqlite.c
Some relevant links: http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/
89 comments
[ 3.7 ms ] story [ 150 ms ] threadSee also http://www.postgresql.org/developer/
Good idea though, time I started getting some C skills but on a less important bit of code.
If you end up doing it yourself, I'll volunteer to do a quick review.
Don't fish for free work when you could turn this into a learning opportunity.
Besides, why would you want to use PG as a backing store unless you were doing weird multitenant stuff?
Same as many other developers here on HN I suspect. Coding in the hope of building something people want to use.
Truly rich companies are more likely just to pay for talented developers or services than small folks--see also aquihires.
Anyways, enough of the business/philosophy stuff...what are you looking to gain by using PG as a backing store?
It's also a great chance for someone who perhaps is looking for a gig to put some code up here that quite a few people will review.
I propose we use "Task HN:" and do these requests more frequently. We'll surely learn a lot, come to ingenious solutions, and maybe, just maybe, make our time on HN more productive. I, for one, want to at least see, if not help, what small but interesting byte-size hurdles others encounter and how others can solve it in different ways, and all the discussion around it.
Working on something together binds communities even tighter.
Surely the overall quality of HN won't suffer even more from people asking other hackers to write their C for them, what a great idea!
fucking /s
Surely it won't have as detrimental effect as anonymous throwaway accounts rich with shallow sarcasm and curse words.
The large majority of HN readers are lurkers, who don't vote at all, the next largest group are those who aren't able to downvote, followed by the smallest group, the downvote-capable.
I've got a few accounts that have downvote and dozens without, but it's not enough to stop the constant barrage of stupidity that floods what is supposed to be a news aggregation list. I don't mind seeing SHOW HN:'s, I like seeing what hackers are working on. On the other hand, a bunch of posts TASK HN: Write a C wrapper because I can't do it, that's gonna get old real fuckin' quick.
I will shill as hard as I can, but I fear that it won't be enough.
HN has always been a bit more than a news aggregator, there are "Tell HN"s and "Show HN"s too.
Should hacker news be the place where interesting suggestions for stimulating open source tasks were shared? I think yes, more so than all the politics that creeps up here.
And if the requirement is that it must be open source and not directly something the asker will make money of then it stands very little chance of being abused and if it does, is it any worse than all the other spam we downvote/report?
And it sounds like this is one of them. The OP is "trying to come up with an idea that will make an income when I'm not doing my day job to pay the bills":
https://news.ycombinator.com/item?id=9833548
If you can't pay someone to do it, even at some of the low rates I see on elance, then learn to do it yourself. It would be a different matter if this were someone asking for input on the code he had written, but just asking someone to do it for him is unseemly.
I'm 100% in favor of a feature on HN where existing open source projects ask for help and/or contributions, but someone who's starting up a for-profit business and asking for free labor leaves a very bad taste.
I've had would-be clients like this, who ask me to do work for free or at a significant discount since "it's related to your open source work, anyway, and you could even use it in your project". Made the mistake of accepting work from a client like that once, but never again.
On The other hand, I think there is little to be gained from frictionless requests for volunteers. That invites spam and low quality requests because there's nothing to lose. A linked blog post with technical details and relevant context is probably the ideal level of impedence with the HN format. The format of this request is a nice hack, but standardizing on a kludge is not the way to go.
Maybe the structure is a monthly "Can you help?" thread...and perhaps a complimentary "Can I help?" thread.
I think you may be underestimating desire for geek cred. Non-throwaway account would have little incentive to just spam with trivial tasks because they won't be upvoted, might even be flagged, and are tarnishing individuals reputation.
You may have a point about format not being ideal and having too little friction. Not sure.
> Maybe the structure is a monthly "Can you help?" thread...and perhaps a complimentary "Can I help?" thread.
This most certainly wouldn't work for these types of requests, as they are rarely in opportunity to be scheduled for the next month, or at the very least, get solved by then by ugly hacks.
Again, not sure what would emerge out of it really.
If an open source project can't schedule major features a month out, then throwing more bodies at it won't solve the fundamental problem of disorganization and isn't prepared to make appropriate use of volunteer's time in ways that really value that time. Structuring policy around a continuous stream of "emergencies" invites low quality requests. A policy which parallels HN's job listing policy favors HN members rather than people for whom everything above zero euros is paying too much. The HN community benefits from a slower process with a higher barrier to posting requests.
I do not see why you focus so much on team building when that is completely unrelated to what is being discussed. Not all open source projects are big enough to warrant any kind of team. They're literally just someone's side project, which may or may not be useful to other people.
But eventually, it became what any cynic would expect: a mechanism for "We will build your iOS app for $2500" offers and today it's dead. This isn't a 1.0 release. It's a 0.1 alpha. Nobody has written the 400 lines of good expert code requested. The existing C code that has been offered may or may not blow up under the OP's load or not meet their exact needs. No one is claiming it is bet-your-business ready. The fundamental open-source project problems of maintenance and further development are not solved or even addressed.
More importantly, it hasn't been demonstrated that this mechanism works for getting people to substantially support the project with more than the goodwill of contributing existing code developed for another purpose. Don't misunderstand me, that's a great thing. Which is why a once a month format is a reasonable starting point.
If it was in small bitsized chunks, i'd definitely surf it looking for easy things I could contribute that could help a few people.
Even better is if the site could certify it helped them, and I could get some kind of tax credit for it :D
I've thought about this idea and I think without proper "project managers" (I don't mean someone with a project management degree - but just someone to coordinate all the efforts) it seems like it could be a total failure.
Here is an example: I have a C++ game engine that I need help with feature X. I think any C++ developer could come up with an implementation of X, but does the style fit my game engine? Does it interconnect with the rest of the engine? (of course ignoring the fact that any discussion of C++ would generate gigabytes worth of comments - I've read some of the newsgroup discussions on style alone...). You can't just be like "create a logging interface" without having studied the rest of the code base. It would be like trying to create a feature for Apache or Linux kernel - if we didn't study their code base and style anything we submit would be laughed out the room - not because what we did won't work or wrong but it doesn't fit with the rest of the code.
This is where the project managers come in - they already know the style and inner workings and take what you submit and hack it into the right style and push it into a SCM or push it upstream (or reject it). Personally I would feel more inclined to make contributions to the Linux kernel if there was a friendly middle man I could look over my work before it gets to Linus - only because I fear if I submit something stupid I'll get chewed out by Linus.
Even if this were to happen - I would not want to read 10k comments of tail call recursion optimization, smart pointer usage, or discussion of non-portable code that will work on 99% of systems except for AIX Unix and Blue Gene/Q.
Here is a lib to match python's string functions in C++.
https://code.google.com/p/pystring/
When working with string and vectors C++11 can actually be pretty straight forward, productive, and clear with no manual memory management. It definitely doesn't have as many string functions out of the box as modern scripting languages though.
This project uses PQescapeLiteral() and asprintf() to build queries. I'd recommend prepared statements over this approach (which is how the MySQL backend builds queries).
It doesn't implement anything but write() and free() yet, so it's not actually functional.
Unfortunately I believe this would clobber the library license (GPLv3 on this Postgres plugin vs GPLv2 with a linking exception for libgit). With this in mind I'd probably just recommend a rewrite. It's not that much code.
Update: it looks like the libgit2 backends aren't well maintained: https://github.com/libgit2/libgit2-backends/issues/13#issuec...
The demand for the highest quality code for what is essentially begging is also not becoming. Even contributing a terrible implementation would tickle people's motivation button (so they can teach someone without building your entire project, show they are measurable better than someone, etc)
"Beggars can't be choosers"
1: it's open source snd presumably would be committed to the libgit2 project.
2: it's likely to be useful to others
3: it's central, critical code that isn't well suited to beginners. Of course it should be quality code, it's responsible for storing data.
4: it would strongly benefit from the eye of a Postgres expert, which I am not
5: I do not have money to hire developers
6: I write open source code too so I contribute my time to the public
This is a good idea. Other people could abuse it but this is certainly not abuse, any more than it would be to just file a feature request as an issue on a project. Hope it comes through.
https://en.m.wikipedia.org/wiki/Stone_Soup
Are you looking for this to complete a $xxK client project? Or are you looking for this to help students understand a C implementation?
It makes a world of difference. If you are getting paid for the result, then you should share with the developer. If you are not, and doing this for the greater good, then it's fine.
We should really define the line here. There is enough exploitation in the dev world and the last thing we want is developers exploiting other developers.
Please note that I also don't know much C, but this implementation does work. Also included is a Postgres version of the Ref DB backend (so nothing hits the filesystem). There are a few bits that are not implemented since we didn't have use for the reflog and those parts are technically optional.
Would probably be good to get another set of eyes on this from someone much more familiar with C.
Hope this helps!
What is the license?
Any reason you didn't use it in the end? What was your use case for it?
Question for you... and I'll read the source in the morning...but just quickly does it prevent storage of duplicate objects? That's one of the main things Im interested in is saving space when multiple git repos contain exactly the same object.
And THANKS again. Awesome. Can I do anything for you? Send you a bottle of wine? Help with some Python or Linux? If you put your contact in your profile I'll drop you an email.
In terms of duplicating objects, I believe that if you do choose to store objects from many repos in the same table, they will NOT be duplicated and you will get your space savings. Don't take my word for it though.
We actually did use this code in production for a period of time. In the end we realized that one of the main features of Git, immutability, didn't suit our needs well and we designed a versioning system based closely on Git, but built on Postgres directly. The main benefit of this is using primary keys as the object ids, instead of hashes of the content. This means we can change the content without changing the object's id (which in normal Git then means changing the tree, commit, and every parent commit).
Good luck!
I had been hacking together a Kyoto Tycoon-backed implementation for a project (since dropped); our design exposed the ref id to the user (e.g. 'master', 'master/mhodgson', etc) and branch/merge as necessary. This way, our primary keys remained a constant refName that pointed to the HEAD of a commit chain, each of which referenced immutable commits/trees/blobjects.
Although my days of libgit2 hacking are long past, I'm very curious if/how our design could have been improved; immutable pkeys were important for us as well.
Github: https://github.com/anulman/libgit2/tree/kyoto/src/backends/k...
Regardless, sounds fairly implementation-specific. Think I just followed you on Twitter, happy to discuss further offline.
[1] https://libgit2.github.com/libgit2/#HEAD/group/commit/git_co...
Any objections to that?
My naive thoughts were that it would perform extremely well as I had thought that Postgres scales extremely well with multicore.
Is there anything I can read anywhere about such performance limitations? Am I correct in understanding that you found performance limitations - I assume when compared to file system?
Any pointers to info on where github tried this?
- The ruby error reporting can be ported to giterr_set() https://libgit2.github.com/libgit2/#HEAD/group/giterr/giterr...
- Uses prepared statements √
- Requires the libgit2 source tree to be in the include path to build as it uses some internal headers.
- Should probably escape input to git_buf_printf() before it's passed to the DB.
- Should change return values from magic (0, -1, etc) to constants (like GIT_OK, GIT_ERROR, GITERR_NOMEMORY)
- Memory allocation is very light (mostly uses stack buffers) and seems sane at a glance.
- I'd recommend a -Wall -Wextra -Wpedantic compile on clang or a clang static analyzer run to see if there's anything weird or undefined I missed.
Update 2: Nevermind, what I thought was a bug in read_prefix() is probably just poorly documented libgit2 interface - I believe read_prefix() operates on GIT_OID_HEXSZ due to the git_oid_ncmp() function which does memcmp with 4-bit precision (so you can use a short hex id with an odd length).
https://github.com/cbdevnet/libgit2-backends/blob/master/pos...
Caveat Emptor: I did not test this since somehow the Debian package of libgit2-dev seems not to include the GIT_* constants. It should probably work, though.