Ask HN: Whats your opinion on open source contribution as a hiring challenge?

We are an open-source company, why not require applicants to create a contribution to our project as a hiring task? And since I don't want this to be free labour, I'd even pay a fixed rate. I’m curious if you have seen this work, or if you would apply for a position that requires it? What would be your concerns?