YC Fellowship optimal cut-off from 6500 applicants

Any automated solution presumes that you can program a query that detects signals of desirability ( whatever they are in this context ). Yet if you can do that, you have no trouble no matter the number of applies.

Since there is no transfer of expertise from the expert assessors to such a query that is possible, you have to stick with humans.

Perhaps the large number suggests a different opportunity however. It's now an opportunity to get more aggressive about quality. Raise the bar ( as long as your signals still hold under their amplification ) because now you have a bigger pool.

Previously looking for semi-coherent responses that piqued interest? Now look for decisively coherent ones.

Previously looking for some evidence of lack of confidence and unassuredness in the idea ( indicating self-critical thinking and honesty which are useful traits in themselves and, by the heuristic that if your answers are so bulletproof you wouldn't need funding or help, suggests desirability ( and by the secondary heuristic that investors like to invest in risk otherwise it is : A ) not interesting for them and B ) not okay if it fails ( since it was such a sure thing ) ) )? Now go for absolute divulgence and transparency.

Then how do you gauge the effectiveness of this approach ?

Try it on a small batch, and see how well it correlates with an existing ranking from an expert assessment of another small batch.

If you iterate like this, maybe you will find there is some technological signal you can query ( like frequency of green and red flag words, or like clustering texts into batches based on sentiment or topic, or bag of words and seeing if you can't exclude whole batches, or like using cosine similarity with some canonical great prior applications and terrible prior applications to partition the applies ).

Another idea is it's like you have to mark 6500 test papers, so why not employ someone like CrowdFlower ( who shepherds human intelligence tasks ) to apply your score-sheet to each application, cross-checking a sample to validate stability, to give each application a number?

As much as possible this score sheet distills the subjective appraisals of YC.

Finally I think it is unreliable for people to choose the top 5% or 2% of a number of things. People I feel are far better at choosing the top 1 out of 3 things. So iteratively apply this by doing a "facemash" of 3 applications side by side, making the selection into a game, and get the partners, part time partners, and associates to play this until a stable ranking emerges. If this takes too long then make it 1 out of 10. So each round of the game produces 10 preference relations that the selected application is better than 9 others.

A final, and perhaps the best idea, is to pre-compute a ranking based on meta signals from the application process itself such as: number of revisions ( suggesting a lack of confidence in the pitch and anxiety over a small amount of money ), length of the video ( longer suggesting unfamiliarity with the idea or non-empathetic desperation ), Benford's law over the character ( or word ) count distribution of response texts for the answers to detect anomalies, "alignment" ( YC weights the application questions in importance and computes the inner product ( or cosine similarity ) of the weights vector and the word ( or character ) lengths of responses vector, to see how well applicants weighting of importance aligns with YC. Interesting outliers ( such a orthogonal perspectives ) could also be tagged for a closer look.

It seems that unless they have done this before, the workable procedure for 6500 applications is itself an experiment.

Assign each application a rating of 1000. Randomly select two applications and show them to a partner. The partner selects the better application, or declares a tie. The Elo ratings are updated accordingly. Aggregate this over many partners and many applications, and you now have a quantifiable measure of YC's aggregate preferences for the applications. Further, you can now calculate the probability that YC will prefer one application over the other by comparing ratings, even if the two applications have never been directly compared before.

Then, rank based on ratings and select the top N (N being the number they want to interview).

The cool thing about this is it's very simple to implement (easily done in a day, or even a few hours). It'd be a useful measure even for YC's regular batches, which have more manageable numbers.

Is this online anywhere?

As far as cut-off: if the major experiment here is working with teams remotely, why question increasing the 20 seats? The number of applicants shouldn't affect that experiment as far as I know. If they're using this as a means to increase their reach for YC batches, that's another topic.