HN Explorer

Ask HN: Is this normal? Long-form transformer take home

11 points by bpiche ↗ HN
Mostly an ML/NLP engineer, and been interviewing a lot lately, and therefore have been doing a lot of NLP take home tests lately, some more complex than others. Certainly prefer it over any LeetCode tests, though some of these take homes are so complex, they verge on pro bono work and I often feel I should be compensated.

One organization recently gave me a take home with the following demands:

--

Here are 150k long-form (6000+ words) documents, and a list of labels.

Please use a recent transformer model as the vectorization/representation layer to train a multilabel classifier on this data set.

You can use CoLab and their free GPU tier, but we won't pay for any GPU/TPU time.

Also, please compare this solution to other algorithms (linearSVM, XGBoost) and write 1000 words about the performance tradeoffs.

---

I'm not a deep learning expert, and I assumed that transformer models are basically limited to short form text with the exception of the Longformer and Big Bird architecture, and I was under the impression that those solutions were pretty memory intense. Other solutions are limited to only looking at the first 512 characters or so. And I'm not even sure CoLab's free tier can handle this.

Is this too much? Part of me is really excited to try this but another part of me is already imagining the compute time/space required to run this thing.

25 comments

[ 3.5 ms ] story [ 83.7 ms ] thread
formerkrogemp
Why are you working for free? I'm not in machine learning, but if I got this in an accounting field I would think it was a scam and not a legitimate job offer. Just my two cents. Best of luck friend.
bpiche
From the mouths of babes. You get so used to grinding LeetCode and doing take homes weekend after weekend, you stop questioning reality. Maybe it takes someone from an outside field looking in to see it. Thank you for the refreshing perspective..
formerkrogemp
I've enjoyed reading about programming on this site for a few years, so, no, thank you for providing a peek into your reality.
hjkl0
FWIW, some companies do compensate you for trial work.
[deleted]
(comment deleted)
jstx1
> Here are 150k long-form (6000+ words) documents, and a list of labels.

> write 1000 words about the performance tradeoffs.

Big no. And it shouldn't be a close decisions for you - this is way too much work to do for someone you aren't working for. Who in 2022 is asking for 1000 words as an explanation? And who is actually reading it???

Fascinating stuff, thanks for sharing.

(I kind of wish take homes were less common and more companies were doing LC for data science and ML)

[deleted]
(comment deleted)
brudgers
They are looking for someone with the level of experience that allows treating the problem as trivial.

That means plugging in the data to the tool they use all the time, adjusting a few parameters, and writing the 1000 words while waiting for the results…that’s possible because they know the tradeoffs they made as they made them.

If it looks like a hard and interesting challenge, that desired candidate isn’t you.

Nothing wrong with that.

They want someone who can knock this class of problems out before lunch on a Tuesday when the boss asks at 10am.

p1esk
To put things into perspective, someone like that makes $600k+ at FAANG. At a hot new startup they would be making $300k+ base. And these people are in huge demand currently.

If you're expecting this level of compensation I believe a take home task like this is not too much.

jstx1
That much? For the task that OP is describing? I know that FAANG salaries are high in the upper bands but I think that OP's take-home is doable at a much lower level. Or I'm very underleveled and underpaid.

And of course, the task being doable doesn't make it a good take home, I still think OP should turn it down.

p1esk
For the person described in brudgers’ comment.
bpiche
This role pays around 170-190k
p1esk
How many years of ML/NLP experience do you have? 190k is probably reasonable for someone with less than 2 years. If you have more, definitely try more companies.

A dollar today is worth is lot less than even 1 year ago, just look at the housing prices.

bpiche
What if you're expecting a third of that?
sjg007
Take a pre-trained Bert model and then fine-tune the label classifier part using the documents.
bpiche
It really is that simple, but I didn't think we were there yet!
sjg007
Hey man, google pytorch.
bpiche
Spent a lot of time trying to get PyTorch and CUDA to play with my 3070ti yesterday and finally gave up and made a CoLab notebook. Got a pipeline working and trained a multiclass classifier, but the f1 is in the toilet. Might try a different dataset, then maybe a tensorflow implementation next if that doesn't move the needle. Hard to believe the accuracy could be this bad compared to a linearSVM baseline, must be giving it the wrong x/y labels or something.
leobg
For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT model.

Re: the length problem, some ideas:

a) Just use the first part of each doc. For most humans, in most cases, that would be enough to do the classification.

b) Mix head and tail of each doc (i.e. cut out as much of the middle as necessary).

c) Split each doc into as many parts as needed, classify all of them, and then go with the majority vote.

Good luck with the interview!

bpiche
Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004.

Found a tutorial which at least makes classification with RoBERTa pretty straightforward.

I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged.

[0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683

[1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classifica...

parentheses
Don’t know ML but this is sounding like a lot of work. I almost always say no to take homes. There’s also a limit to how much the a take home should take. >2h of your time is asking a lot. I philosophically disagree with take home interviews for many reasons.
throwaway81523
Yes it is too much. I'd decline to do that much work on an unpaid basis without more commitment from the company.

Regarding jstx1's question of who was going to read the 1000 words: obviously, an NLP text summarization package that will spit out a number from 1 to 5.

aborsy
More than a couple of hours, no, it’s a scam.