btw - OpenEvidence is also the name that competitive debaters used for their giant archive of policy debate, LD debate, and (small amounts of) PF debate evidence. That project has been going on for decades now.
We turned that into a proper, ready-for-use-in-AI dataset and contributed it to the mainstream AI community under the name OpenDebateEvidence. Presented at NeurIPS 2024 Dataset and Benchmark track.
OpenEvidence does use an LLM. It's ChatGPT for doctors/medical research, tuned to give references in respected journals etc. Hospitals (at least US, Canada, UK) allow and even encourage (US) staff to use it as a quick lookup, the way they'd otherwise Google for a dosage say, it just does that better.
(My wife's a hospital doctor & author and introduced me to it; other family in other countries.)
See this. I use OpenEvidence. It has access to full text from some of the major medical journals. But generalist models seem to outperform it. Not sure what is going on there.
Can this really be a kind of herding stampede behavior over Cowork? It’s been out several days now and just all the sudden today, all the traders suddenly got it into their little herd animal heads that everyone should rush to the exists… after that equally sketchy silver and gold rug pull type action last week?
I am with you. I think that it is more likely to be related to Japanese carry trade unwind starting to worry the banks, while continuing to drive the “AI disrupt everything” narrative via mainstream news.
I might be not across the detail, but to me the legal plugin seems like it’s mostly adding some skills (prompts) that are fairly basic that any technically minded people could do, and is not enough of an improvement for completely non technical people to use.
If it turns out that AI isn't much more productive, it could also turn out that people still believe it is, and therefore don't value software companies.
If that happens, some software companies will struggle to find funding and collapse, and people who might consider starting a software company will do something else, too.
Ultimately that could mean less competition for the same pot of money.
So the value of a skill file is that it tells the model how to format its response for use within the software environment surrounding the model.
With programming, it's mostly about how to tell it to use some API.
But all the model can do is reply some text, and the actual work needs to be done by the software(the agent harness) which needs to parse the model response and translate it into actual work.
My point is there is no magic: the model just reads the skill file and then uses that as a template for a textual response, which is then parsed and processed by traditional software.
Yes, the model will read it and it will influence its response, but without some extensive software harness around the model to give it data for context and and so on: totaly useless.
Why? Because garbage in is garbage out.
So telling the model to review a contract and pay attention to "Whether indemnification is mutual or unilateral" will result is some response from the model, but without additional data it will be at the same level as what you can get from a google search.
The effect on established companies is exactly zero.
Now, having an in-house skills and proprietary software around the model to integrate it into your system, that would be valuable indeed, but not something an AI lab can replicate without building the whole company from scratch.
14 comments
[ 0.37 ms ] story [ 54.5 ms ] threadHow could it possibly keep up with LLM based search?
We turned that into a proper, ready-for-use-in-AI dataset and contributed it to the mainstream AI community under the name OpenDebateEvidence. Presented at NeurIPS 2024 Dataset and Benchmark track.
https://neurips.cc/virtual/2024/poster/97854
https://huggingface.co/datasets/Yusuf5/OpenCaselist
(My wife's a hospital doctor & author and introduced me to it; other family in other countries.)
See this. I use OpenEvidence. It has access to full text from some of the major medical journals. But generalist models seem to outperform it. Not sure what is going on there.
Something seems quite off. Am I the only one?
I might be not across the detail, but to me the legal plugin seems like it’s mostly adding some skills (prompts) that are fairly basic that any technically minded people could do, and is not enough of an improvement for completely non technical people to use.
If that happens, some software companies will struggle to find funding and collapse, and people who might consider starting a software company will do something else, too.
Ultimately that could mean less competition for the same pot of money.
I wonder.
I'm like: oh that's it, a bunch of skills files?
So the value of a skill file is that it tells the model how to format its response for use within the software environment surrounding the model.
With programming, it's mostly about how to tell it to use some API.
But all the model can do is reply some text, and the actual work needs to be done by the software(the agent harness) which needs to parse the model response and translate it into actual work.
My point is there is no magic: the model just reads the skill file and then uses that as a template for a textual response, which is then parsed and processed by traditional software.
So in terms of legal skills, a stand-alone skill like the contract review skill at https://github.com/anthropics/knowledge-work-plugins/blob/ma... is basically useless.
Yes, the model will read it and it will influence its response, but without some extensive software harness around the model to give it data for context and and so on: totaly useless.
Why? Because garbage in is garbage out.
So telling the model to review a contract and pay attention to "Whether indemnification is mutual or unilateral" will result is some response from the model, but without additional data it will be at the same level as what you can get from a google search.
The effect on established companies is exactly zero.
Now, having an in-house skills and proprietary software around the model to integrate it into your system, that would be valuable indeed, but not something an AI lab can replicate without building the whole company from scratch.