My biggest release of the year: a series of 7 specialized embedding models for information retrieval within tax documents, is now available for free on @huggingface
These new models aim to offer an open-source alternative for in-domain semantic search from large text corpora and will improve RAG systems and context addition for large language models.
Trained on more than 43 million tax tokens derived from semi-synthetic and raw-synthetic data, enriched by various methods (in particular MSFT's evol-instruct by @WizardLM_AI), and corrected by humans, this project is the fruit of hundreds of hours of work and is the culmination of a global effort to open up legal technologies that has only just begun.
A big thank you to @microsoftfrance for giving me access to state-of-the-art infrastructure to train these models, and to @julien_c, @ClemDelangue , @Thom_Wolf and the whole HF team for the inference endpoint API and the generous provision of @AIatMeta LLama-3.1-70B. Special thanks also to @tomaarsen for his invaluable advice on training embedding models and Loss functions
1 comment
[ 2.7 ms ] story [ 10.3 ms ] threadThese new models aim to offer an open-source alternative for in-domain semantic search from large text corpora and will improve RAG systems and context addition for large language models.
Trained on more than 43 million tax tokens derived from semi-synthetic and raw-synthetic data, enriched by various methods (in particular MSFT's evol-instruct by @WizardLM_AI), and corrected by humans, this project is the fruit of hundreds of hours of work and is the culmination of a global effort to open up legal technologies that has only just begun.
A big thank you to @microsoftfrance for giving me access to state-of-the-art infrastructure to train these models, and to @julien_c, @ClemDelangue , @Thom_Wolf and the whole HF team for the inference endpoint API and the generous provision of @AIatMeta LLama-3.1-70B. Special thanks also to @tomaarsen for his invaluable advice on training embedding models and Loss functions