Upvoting to encourage discussion of these differentiators:
"Apertus is a 70B and 8B parameter language model designed to push the boundaries of fully-open multilingual and transparent models. The model supports over 1000 languages and long context, it uses only fully compliant and open training data, and achieves comparable performance to models trained behind closed doors."
"pretrained on 15T tokens with a staged curriculum of web, code and math data"
"open weights + open data + full training details including all data and training recipes"
"Apertus is trained while respecting opt-out consent of data owners (even retrospectivey), and avoiding memorization of training data"
Their struggle with Nvidia driver bugs they had to work around was very relatable. You'd think if someone buys 10,752 of their high-end GPUs you'd get some support with it.
Agreed, but the problem seems to be even worse with AMD from what I hear, or at least it was when I checked with some of my HPC buddies a little over a year ago. Constant driver bugs and crickets from upstream "support".
In my opinion, we need more models trained on fully traceable and clean data instead of closed models that we later find out were trained on Reddit and Facebook discussion threads.
I want to see something trained _only_ on stuff like encyclopedias, programming books, etc. I'm interested in how different it would be compared to something with a lot of social media in it.
Does their training corpus respect copyrights or do you have to follow their opt out procedure to keep them from consuming your data? Assuming it’s the latter, it’s open-er but still not quite there.
Afaik they respect robots.txt on crawl and later when using the data they re-check the robots.txt and will exclude the data if the new robots.txt was updated to deny access. They have further data filtering bit for that you better check the technical report.
> Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting robots.txt exclusions and filtering for copyrighted, non-permissive, toxic, and personally identifiable content.
Imagine regulators doing their job for once and creating a clean regulation that removes the uncertainty about the liability for such releases. Such that they can just slap Apache or MIT on it and call it a day and don't require to collect personal data to comply with the "acceptable use policy".
I want and hope this to succeed. But the tea leaves don't look good at the moment:
- model sizes that the industry was at 2-3 gens ago (llama 3.1 era)
- Conspicuous lack of benchmark results in announcements
- not on openrouter, no ggufs as yet
Really happy to see this and will give it a good spin. They seem to be doing things the right way in my subjective opinion:
""" To implement this filter, we begin by ranking URL domains according to the volume of
texts they contribute to the FineWeb (Penedo et al., 2024a) and FineWeb-2 (Penedo et al.,
2025) corpus, as an approximation of web-level English and multilingual data. From this
ranking, we select the top one million English domains and the top one million non-English
domains. Due to domain overlap and the fact that some sites are now offline, the total
number of accessible robots.txt files is smaller than two million. For each domain that
remains reachable, we retrieve its robots.txt file as of January 2025 and examine the
directives relevant to AI training. In particular, we focus on those targeting the AI-specific
user agents listed in Appendix A. Any contents blocked by the current robots.txt is
removed retroactively from the entire 2013-2024 range of the training dataset. We follow
an opt-out policy, that is, if the corresponding robots.txt files are not available, we
consider the data usable for training. The filtering process results in an estimated token
loss of approximately 8% in English data and 4% in multilingual data.
"""
Very cool. Love this. Was the training more heavily weighted towards swiss languages and how does the model perform on swiss languages compared to others?
Are there any plans for further models after this one?
“The file reflects data protection deletion requests which have been addressed to SNAI as the developer of the Apertus LLM. It allows you to remove Personal Data contained in the model output. We strongly advise downloading and applying this output filter from SNAI every six months following the release of the model.”
I can't imagine that this actually complies with the law.
This is why predictions of eventual AI failure due to copyright lawsuits are likely wrong.
Assumedly, an organization training and then distributing this model cannot be stopped via copyright or breach of contract lawsuit. It may be that folks will figure out copyright-free versions of text-to-image and text-to-video, etc., models as well.
It seems that there is plenty of copyright-free data available to train a useful model. Therefore, when content creators upset about AI companies training models on their content are asked about this model, they have nothing to do but shrug.
32 comments
[ 2.8 ms ] story [ 66.3 ms ] thread"Apertus is a 70B and 8B parameter language model designed to push the boundaries of fully-open multilingual and transparent models. The model supports over 1000 languages and long context, it uses only fully compliant and open training data, and achieves comparable performance to models trained behind closed doors."
"pretrained on 15T tokens with a staged curriculum of web, code and math data"
"open weights + open data + full training details including all data and training recipes"
"Apertus is trained while respecting opt-out consent of data owners (even retrospectivey), and avoiding memorization of training data"
> "open weights + open data + full training details including all data and training recipes"
Is it reproducible?
> respecting opt-out consent of data owners (even retrospectivey)
Were they notified and given an option to opt out? Owners and authors are not the same. Data owners aren't copyright owners either.
> avoiding memorization of training data
Not convincing.
Key features
Fully open model: open weights + open data + full training details including all data and training recipes
Massively Multilingual: 1811 natively supported languages
Compliant: Apertus is trained while respecting opt-out consent of data owners (even retrospectivey), and avoiding memorization of training data
> Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting robots.txt exclusions and filtering for copyrighted, non-permissive, toxic, and personally identifiable content.
- model sizes that the industry was at 2-3 gens ago (llama 3.1 era) - Conspicuous lack of benchmark results in announcements - not on openrouter, no ggufs as yet
""" To implement this filter, we begin by ranking URL domains according to the volume of texts they contribute to the FineWeb (Penedo et al., 2024a) and FineWeb-2 (Penedo et al., 2025) corpus, as an approximation of web-level English and multilingual data. From this ranking, we select the top one million English domains and the top one million non-English domains. Due to domain overlap and the fact that some sites are now offline, the total number of accessible robots.txt files is smaller than two million. For each domain that remains reachable, we retrieve its robots.txt file as of January 2025 and examine the directives relevant to AI training. In particular, we focus on those targeting the AI-specific user agents listed in Appendix A. Any contents blocked by the current robots.txt is removed retroactively from the entire 2013-2024 range of the training dataset. We follow an opt-out policy, that is, if the corresponding robots.txt files are not available, we consider the data usable for training. The filtering process results in an estimated token loss of approximately 8% in English data and 4% in multilingual data. """
It’s easy to become jaded with so many huge models being released, but the reality is they are still from a relatively small group of countries.
For example India has no indigenous models this big despite having a world class talent pool.
the full collection of models is here: https://huggingface.co/collections/swiss-ai/apertus-llm-68b6...
PS: you can run this locally on your mac with this one-liner:
pip install mlx-lm
mlx_lm.generate --model mlx-community/Apertus-8B-Instruct-2509-8bit --prompt "who are you?"
Are there any plans for further models after this one?
I can't imagine that this actually complies with the law.
Assumedly, an organization training and then distributing this model cannot be stopped via copyright or breach of contract lawsuit. It may be that folks will figure out copyright-free versions of text-to-image and text-to-video, etc., models as well.
It seems that there is plenty of copyright-free data available to train a useful model. Therefore, when content creators upset about AI companies training models on their content are asked about this model, they have nothing to do but shrug.
The cat is out of the bag.