Sometimes I get into arguments about AI and people comment weird stuff here on Lemmy. Like we have ethical AI models (meaning they’re not made up of unethically sourced training material). Or we got plenty “libre” or “free and open-source” AI models.
Am I missing something? Or is this just a bunch of people trying to bullshit me? Is anyone here aware of such large language models and would be so kind as to educate me? I mean we do have a ton of open-weights models. But proper open-source or ethical??? I’ve been following the news for quite a while and I’m aware of a very select few I’d call LLMs with training datasets available. I’m aware of:
- Pythia by EleutherAI (up to 12B, from 2023) and Dolly 2 by Databricks (12B, also from 2023)
- BLOOM by BigScience (176B, from 2022)
- GPT-2 by OpenAI (1.5B, from 2019)
- The recreation of LLaMA by the RedPajama-Project
- Apertus by the ETH Zürich
Edit: In the comments we got:
- LLM360 with K2-65B, Crystal-7b, Amber-7B
- the Soofi-Project
- OLMoE
That list needs a lot of footnotes, though. Some of them come with restrictions, terms and conditions… And none of them can do modern tasks like AI-assisted coding. The only one getting close is the Swiss model. Sadly the earlier version I tried didn’t perform well. Seems they released an updated v1.5 version this summer, I wonder if that’s better and anybody uses it for real-world applications.
So… Back to my initial question: Are there models out there with datasets available? I mean except the exactly one somewhat usable one I already found?
IIRC OLMoE might fit that definition
Looks fine, thanks!
LLM 360 probably deserves a mention there, trying to be open, reproducible and publishing intermediate training checkpoints.
meaning they’re not made up of unethically sourced training material
I am old enough to remember the days before ‘piracy’ was considered unethical. I still stand by that. Authors need to be compensated but also “publication” is supposed to give some rights to the “public”. Sharing non-commercially things you love over internet used to be a gray zone that should have been the subject of a political debate. Instead, right holders decided that the letter of the law should strictly apply to conditions it was never designed for. I have little empathy for them moaning now that AI training, while not formally illegal, should be considered unethical.
A copyright reform is long overdue. Let’s discuss it seriously instead of playing the game of “who can lobby the most?”. In the meantime, enjoy the gray zone that refusing to update 19th century laws to the century of internet causes.
I was trying to reply to this, bringing up the controversy that erupted when image generators were able to use prompts with artist names to simulate their style, and how that shows how blurry the copyright line is. As I wrote it, I realized how complicated the issue really is, and how little I understand current copyright laws. Some may argue there isn’t any vagueness, an artist’s work in any format is theirs and shouldn’t be copied for other use, training or otherwise. But we all use other people’s work in our own, don’t we? There is no solid line where one is ethical and one isn’t. Plagiarism without proper attribution has always been wrong, but if you take the same content and tweak it a bit, it’s not. LLMs can and do that, through statistical randomness, while using someone else’s material, just like human brains would.
The compensation issue is definitely a problem and where the ethics come into play. Even if many people freely read a book in a library, it has been bought at some point. But how do you determine who was unfairly “stolen” from? Isn’t it basically everyone at this point?
I know I’m way too nitpicky… But I feel I should point out there’s some confusion hidden here…
There’s stuff which is moral to do. And stuff which is legal to do. Those aren’t the same! When talking about who can mention what, what copyright law demands people to do and those blurry lines, we’re concerned with legality. That doesn’t have anything to do with ethics. (At least not directly.)
Compensation is actually the only issue. If they were fairly paid, most artists would be ok with people enjoying their works in any way they can.
Thanks! Yes, definitely deserves a mention.
Looks like they go over their data sets here:
https://www.sciencedirect.com/science/article/pii/S1877050925006210
https://huggingface.co/togethercomputer
Hopefully someone else knows more, I don’t really care, I just want Europe to catch up to China/America otherwise it won’t matter how ethical the LLM is because everyone will be using Chinese/American anyway like 99% of people already are
I see Olmo by AllenAI frequently used in research. The publish all the checkpoints of training and all training sources. https://allenai.org/olmo / https://huggingface.co/collections/allenai/olmo-3
Here’s the link to the dataset used to train Olmo 3: https://huggingface.co/datasets/allenai/dolma3_mix-6T
It’s fairly modern. Olmo 3.1 was released January 2026.
Awesome, I thought there were none.
I recall OpenAI saying to a courtroom judge that modern AI requires training off copyrighted material. And obviously the AI labs cannot just release those.
Distlled models (aka models that train themselves off the responses from other AI models) could probably release all the AI responses they used for training, since afaik AI generated content is not copyrightable…
However I suspect that most modern models are built from previous models. For example Claude Fable is probably based on Claude Sonnet or something. And for a downstream model to be fully “open source” and reproducible, you would need to reproduce the upstream model too. So even if a downstream model didn’t use copyrighted data (eg it only used distills), one of the upstream models likely did.
They could still release a manifest of hashed contents and metadata.
What would be an application for this? More transparency for court cases? Or enable people to search for an alternative source for the data, like a torrent?
It would allow users to audit the dataset for ethical concerns, to interrogate the set to understand what a model does and does not “know”, to identify weaknesses and biases, and to ensure that nothing has been inserted specifically to engineer prompt responses. Just examples off the top of my head.
Knowledge should be relatively impossible, though, if you just get the names and hashsums of the files? I mean I guess you could look up if a specific file you already have is in it. Might have some application. It needs to be very detailed, though. If they just call it GitHub-dump-timestamp or Wikipedia-dump-2026 with some overall hash, that doesn’t tell us anything.
Most resources have a URI, Universal Resource Identifier. Related to a URL, except that it doesn’t necessarily tell you how to locate the resource it identifies, URIs have dozens of schemas (eg isbn: for books) that can identify virtually any kind of media. Ideally they would tell us the URI, all metadata, source/provenance/licensing information, and the hash of the resource. If they truly are training on anonymous unattributed blobs that they just mysteriously found in their dataset, well that would be interesting to learn.
Uh, that one looks nice. Thanks!
Nemotron is a fairly significant model family in this space, IIRC. e.g. here’s one of the recent releases: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Edit: Not fully open – see thread below.
Huh. It doesn’t come with all the datasets, though. Does it? The model card contains a long list of “Private Non-publicly Accessible Datasets”, both by NVIDIA and by third parties. And some of the stuff just says “Undisclosed”.
“Private Non-publicly Accessible Datasets”
Huh. I thought they were fully open. I missed that. Most of the data sets and methodology is open though. (Maybe I’m remembering an older model? Or maybe I just fell for their marketing a while back… 😔️)
NVIDIA’s own proprietary stuff looks largely like machine translation examples, and some sort of game logs (Sudoku and Minesweeper 🤨️), plus a few things I’m not quite sure I understand (“Multitool Typewriter Hard”? Is that for OCR maybe? Edit: or maybe they mean ‘multitool’ in an agentic sense… and that’s for tool call reasoning?).
The third-party stuff is these six entries:
- Global Regulation
- TAUS Translation Memory
- Scale HLE
- HackerRank Coding
- RL data for Search
- Mercor SWE-AgentsV1
I wonder how hard it would be to rebuild something equivalent that is fully open. 🤔️
Sudoku and Minesweeper could very well be some internal project names and have nothing to do with what the words mean. I have no clue.
Seems to me their announcements ooze with marketing speech. I bet they put it all in one sentence to mislead about the datasets. Put a comma there and omitted the word “partly”. It’s technically borderline correct. I’ll have to read up on it, whether that’s a good amount of data, or just some tiny fraction. It’s not looking good, though.
It could be, but using the games as logical puzzle solver/reasoning tests might have value – I don’t know.
Many of the Chinese LLMs have open source models, even frontier models.
Open source or open weights?
Please read the whole post. OP is asking for the datasets used to train the models.
Who cares? Might as well so how many organic grass fed Nazis there are. FOSS doesn’t clean off the stink of AI
Your brain on Lemmy
If the dataset is ethical and can be inspected, and the model is self-hosted, what is your objection
I don’t think this is an easy debate to have. Even those kinds of models have effects on society, on people, on the environment. A lot of resources are needed for research and training. Same for inference even if you own the computer. And we’d need to agree on what’s ethical. Usually that means treat the entire world like a self-service outlet and just take everything unless someone calls you and explicitly said no. At that point it’ll become a nasty technical problem because there’s no easy way to get rid of data after the fact.
Models could in theory be ethically sourced, but used for evil. Which is gonna be a problem to society. And then all of the debate is hypothetical because that’s not even what’s in use out there…
My post was more concerned with the facts. What kinds of models we do have. Not what we should do or an universally accepted definition of ethical.
An inspectable dataset would allow users audit according to their own project’s values and obligations. That’s the only standard that matters. I don’t yet see the need to bring a discussion of universal ethics into this.
Datasets containing unlicensed, stolen, and even privileged information have been used for major commercial models. This is a concrete problem for some people, and open source datasets address directly address it.
You tangentially bring up a bunch of other stuff but don’t appear willing to defend a position on any of it. I don’t really understand your point. You can speculate about what Jesus’s objection might be, but if you’re not willing to bring forward your own objections, I’m not sure what you’re trying to accomplish at this point.
IMHO, the problem with AI isn’t AI, it’s that we are permitting states and obscenely wealthy individuals to engage in an extremely wasteful winner-takes-all arms race. IMHO, there is nothing wrong with private individuals exploring LLMs and other new AI technologies, and self-hosted open source models are the only way we get to participate and potentially protect ourselves.
