What gets me excited about this project is its legibility for philanthropy, like how the Internet Archive is the destination for random recordings found in Grandma's attic.
Having a trusted steward to handle the anonymization and eval design process could kickstart a world where retiring tax accountants, or even active and generous ones, are willing to donate their data to the posttraining commons at a scale that's actually much larger than the "data industry".
Yeah exactly. I like to joke that I wrote this as a sublimated rant, but it's as much a meditation on why it was impossible to expect it to have been there.
I don't get to break out my academic voice that often these days, so I appreciate the indulgence.
A posttraining data commons and the infrastructure surrounding it is the next great open source project IMO, akin to Postgres or Linux.
The Marin project from Stanford (and I believe now ran by Openathena, the authors of this blog post) is really promising and cool: https://marin.community/
I think a post-training common would have to be somewhat protected, since these things would be created with great effort and would potentially impact people's ability to earn money in their jobs.
I could also imagining them happening as state projects for the sake of competitiveness.
One of the things I'm/we're thinking through is how to do this in a way that’s participatory rather than extractive. It's not necessary that what scientists would choose to automate with agents is the same as what a company trying to automate scientists would choose.
If we can help people automate drudgery while giving them a real say in what gets built and control over how their work is used, I think there’s a possibility of a much better outcome than the current trajectory. I’m not sure we’re in that world, but I think we should try.
I'm not sure I'll do it justice, but one of the general ideas behind open-model initiatives is that the economic impacts of specialized post-training are a fait accompli. OpenAI and Anthropic have spent $many-millions procuring closed posttraining data.
The relevant question here is whether you want open-weight models — projects like Open Athena — to remain or become competitive in the same range of specialties.
NVIDIA has done great work with the open datasets and they're much appreciated, but it's worth mentioning that a large fraction of their post-training data is not released. cf. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1... But at leased they disclosed what wasn't released.
> I also wonder when the NSA starts sharing their wiretaps/etc with labs for 'national security'.
What would this benefit the NSA? If they share wiretap records with labs such as Anthropic, then suddenly anyone who can jailbreak a model gets access to this data. You don't need to train a model on data to understand or interpret it, as far as I know - providing others (esp. AI labs) with sensitive data is a breach of security, or I am not understanding.
I should’ve been more specific, I mean more industrial espionage and getting access to internal data that would be useful to frontier labs for training.
See the well-documented case of NSA conducting espionage against EU’s Airbus to help Boeing.
Wrong direction. NSA says “give us the weights to Fable and loan us some engineers who can get a TS-SCI clearance so we can train our own model in house, if you don’t well just take them, in fact maybe we already did, it’s for national security”.
No, NSA programs are not SCI, haven't been for a few years. They are SAP.
NSA doesn't have to post train Fable. Maven works well enough and that same infra. can be used for Fable. I imagine that's what's going to happen in 2 years once the current admin. moves out.
post training is both the future and the secret sauce.
The next iteration of AI assisted devs will not have all the devs use the same harness and model like most do now.
Instead, devs will build their own post trained models that are experts of their own domain, that have their secret sauce..
I imagine any dev would want most of those models to be something they can self host because of multiple reasons - one of them being that their employers or customers might not agree to use a cloud hosted or managed model. Plus, costs. And control.
Post training is in the future because we haven't yet reached a shared agreement on what that first "good enough" model will be like - and also because the current AI scape is in a state of flux where post training will have a lot of churn.
Hi, I wrote this a few weeks ago. Didn't expect to see this here, etc., etc.
Happy to answer any questions. We have a discord https://marin.community/
What gets me excited about this project is its legibility for philanthropy, like how the Internet Archive is the destination for random recordings found in Grandma's attic.
Having a trusted steward to handle the anonymization and eval design process could kickstart a world where retiring tax accountants, or even active and generous ones, are willing to donate their data to the posttraining commons at a scale that's actually much larger than the "data industry".
A very academic way of saying We have all the data in the world, except the data we actually need. You ask a company how something works and they say:
"Oh, theres a process."
Great. Where is it documented?
"in a spreadsheet."
Which spreadsheet?
"Well, theres the old spreadsheet, new spreadsheet, and the spreadsheet that Karen maintains."
Where's Karen?
"She left in 2021."
Yeah exactly. I like to joke that I wrote this as a sublimated rant, but it's as much a meditation on why it was impossible to expect it to have been there.
I don't get to break out my academic voice that often these days, so I appreciate the indulgence.
A posttraining data commons and the infrastructure surrounding it is the next great open source project IMO, akin to Postgres or Linux.
The Marin project from Stanford (and I believe now ran by Openathena, the authors of this blog post) is really promising and cool: https://marin.community/
I was impressed by Marin's published evals policies: https://github.com/marin-community/marin/issues/9409
Precise and very clearly written.
I think a post-training common would have to be somewhat protected, since these things would be created with great effort and would potentially impact people's ability to earn money in their jobs.
I could also imagining them happening as state projects for the sake of competitiveness.
One of the things I'm/we're thinking through is how to do this in a way that’s participatory rather than extractive. It's not necessary that what scientists would choose to automate with agents is the same as what a company trying to automate scientists would choose.
If we can help people automate drudgery while giving them a real say in what gets built and control over how their work is used, I think there’s a possibility of a much better outcome than the current trajectory. I’m not sure we’re in that world, but I think we should try.
I'm not sure I'll do it justice, but one of the general ideas behind open-model initiatives is that the economic impacts of specialized post-training are a fait accompli. OpenAI and Anthropic have spent $many-millions procuring closed posttraining data.
The relevant question here is whether you want open-weight models — projects like Open Athena — to remain or become competitive in the same range of specialties.
They're already kinda protected, open weight releases rarely come with datasets, especially post-training datasets. NVIDIA is probably the exception.
I also wonder when the NSA starts sharing their wiretaps/etc with labs for 'national security'. Probably already happening, to be honest.
NVIDIA has done great work with the open datasets and they're much appreciated, but it's worth mentioning that a large fraction of their post-training data is not released. cf. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1... But at leased they disclosed what wasn't released.
> I also wonder when the NSA starts sharing their wiretaps/etc with labs for 'national security'.
What would this benefit the NSA? If they share wiretap records with labs such as Anthropic, then suddenly anyone who can jailbreak a model gets access to this data. You don't need to train a model on data to understand or interpret it, as far as I know - providing others (esp. AI labs) with sensitive data is a breach of security, or I am not understanding.
I should’ve been more specific, I mean more industrial espionage and getting access to internal data that would be useful to frontier labs for training.
See the well-documented case of NSA conducting espionage against EU’s Airbus to help Boeing.
Wrong direction. NSA says “give us the weights to Fable and loan us some engineers who can get a TS-SCI clearance so we can train our own model in house, if you don’t well just take them, in fact maybe we already did, it’s for national security”.
No, NSA programs are not SCI, haven't been for a few years. They are SAP.
NSA doesn't have to post train Fable. Maven works well enough and that same infra. can be used for Fable. I imagine that's what's going to happen in 2 years once the current admin. moves out.
post training is both the future and the secret sauce.
The next iteration of AI assisted devs will not have all the devs use the same harness and model like most do now.
Instead, devs will build their own post trained models that are experts of their own domain, that have their secret sauce..
I imagine any dev would want most of those models to be something they can self host because of multiple reasons - one of them being that their employers or customers might not agree to use a cloud hosted or managed model. Plus, costs. And control.
Post training is in the future because we haven't yet reached a shared agreement on what that first "good enough" model will be like - and also because the current AI scape is in a state of flux where post training will have a lot of churn.