Common Crawl is a non-profit foundation dedicated to the Open Web.
San Francisco, CA
Joined February 2010
- Tweets1.5K
- Following1.6K
- Followers7.8K
- Likes706
Findings of the WMT 2026 Shared Task of the Open Language Data Initiative
Laurie Burchell Common Crawl Foundation
www2.statmt.org/wmt26/pdf/20…
We present the results of the WMT 2026 shared task of the Open Language Data Initiative (OLDI). Participants were invited to contribute to the massively multilingual open datasets supported by OLDI, FLORES+ and OLDI Seed, or to create new resources in line with OLDI’s ethos. We accepted ten submissions: five extending FLORES+, and five extending other open and massively-parallel datasets (BOUQuET, Seed, SMOL, and WMT24+). These contributions advance the coverage and quality of multilingual datasets, especially for underserved language varieties. All contributions are released under permissive open-source licenses.
Common Crawl Foundation retweeted
My bet: low-resource languages would lose at the filter.
Quality classifiers trained mostly on English throw out good Icelandic pages as noise. Smarter crawl budgets won't help if the filter can't read the language.
arxiv.org/abs/2610.13321
Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages.
We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data.
Common Crawl - the ultimate time capsule for knowledge | Amazon Web Services
youtube.com/watch?v=rGT-al-F…
Common Crawl Foundation retweeted
we will have two papers @NeurIPSConf this year from the croissant community 🥐
1. "croissant baker" led by rafi al attrach and @tompollard
2. "croissant miner" led by berke arda and @akhtarmubashara
i took a quick glance at the rearview mirror
since early 2023, our idea for a unified ml data access layer has grown into an expansive ecosystem, incl
- data aggregators spanning hundreds of thousands of datasets incl @huggingface @kaggle @dataverseorg @CommonCrawl and others
- tooling spanning auto-generation and -verification of croissant files
- the formal spec maintenance under @MLCommons
- and a growing list of community extensions spanning verticals such geo, responsible AI, health and others
onwards towards the composable data dream(:
Common Crawl Foundation retweeted
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
ALT Flow diagram of the Marin data pipeline. Starting with 25.25 trillion tokens, deduplication removes 2.13 trillion and benchmark decontamination removes another 13.66 billion, leaving 23.11 trillion. Documents are organized into 40 topics and five quality bands, yielding 200 sampling groups. Results from 872 small-model experiments feed a regression used to choose sampling weights.
Information Stewardship Forum 2026: Creating Community and Purpose Around US Government Information
A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 people working to preserve and provide access to United States government information.
