
Researchers project that AI labs could exhaust the stock of public human-written text within a few years, and websites are locking their content away. Here is what the data shortage means for businesses that hold unique operational data.
For most of the last decade, the recipe for a better AI model was simple: more compute, more parameters and more data scraped from the open web. The first two can be bought. The third is turning out to be finite. That shift is the single most important reason the market for private, real-world business data exists at all, and why the timing matters so much for anyone who owns some.
By OpenDataPolicies Editorial · October 1, 2026 · 6 min read
The short answer: the public web is a finite resource
Yes, by several credible measures, the supply of high-quality, human-written public text is being used up faster than it is being created. That does not mean AI progress stops. It means the cheapest input to AI progress is getting scarcer, and the labs are looking elsewhere.
The most cited estimate comes from Epoch AI, a research group that tracks trends in machine learning. In a paper published in June 2024, its researchers estimated the effective stock of quality-adjusted, human-generated public text at roughly 300 trillion tokens, with a wide 90% confidence interval of 100 trillion to 1,000 trillion. If training trends continue, they projected that this stock would be fully used somewhere between 2026 and 2032, and potentially earlier if models are "overtrained" well past the compute-optimal point.
Notice the date range. We are now inside it.
"Peak data": the industry is saying it out loud
It is not only outside researchers who see the ceiling. At the NeurIPS conference in December 2024, OpenAI co-founder and former chief scientist Ilya Sutskever told an audience of AI researchers, as reported by The Verge: "We've achieved peak data and there'll be no more." He compared data to fossil fuel. There is, as he put it, "only one internet."
In the same talk he predicted that the next generation of models would be "agentic in real ways": systems that do multi-step work rather than only answer questions. That detail matters for business owners, and we come back to it below.
When a person with that vantage point calls the main raw material finite, the market listens. The practical consequence is that labs and model builders now compete for data that is not already in everyone's training set.
The open web is also closing its doors
Running out is only half the story. The other half is that a growing share of what is on the public web is being fenced off.
A 2024 study from the Data Provenance Initiative, titled Consent in Crisis: The Rapid Decline of the AI Data Commons, audited the websites behind three widely used training corpora (C4, RefinedWeb and Dolma). In a single year, 2023 to 2024, the authors found a sharp rise in restrictions on AI crawlers:
- About 5% or more of all tokens in C4 had become fully restricted through robots.txt.
- 28% or more of the most actively maintained, critical sources in C4 were fully restricted.
- Counting Terms of Service restrictions on crawling, a full 45% of C4 was restricted.
Publishers, forums and media companies have realised their content has value and they want to be paid for it, or at least asked. The result is what the researchers called a decline in the "data commons." The free buffet is shrinking at the same time as demand for it grows.
Why scarcity turns private data into an asset
Basic economics follows. When a free input becomes scarce, buyers start paying for substitutes. That is exactly what has happened: large content holders have signed licensing deals with AI developers (we summarise the publicly reported ones in What AI companies have paid for data).
But the most interesting substitute is not more of the same web text. It is data that was never on the web in the first place: the records of how real businesses actually operate. Think of support tickets and how they get resolved, the email thread that turns a quote into a signed job, the project board that shows how a renovation slipped two weeks, or the accounting entries that reveal a seasonal cash cycle.
None of that is in a public crawl. It sits inside email servers, help desks, CRMs and industry software at millions of companies. To a model builder, it is new signal. To the business that owns it, it is an asset that was invisible on the balance sheet until very recently. Our guide on operational data for machine learning explains what buyers look for in it.
Why the window will not stay open forever
Here is the uncomfortable part. Scarcity creates value, but uniqueness decays. Once a model has learned a pattern, such as how dental offices triage scheduling requests or how an HVAC dispatcher re-routes a crew, the next dataset showing the same pattern is worth less. The first sellers in a category teach the model; later sellers mostly confirm what it already knows.
That is why OpenDataPolicies describes the current period as a limited window. Unique data is king, and whoever holds the keys profits while it is still unique. Three forces are working against people who wait:
- Coverage is filling in. Every licensing deal and every dataset purchase reduces the novelty of similar data.
- Synthetic data is improving. Labs increasingly generate training examples with models. Real-world data is still needed to keep those models grounded (see synthetic data vs real operational data), but synthetic data compresses the premium on generic examples.
- Agents are being deployed now. The workflows that agentic AI will automate are being modelled right now. Datasets that arrive after a workflow is solved command less.
Founder Jeff Cline puts the thesis bluntly: "Every industry is a geek away from being uberized." The data that teaches that "geek's" model how an industry really works is most valuable before the model exists.
What this means for a small or mid-sized business
You do not need to be Reddit or a newspaper group to have data that matters. In many ways a ten-person insurance agency or a regional logistics firm holds something more scarce than a news archive: a detailed, years-long record of a specific kind of work, in a specific industry, done by real people.
According to founder Jeff Cline, small companies have received more than $100,000 for their de-identified operational data, bids can arrive within 72 hours when the data is correct and a header sample is provided, sellers can receive offers from multiple buyers so they can take the highest bid, and deals typically close in about 90 days. Results vary; not a guarantee.
What decides where you land is mostly volume, history, uniqueness and cleanliness. Read how to value my data for the detail, or put rough numbers into the valuation calculator to get a sense of scale.
What buyers do not want, and what we never take
Scarcity does not mean "anything goes." Model builders want operational patterns, not people's private lives, and the legal and reputational risk of personal data makes it a liability rather than an asset.
We never take HIPAA/PHI data or personal customer data. Everything is de-identified before it leaves your control.
In practice that means names, contact details, account numbers and other direct identifiers are stripped or replaced before anything is shared, and healthcare businesses participate only on the administrative side (scheduling, billing operations, workflow) without patient records. Our guide to de-identified data covers the methods.
How to act while the window is open
If you have years of operational records in tools like Microsoft 365, Zendesk, Salesforce or QuickBooks, the steps are straightforward:
- Inventory what you have. List systems, record counts and how many years of history each holds. The data readiness checklist helps.
- Prepare a header sample. Column names and structure only, with no actual records. It is the fastest way to a bid; see the header sample how-to.
- Confirm your rights. Check contracts and vendor terms first; our guide is it legal to sell business data? is a starting point (informational, not legal advice).
- Request a bid. It is free, and you can say no to any offer.
The data shortage is not a forecast anymore; it is a measured trend with a date range we are already living in. The question for business owners is simple: will your operational history be one of the datasets that teaches the next generation of AI, or one that arrives after the lesson is learned?
Request a free bid and find out what your data is worth while it is still unique.
Frequently asked questions
Is AI really running out of training data?
Public, human-written text is finite. Epoch AI estimated in 2024 that the quality-adjusted stock of public text could be fully used between 2026 and 2032 if trends continue. AI will keep improving, but labs increasingly need data that is not on the public web.
Why are websites blocking AI crawlers?
Publishers want control and payment for their content. A 2024 Data Provenance Initiative study found that 28% or more of the most critical sources in the C4 corpus had fully restricted AI crawling via robots.txt within a single year.
Why would AI companies want my business data?
Operational records, such as tickets, email workflows, CRM pipelines and project histories, show how real work gets done. That data is rarely on the public web, so it adds new signal for models and AI agents.
Will my data be worth less later?
Likely, for many categories. Once models learn a workflow, similar data adds less. That is why we describe the current period as a limited window.
Sources
Updated November 19, 2026.
OpenDataPolicies.org connects businesses with buyers of de-identified operational data. We are not a law firm, broker-dealer or investment adviser, and nothing on this site is legal, tax or financial advice. Statements about amounts, timing and outcomes are Jeff Cline's statements based on past transactions. Results vary; nothing here is a guarantee of any bid, price, buyer or closing. Every deal depends on the data, its quality, its rights and buyer demand at the time.