Towards a fair, thriving Web in the age of AI

In this issue:Recognyze.

Towards a fair and thriving Web

The explosive success of Large-Language-Model chatbots has been made possible by the enormous wealth of information provided by the World-Wide Web — an estimated 50 billion pages — but in return threatens the Web. Traffic to Web sites has been declining, as many people prefer to go directly to LLMs which aggregate information from various sites. 

This growing phenomenon threatens not only the Web but LLMs themselves: if they cease having solid information from which to build their knowledge base and their answers, they will end up just recycling “AI slop”.

There has to be an incentive for Web content creator to continue the hard work of enriching their sites with high-quality, verified material. The informational Web never really found its financial model: advertisement demeans and sabotages good sites (who really wants to have his top-quality material sitting next to get-refunded-for-solar-panels scams?), and in most cases draws minuscules proceeds anyway.

The advent of LLMs holds the promise of a much more coherent and sustainable model. Content aggregators (such as the major LLM companies) charge their users for the information and help they provide. Content creators are entitled to their share.

Recognyze AI has developed a business model and technology which makes this solution rewarding and realistic for all parties, including Web content owners (“publishers”), the authors of the corresponding content (“authors”) and LLM companies (“aggregators”).

Between publishers and aggregators, Recognyze AI acts as a clearinghouse, monitoring aggregator access to publisher content and billing aggregators at the end of each month. The fees can be set by publishers, on a page-by-page basis, with default values which have been calculated to represent a sustainable charge for aggregators, commensurate with their business model and respectful of the benefit they get from being able to scrape publisher content.

If you have a Web site, a blog or any other Internet resource of potential interest to aggregators, you can register it with Recognyze by just going to https://recognyze.ai and signing up as a publisher. The process is straightforward and free.

Along with publishers (content owners), Recognyze also takes into account authors, the people who actually produce that content. (As a simple example, think of a newspaper as the publisher and individual journalists as authors, but the distinction exists in many other settings.) Any financial stake for authors is between them and their publishers, but Recognyze AI highlights the reputational value which is so important for authors: when aggregators use their work, they should be acknowledged.

Recognyze has developed the AI Index, which serves precisely that purpose. A particular challenge was the wide variation of web sites' appeal, all the way from a small site directed at a limited specialist community to a site with millions of views per day. Using a double-log formula, the AI Index (a value between 0 and 1000) handles this variation, making it possible even for a fairly small site to detect measurable growth over weeks and months.

If you are an author, you can register your work (also at https://recognyze.ai) and list the works that should be tracked. You should also encourage your publisher (your content owner, if that's not also you) to register as a publisher. If you are both an author and a publisher, you do not need separate accounts; just open one account and change roles on the site as needed.

To both registered publishers and registered authors, Recognyze provides (for free) powerful graphical analytics enabling them to monitor closely access to their sites by LLMs and other aggregators. This benefit alone justifies registering.

The technology is in place but still fresh, which is why Recognyze have not gone full-scale yet in publicizing it; this newsletter is one of the first public announcements. If you encounter any problems or have suggestions feel free to share them.

It is in everyone's interest — AI companies, Web publishers, Web authors — to support the healthy continued development of Internet resources providing a wealth of high-quality information. Recognyze AI is the financial and reputational mechanism that fuels this development for the benefit of all. Whether you are a publisher, an author or an AI developer, get involved!

Cover photo: Sea lions, Santa Barbara harbor.