The article suggests Common Crawl as a replacement which probably doesn’t make sense on its own due to low update frequency (monthly) and somewhat limited crawl scope.
(Looks like the sentence following the suggestion addresses this somewhat.)
An LLM does not continuously index the Internet and is therefore stale the second it starts training. You need to use something like RAG. And the R in RAG needs an index.
This was announced in January.
https://programmablesearchengine.googleblog.com/2026/01/upda...
The article suggests Common Crawl as a replacement which probably doesn’t make sense on its own due to low update frequency (monthly) and somewhat limited crawl scope.
(Looks like the sentence following the suggestion addresses this somewhat.)
Good. Duck google and use other indexing.
ps. Modern AI is in reality talking web search engine imo.
An LLM does not continuously index the Internet and is therefore stale the second it starts training. You need to use something like RAG. And the R in RAG needs an index.
I just see another small competitor becoming big by taking over
Is this good for Kagi?
Does Kagi have a free developer api?