Short answer: AI search tools can only use information they are allowed and able to reach. That includes their own index, live web browsing, and any connectors a user has authorized. Content that is unindexed, generated on demand, behind a login, or on the dark web is usually out of reach. When a tool can’t reach the source, it may still produce a confident but wrong answer. This infographic from the Association of Internet Research Specialists (AOFIRS) explains these layers and shows how to verify what AI tells you.
Learn more, watch a video on “Hidden Web vs AI Search”. Read more on “The Invisible Web in the Age of AI” or read the research report “The Hidden Web in the Age of AI Search”.
What the infographic shows
The iceberg splits the web into what search engines have indexed (above water) and everything they haven’t (below). Below the surface, the graphic shows the deep web (paywalls, databases, academic resources) and a much smaller dark web. It also shows two structural blind spots, orphan pages and query-generated content. The lower panel explains the risk of “confident hallucinations” and why AI citations should be treated as leads to check, not as established facts.
Four kinds of “hidden” information and how they differ
1. The indexed web
Pages a search engine has discovered, crawled and chosen to store. Google says it does not guarantee that it will crawl or index every page it finds. Being public is not the same as being indexed.
2. Unindexed and dynamically generated information
Results that only exist after you search inside a site (court dockets, product catalogs, statistical tables), pages excluded with robots.txt or noindex, and orphan pages that nothing links to. These pages are often freely available but you have to visit the site and search it directly. For more on why professional researchers still need these sources, read The Invisible Web in the Age of AI.
3. Login-protected and subscription resources
Library databases, journal archives, news paywalls, company intranets and personal accounts. Crawlers can’t log in, so most AI tools can’t see this content unless the publisher shares it or you connect your own authorized account.
4. The dark web
Sites reachable only through special software such as Tor onion services. It is a small part of the hidden web, and it is not the same as the “deep web”. Most deep web content is ordinary and legal: your email, your bank, a library catalog.
Where AI tools fit
AI tools differ. Some answer from training data alone. Others search a web index in real time, and some can use connectors you authorize, such as a linked drive or mailbox. None can legitimately get around logins, paywalls or access controls. Check each tool’s documentation to see what it can actually reach. Our video Hidden Web vs AI Search walks through these differences.
Comparison: access limits and research approaches
| Information type | Typical access limitation | Appropriate research approach |
|---|---|---|
| Indexed web | Ranking and freshness; may be summarized inaccurately | Advanced operators, multiple engines, open the original page |
| Unindexed / dynamic | Only generated after a site search or form query | Use the site’s own search, official portals and directories of databases |
| Login / subscription | Authentication, licensing, paywalls | Library access, institutional subscriptions, publisher open-access versions, authorized connectors |
| Dark web | Special software; legal and safety risks | Only with a lawful purpose, organizational approval and specialist training |
| AI-tool answers | Depends on the tool’s index, browsing and connectors; may fabricate | Treat citations as leads; confirm every claim at its source |
What the AI citation-error figures really mean
The 37%, 67% and 94% figures come from a test by the Tow Center for Digital Journalism, published in the Columbia Journalism Review on March 6, 2025. Researchers gave eight AI search tools excerpts from news articles and asked them to identify the original headline, publisher, date and URL. Perplexity answered 37% of queries incorrectly, ChatGPT Search 67% and Grok 3 94%. The tools rarely admitted they didn’t know.
Limits: the study tested news-source identification at one point in time, and tools change often. It doesn’t measure overall accuracy or hidden-web access. Its lesson is that AI citations can sound precise and still be wrong. For a deeper analysis of deep-web access and AI search, see the AOFIRS research report The Hidden Web in the Age of AI Search.
Practical implications for researchers
- Use AI to plan searches and find leads, not as the final source.
- Open every cited link. Confirm that the page exists and actually says what the AI claims.
- For specialist data, go straight to databases, official portals and library resources.
- Record where each fact came from and when you checked it.
Illustrative research example
Illustrative scenario, not a recorded test. A researcher asks an AI tool for a city council’s 2025 zoning decision on a specific address. The tool gives a confident summary and a link. The link leads to a general news page that doesn’t mention the address. The researcher then opens the council’s meeting-minutes portal, searches by date and address, and finds the actual decision. It differs in one key condition. The AI couldn’t reach the query-generated minutes, so it filled the gap with a guess. Going to the source fixed it.
Learn more from AOFIRS
- The Invisible Web in the Age of AI (article)
- The Hidden Web in the Age of AI Search (research report)
- Hidden Web vs AI Search (video)
- The AI Trust Illusion: Why It Sounds Right Even When It’s Wrong
- CIRS certification: build structured online research skills
Frequently asked questions
Is the hidden web the same as the dark web?
No. The hidden (deep) web is everything search engines haven’t indexed, and most of it is ordinary and legal. The dark web is a small part that needs special software such as Tor.
Can AI tools search paywalled or login-protected databases?
Not on their own. They can only use such content if the publisher makes it available or you connect an account you are authorized to use.
Is 96% of the web really hidden?
That figure isn’t supported by current primary research. Large parts of the web are unindexed, but the exact share is unknown.
How do I check an AI citation?
Open the link, confirm the page exists, find the exact passage, and compare its date and author with what the AI said. If the link fails, search the publisher’s site directly.
Sources
- Jażwińska, K. & Chandrasekar, A. (2025, March 6). AI Search Has a Citation Problem. Columbia Journalism Review, Tow Center.
- Google Search Central. In-depth guide to how Google Search works.
- Google Search Central. Block search indexing with noindex.
- Tor Project. Onion services.
