The AI Web Revolution: How Bots Are Reshaping Discovery and the Future of Websites

The web as we know it is on the cusp of a significant transformation, driven by the proliferation of Large Language Model (LLM) AI bots. A recent Hacker News discussion delved into the potential ramifications of these bots becoming users' primary interface, largely bypassing traditional websites. The original poster (OP), who runs a book discovery website, shared a personal concern: a massive drop in Google traffic as AI-powered search results take precedence, prompting consideration of a mobile app to maintain discovery.

The Shifting Sands of Web Discovery and Monetization

Commenters largely agreed that the fundamental commercial nature of the web is too lucrative to disappear. However, the ways users discover and interact with content are set for a major overhaul. Key predictions include:

• Decline of Traditional Search: Several participants, including the OP, noted that traditional search engines like Google might become less relevant. One user shared an anecdote of using ChatGPT to find a specific product (a 58-inch coffee table) after Google and e-commerce site searches failed, highlighting AI's potential for highly specific queries.
• Rise of Walled Gardens: A prominent theme was the expectation that the web will become more fragmented into 'walled gardens.' This move would aim to reduce competition, make content harder to scrape or copy by AI models, and increase the barrier to entry for new players, making the web even more 'pay-to-play.'
• New Discovery Channels: With traditional search potentially waning, discovery is likely to shift. Predictions include:
  • AI-driven recommendations: Users might directly prompt AI for information or recommendations, bypassing websites.
  • Community-centric discovery: A return to more curated content shared within communities, similar to pre-Facebook forums, but now on platforms like Slack channels or Discord groups.
  • Revival of curated platforms: Mentions of StumbleUpon and the potential comeback of Digg suggest a desire for alternative discovery mechanisms beyond algorithmic feeds.
• Monetization Models: While not deeply explored, the OP questioned if 'Monetized MCP (Content Provider?) servers' could become a new version of websites, hinting at evolving business models.

Impact on Websites and Creator Strategies

The consensus is that websites won't disappear entirely, but their role and how they attract users will change. The OP's experience with their book discovery site, Shepherd.com, losing Google traffic underscores this challenge. Strategies discussed and implied include:

• Focus on Direct Audience: Building a loyal user base that visits directly will be crucial, as relying on search engine referrals becomes less viable.
• Mobile Apps: The OP is considering a mobile app, reflecting a belief that app stores might become critical discovery points, offering a more controlled environment.
• Leveraging AI Internally: The OP mentioned using AI as a tool for features like 'book twins' for personalized recommendations and exploring prompt-based search on their own site.
• Adapting to Niche Expectations: A detailed exchange between a commenter (eimrine) and the OP (bwb) regarding Shepherd.com highlighted the challenges of catering to specific user needs. Criticisms around book categorization, depth for specialist readers (e.g., academic math vs. popular math books), UI design (memory usage, information density), and perceived focus on new books (which the OP clarified was not the case) underscore the importance of understanding and serving a target audience effectively. This debate also touched on the ethics of content consumption (piracy vs. supporting authors).

Worries and Hopes for the Future Web

While concerns exist about websites being 'destroyed' and the difficulty for new projects to gain visibility ('Show HN' but then what?), there are also hopeful perspectives:

• Reduced 'AI Slop': A hope was expressed that this shift might help clear the web of low-quality, AI-generated content.
• Prominence of Other Protocols: Disruptions could lead to a 'fracture or two' in the internet, potentially allowing other protocols to gain prominence and keep the internet useful and diverse.

The overarching sentiment is that the web is entering a period of significant flux. Adaptability, community building, and exploring new avenues for discovery will be key for content creators and website owners navigating this AI-driven future.