Close Menu

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    How Water Collects on Flat Streets Near the Harbor

    November 20, 2026

    What it means for your traffic

    November 20, 2026

    How Often Do Bonds Beat Stocks?

    November 20, 2026
    Facebook X (Twitter) Instagram
    Trending
    • How Water Collects on Flat Streets Near the Harbor
    • What it means for your traffic
    • How Often Do Bonds Beat Stocks?
    • Why Black Friday and Cyber Monday Are Bad Days to Impulse Buy Electronics
    • You Don’t Need Cash to Buy a Competitor
    • Anthropic’s IPO could deliver $156 billion to charity
    • Sweden’s H100 Lines Up Bitcoin Trading Revenue for Share Buybacks – BitRss
    • Nurses Share The Worst Mistakes They’ve Ever Seen
    Facebook X (Twitter)
    SBM Global News
    Demo
    • Home
    • Top Stories
      • Politics
    • Business
      • Small Business
      • Marketing
    • Finance
      • Investment
    • Technology

      Anthropic’s IPO could deliver $156 billion to charity

      November 20, 2026
      Read More

      Biz4Group LLC – Company Profile

      November 19, 2026
      Read More

      Nvidia’s startup stakes ballooned to $65B, but Huang’s exit puts future deals in doubt

      November 18, 2026
      Read More

      Authenticom, Inc. – Company Profile

      November 17, 2026
      Read More

      Island is a new screen-time app that turns staying off your phone into a game

      November 17, 2026
      Read More
    • Lifestyle
      • Travel
    • Feel Good
    • Get In Touch
    SBM Global News
    Demo
    Home»Marketing»AI Bots — Who is Blocking and Why?
    Marketing

    AI Bots — Who is Blocking and Why?

    By Staff WriterAugust 28, 20242 Mins Read
    Facebook Twitter LinkedIn Reddit Email
    #image_title
    Share
    Facebook Twitter LinkedIn Pinterest Email

    I covered some of the potential arguments either way in my previous post, but the truth is that right now looking at how little traffic these models are driving, it’s probably not hugely impactful in the short term. If you look at Moz’s robots.txt file at the time of writing, you can see we block GPTBot from our learn center and blog – this is a compromise position, but one which we haven’t really seen any benefit or harm from so far, and nor would we expect to in the short term. I certainly don’t think the comparison to blocking Googlebot is fair – LLMs are primarily a content generation tool, not primarily a traffic referral tool. Indeed, Google has suggested that even their AI Overviews are not affected by Google-Extended, but instead by regular Googlebot. Similarly, at the time of writing OpenAI has just announced their direct Google competitor “SearchGPT,” and also confirmed that, like Google, it is crawling with a separate user agent to other generative AI tools – in this case, “OAI-SearchBot.”

    What I didn’t cover in that article is the case of large publishers. If you are a large publisher and you do think you have leverage, and may be able to strike a deal, you may wish to set a precedent – that these tools are not owed free access unless they reach a formal arrangement. For example, The Verge’s parent company, Vox Media, publicly said they were blocking access before eventually striking a deal. The robots.txt file on theverge.com still explicitly blocks most other AI bots, but not (anymore) GPTbot.

    Of course, the majority of sites and the majority of readers of this blog post are not large publishers. It may well be significantly more valuable for you to be mentioned in AI-written content than it is for you to try to protect the unique value of your content, particularly in a crowded market of competitors with no such qualms. Still, it’s interesting to see the precedents being set here, and it will be even more interesting to see how it plays out.

    View original article here

    Share. Facebook Twitter LinkedIn Email Reddit
    Previous ArticleLiving The Luxury Q Life at Quivira Los Cabos
    Next Article The 1 Thing You Should Never, Ever Do If You Get A Mosquito Bite

    Related Posts

    What it means for your traffic

    November 20, 2026
    Read More

    You Don’t Need Cash to Buy a Competitor

    November 20, 2026
    Read More

    How to track your brand’s visibility in AI search [2026]

    November 19, 2026
    Read More
    Add A Comment

    Leave A Reply Cancel Reply

    Demo
    Top Posts

    Former FBI, CIA Head Has ‘Serious Concerns’ With Trump Cabinet Picks

    December 28, 2024435

    Emirates to operate next-gen A350 on the third daily service to Cape Town

    January 14, 2026256

    AAVE Price Prediction: Target $215-225 by Mid-January 2025 as Technical Indicators Signal Bullish Momentum

    December 15, 2025240

    Wall Street Slumps As Alphabet, Qualcomm And Other Tech Stocks Fall

    February 7, 2026212
    Don't Miss
    Lifestyle

    How Water Collects on Flat Streets Near the Harbor

    By Staff WriterNovember 20, 20268 Mins Read

    What this covers The Setting Why Flat Streets Hold Water Tides and Drains What It…

    Read More

    What it means for your traffic

    November 20, 2026

    How Often Do Bonds Beat Stocks?

    November 20, 2026

    Why Black Friday and Cyber Monday Are Bad Days to Impulse Buy Electronics

    November 20, 2026
    Stay In Touch
    • Facebook
    • Twitter
    Demo
    Find More
    Trending News
    • How Water Collects on Flat Streets Near the Harbor
    • What it means for your traffic
    • How Often Do Bonds Beat Stocks?
    • Why Black Friday and Cyber Monday Are Bad Days to Impulse Buy Electronics
    • You Don’t Need Cash to Buy a Competitor
    • Anthropic’s IPO could deliver $156 billion to charity
    • Sweden’s H100 Lines Up Bitcoin Trading Revenue for Share Buybacks – BitRss
    • Nurses Share The Worst Mistakes They’ve Ever Seen
    • How to track your brand’s visibility in AI search [2026]
    • Biz4Group LLC – Company Profile
    Other News
    • Business
    • Feel Good
    • Finance
    • Fitness
    • Health
    • Investment
    • Lifestyle
    • Marketing
    • Politics
    • Small Business
    • Technology
    • Top Stories
    • Travel
    About Us

    Small Business Minder brings together business and related news from around the world in one place. Follow us for all the business news you'll need.

    Facebook X (Twitter)
    Our Picks

    How Water Collects on Flat Streets Near the Harbor

    November 20, 2026

    What it means for your traffic

    November 20, 2026
    Most Popular

    Former FBI, CIA Head Has ‘Serious Concerns’ With Trump Cabinet Picks

    December 28, 2024435

    Emirates to operate next-gen A350 on the third daily service to Cape Town

    January 14, 2026256
    Categories
    © 2026 Small Business Minder
    • Home
    • Get In Touch

    Type above and press Enter to search. Press Esc to cancel.

    Ad Blocker Enabled!
    Ad Blocker Enabled!
    Our website is made possible by displaying online advertisements to our visitors. To get the most from our site, please disable your Ad Blocker.