YepBot

What is YepBot

YepBot is a web crawler that, along with AhrefsBot, powers the index of web pages for Yep, a revenue-sharing web search engine, and Ahrefs, an online data toolset.

Our bots index fresh, accurate information about websites, their content, and how they link to each other. This data makes it possible to find top-quality content and all the information you need on Yep, a search engine that aims to share ad revenue with content creators and site owners.

AhrefsBot remains the primary source of data for both Yep and Ahrefs. Currently, YepBot is used only to make requests according to IndexNow protocol, such as validating user keys and pinging other search engines. However, we plan to expand the use of the YepBot user-agent more broadly, which will be announced in advance.

YepBot obeys robots.txt

YepBot strictly respects robots.txt on both disallow and allow rules.

Customizing YepBot access

Since YepBot strictly follows the robots.txt file on your website, you're in full control of what it does.

To change the frequency at which YepBot visits your site, just specify the minimum acceptable delay between two consecutive requests in your robots.txt file:

Crawl-Delay value is time in seconds.

If you want to stop YepBot from visiting your site, add these two lines to the robots.txt file on your server:

Please note that YepBot may need some time to pick up the changes in your robots.txt file.

Also, if your robots.txt contains errors, YepBot may parse it partially and follow only those commands that it was able to recognize.

Read more about robots.txt at www.robotstxt.org.

If you think that YepBot is misbehaving on your website or if you have any questions, please reach out to support@ahrefs.com and we'll do our best to help.