MobibeanBot
On-demand website analyzer for LLM-readiness reports
What is MobibeanBot?
MobibeanBot is an automated website analyzer that checks how LLM-ready a website is. It only runs when a user requests a report via mobibean.com/tools/llm-readiness. It is not a continuous crawler and does not index or store page content beyond what is needed for the report.
User-Agent String
What does it access?
When analyzing a website, MobibeanBot requests only these resources:
- •robots.txt
- •sitemap.xml
- •llms.txt
- •llms-full.txt
- •.well-known/security.txt
- •humans.txt
- •The specific page URL requested by the user
When does it crawl?
MobibeanBot operates strictly on-demand. It is not a continuous crawler and does not periodically revisit websites. Each report request generates one set of HTTP requests to the target site, and that is it.
How to block it
Add the following to your robots.txt file:
Disallow: /
If blocked, the tool will skip those fetches and note it in the report. The report will still be generated with the available data.
Rate limits
- •Max 2 MB response size per request
- •10-second timeout per request
- •No cookies or authentication credentials forwarded
Questions?
If you have questions about MobibeanBot or need to report an issue, get in touch.
Contact Us