For Webmasters
Our Crawlers
We believe in an open web, and in respecting the rules you set for it.
LemneBotWeb crawlerOur general-purpose web crawler. It indexes pages for the LemneX search index. It identifies itself via the User-Agent string Mozilla/5.0 (compatible; LemneBot/1.0; +https://lemnex.com/crawlers)
Block with robots.txt
User-agent: LemneBot Disallow: /
LemneAIAI training / synthesisUsed when content is fetched for AI answer synthesis or model training purposes. Separate from the search crawler so you can allow indexing but opt out of AI use or vice versa. Identifies itself as Mozilla/5.0 (compatible; LemneAI/1.0; +https://lemnex.com/crawlers)
Block with robots.txt
User-agent: LemneAI Disallow: /
Our commitments
- We respect robots.txt. If your robots.txt disallows LemneBot or LemneAI, we do not crawl. No exceptions, no "accidental" visits.
- We identify ourselves honestly. Our User-Agent is always truthful. We never masquerade as a human browser to bypass your rules.
- We rate-limit ourselves. Our crawlers back off per-domain to avoid impacting your server performance.
- Separate bots, separate consent. Allow search indexing but block AI training, or the reverse. The choice is yours and it's granular.
- Opt-out is immediate. Once you add the disallow rule, we stop on the next visit. No 30-day processing window, no "legitimate interest" loopholes.
Questions or issues?
If our crawler is causing problems or you have questions about how your content is used, reach out: crawlers@lemne.com