TechLeadsBot

The crawler of TechLeads, a directory of the technologies websites use.

How it identifies itself

Mozilla/5.0 (compatible; TechLeadsBot/1.0; +https://techleads.fyi/bot)

What it fetches

• A site's homepage, and at most three public pages linked from it (contact, legal notice, about).
• Public script files the homepage loads, to recognise the technologies in use.
• Nothing behind a login, no forms, no personal accounts.

How often

• A few requests per site per visit, one site visit at a time, with ordinary timeouts.
• Most sites are visited about once a month or less.

robots.txt

TechLeadsBot follows robots.txt rules addressed to TechLeadsBot or to all user agents (*). To keep it out of your whole site:

User-agent: TechLeadsBot
Disallow: /

Contact

Questions, rate problems, or removal of your site's data: contact@techleads.fyi.