Why is this crawler visiting my site?

securitytxthistory builds a public archive of security.txt files (RFC 9116), the standard for publishing how to report vulnerabilities. It does not look for vulnerabilities and fetches nothing beyond two public paths.

• It only requests https://your-domain/.well-known/security.txt and, if that does not exist, /security.txt.
• Once per domain, following at most 5 redirects, with a 5-second timeout.
• It identifies itself with the User-Agent securitytxthistory/0.1 (+mailto:securitytxthistory@agentmail.to).
• It honors 429 and 503: it does not retry when the site asks it to wait.

Opt-out and contact

To stop your domain from being fetched, or to remove its file from this site, email securitytxthistory@agentmail.to with the domain name. Requests are applied to the crawler within minutes.

References

• RFC 9116: A File Format to Aid in Security Vulnerability DisclosureE. Foudil, Y. Shafranovich (IETF, 2022)
• 1 Million Websites - How Many Use Security.txt?Naz Markuta, Hexiosec (2022). Tranco top 1M survey: 0.37% of domains published a security.txt.
• Follow-Up on Security.txt Adoption: Progress and Pitfalls in 2025Freddie Leeman, URIports (2025). Top 1M domains: 1.25% published a security.txt; 44% of those complied with RFC 9116.
• Characterizing the Adoption of Security.txt FilesWilliam Findlay, AbdelRahman Abdou, Carleton University (MADWeb workshop at NDSS, 2022).