Factset_spyderbot
Factset_spyderbot collects content for model training, and the community reference lists its operator as Factset. Our October 2026 survey found 92 sites blocking it, or 1.5% of the files we could read.
Who runs Factset_spyderbot?
| Fact | Factset_spyderbot reference entry |
|---|---|
| Operator | “Factset” |
| Purpose | “AI model training.” |
| Does Factset_spyderbot obey robots.txt? | “Unclear at this time.” |
| Frequency | “No information provided.” |
| Cloudflare managed list for Factset_spyderbot | No |
These facts quote the stored community ai.robots.txt reference.
“Scrapes data for AI training.”
Model training is this crawler’s listed use. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 96, or 1.6%. Instructions blocking every path, Disallow: /, appear in 92, a 1.5% share. No whole-site ban is recorded for 6,079 files.
Among 552 readable files from the top 1,000 sites, we found 23 files naming it, a 4.2% share, with 22 files blocking the whole site. Blocking is more common among the leading sites, at 4.0%. Outside this group, 5,619 files contribute 70 blocks, or 1.2%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Factset_spyderbot, all ranks | 96 | 1.6% | 92 | 1.5% |
| Factset_spyderbot, top 1,000 | 23 | 4.2% | 22 | 4.0% |
| Factset_spyderbot, remaining ranks | 73 | 1.3% | 70 | 1.2% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 92 blocks, 2 do not also appear in our count of files naming it. Looking just at named files gives 90 blocks and 6 without a whole-site block. For 6,075 readable files, we found no mention by name. We ignore capitalization and repeated names.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming Factset_spyderbot under the robots.txt standard.
Block the whole site
User-agent: Factset_spyderbot
Disallow: /Allow access
User-agent: Factset_spyderbot
Allow: /It's the 62nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For Factset_spyderbot, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
This survey includes 6 files with the name and no whole-site ban, out of 96 naming files.
Are leading sites different?
Within readable robots.txt files near the top of the ranking, 4.2% name it and 4.0% block it. Overall, naming reaches 1.6% and blocking 1.5%.
How does the nearby crawler compare?
The nearby PhindBot entry reports 96 files that name it and 76 whole-site blocks, 1.2% of readable files. Here, 96 files mention the name and 92 block access, or 1.5%.