How to find every company that uses your competitor, part 2: scale to a million websites and more
Check a million websites or more every month on one computer, keep the history, and get an alert every Monday when a company starts using your competitor. The engine we built for Drag, before it became AgentData.
Nick Timms · 12 Nov 2026 · 12 min readWrite down the marks your competitor’s software leaves on a website or in DNS, then check a million websites or more for them every month and keep every result. Start from Common Crawl’s free list of 118 million domains, skip the dead ones with a quick DNS lookup, and call a company a new customer only when an earlier, clean check found nothing. One computer can do it. The prompt below builds all of it, with a Monday alert.
Build your competitor engine in one click
Opens Claude or ChatGPT with the GTM Engine v2 prompt loaded. It writes your competitor’s fingerprint, builds a list of a million websites from Common Crawl and Tranco, skips the dead ones, checks the rest, keeps every result in a database on your computer, and schedules itself with a Monday alert of new customers. You type your competitor and your market, then approve each of the six stages.
The cheapest lead in B2B is your competitor’s customer. They’ve already decided they need what you sell, found the budget and learned the category. You only have to be better.
This spring I wrote a small script to find them for Drag, our shared inbox inside Gmail. Some of our competitors leave a script on every customer’s website, and my script went looking for it.
It found real customers in the first few thousand websites, so we kept going. We called it Sonar. Within weeks it was running through the night on a Mac Mini, and emailing us whenever one of our competitors turned up on a website it hadn’t seen before.
Then it outgrew the Mac Mini. Seven months on, it checks around 87 million websites a month, and it’s a company of its own: AgentData, one of the five startups Duda and I have launched this year, with no new hires.
That still surprises me. A year ago this was a job for a data team. It started as two of us, an AI that wrote most of the code, and one computer. And it changed how we find customers for Drag: every weekday an agent writes to 10 companies using two of our competitors, and nothing goes out until Duda or I say yes.
This note is the version in between. It’s what we ran before it became a company, and what I’d build again if I were starting today: a million websites or more a month on one computer, a history of every check, and an alert every Monday with the companies that just started using your competitor.
New to this? Part 1 is the simple version: one competitor, a few thousand websites, one afternoon, no database. Start there if you’ve never done it before. Everything below also works on its own.
The idea in one minute
Software leaves marks. A chat widget loads a script, a payments tool sets a cookie, an email or e-signature tool asks the company to add a record to its DNS. Those marks are public.
Write down the marks your competitor leaves (that’s the fingerprint), check websites for them, and keep the evidence for every match. Do it every month and keep the history, and the changes tell you who just started.
The recipe card
- Serves
- B2B founders who want every company in their market that uses a competitor, kept fresh. One competitor or ten
- Prep
- An afternoon, most of it waiting for the first run
- Cook
- Runs on its own: about 100,000 websites a night, a million or more a month
- Cost
- $0 to $20 a month, if you already pay for Claude or ChatGPT and have a computer that stays on
- Makes
- A database of every check, and a Monday alert of new customers, with the evidence
- Skill level
- Copy, paste, approve. No coding
Ingredients
Everything here is free apart from the AI plan. The prompt downloads each file itself; the links are so you can see exactly what it uses.
Claude or ChatGPTA paid plan, so you get Claude Code or Codex. They build the engine and set it running.From $20 a month
Node.jsRuns the checks. Download it, or let Claude Code install it.Free
Common Crawl’s domain list118 million domains, ranked by how many websites link to them: cc-main-2026-may-jun-jul-domain-ranks.txt.gz (2.2 GB).Free
Tranco top 1 millionThe most visited websites, as a second opinion on rank: top-1m.csv.zip.Free
webappanalyzerReady-made fingerprints for over 7,000 tools. Your competitor may already be in there.Free
SQLiteThe database is one file on your computer. DB Browser for SQLite lets you open it and look around.Free
A computer that stays onA Mac Mini, an old laptop or a small server. Ours started on a Mac Mini.Free if you have one
Slack or emailWhere the Monday alert lands. For Slack, make an incoming webhook in two minutes.Free
What changes past a few thousand websites
The first surprise at scale is how little of the web is there to read. This is what happened to every website AgentData checked in the last month:
| What happened | Share | What it means |
|---|---|---|
| Read the page | 31% | It loaded, and we could check it |
| No DNS | 27% | The domain points nowhere. Nothing to visit |
| Didn’t answer properly | 27% | Refused, reset or broke halfway |
| Error page | 6% | Server errors, parked pages, broken redirects |
| Almost empty | 5% | Placeholders with nothing to read |
| Robots.txt unreadable | 2% | We skip these to be safe |
| Blocked us | 1% | Rarer than you’d think |
That table shapes the whole method. Most of the work at this size isn’t checking websites. It’s not wasting time on the ones that aren’t there, and not mistaking a bad day for a change.
Method
1. Write your competitor’s fingerprint (15 minutes)
Pick three companies from your competitor’s customers page and look at their websites: View Page Source and DevTools → Network for scripts and cookies, and a free DNS lookup for TXT and MX records. Write one rule per mark, for example:
- Script URL:
widget.intercom.io - DNS TXT:
docusign= - Page variable:
window.Intercom
Match the script or the record, never just the brand name. Part 1 walks through this step by step, and the prompt below does it for you.
2. Get a million websites to check (20 minutes)
Your competitor’s customer pages run out fast. For a big list, start where we did: Common Crawl, a free public archive of the web. Most of the 137 million domains AgentData has found came from it.
Its domain list ranks 118 million domains by how many other websites link to them. Cut it down to your market:
- By country. Keep
.co.ukand.ukfor the UK,.defor Germany. For.com, filter later by language or address. - By rank. The top few million are real, linked-to businesses. Further down it’s mostly parked domains and spam.
- By what you already know. Add your sign-ups, your CRM and your competitor’s customer pages. They always beat a random slice.
A million is plenty to start, and nothing stops you going further. One quirk: the file writes domains backwards (com.example). The prompt flips them.
3. Drop the dead ones first (an hour, once)
This is the cheapest win in the whole build. Over a quarter of a big list points nowhere: the domain has lapsed, or never had a website. A DNS lookup answers in a fraction of a second; loading a page takes a few seconds. So look up every domain first, and only visit the ones that answer.
Keep the records while you’re there. MX and TXT records are where you find the tools that never show on a website. In AgentData, about 25 million detections come from DNS alone.
4. Keep history, not a spreadsheet (10 minutes)
A spreadsheet tells you what’s true today. To see what changed, you need memory, so the prompt sets up SQLite: one file on your computer, a few tables.
- domains: every website, where it came from, when it was last checked
- checks: every visit, whether the page loaded, and which version of your fingerprints you used
- detections: every match, with the exact evidence, first seen and last seen
First seen and last seen are the product. Everything else is plumbing.
5. Run it on a schedule, on its own (5 minutes)
The prompt schedules it for you: launchd on a Mac, cron on Linux, Task Scheduler on Windows. Then you leave it.
In our experience, one computer gets through roughly 10,000 websites an hour once the dead ones are gone. That’s 100,000 in a night and a million in under a week, so a monthly run fits with room to spare.
Not everything needs checking monthly. Check your best prospects every week, the rest every month, and anything that keeps failing every three months. After five failures in a row, it’s gone: stop checking.
6. Decide what counts as new
This is where homemade versions go wrong, and where we spent most of our time.
A tool you see for the first time isn’t necessarily new. Maybe the site failed to load last month. Maybe you only wrote that fingerprint this week. Call it new, and you’ll congratulate a company on switching when it has used your competitor for five years. So:
- New customer: the tool is there now, and an earlier check that loaded the page properly, with the same fingerprints, didn’t find it.
- First look: you’d never checked that site with these fingerprints before. It’s found, not new.
- Stopped: only after two clean checks in a row without it. A page that failed to load proves nothing.
Your first run is all first looks. The first real alert comes after the second. And treat stops as a hint, not a fact: a redesign looks exactly like a cancellation, which is why AgentData doesn’t report them yet.
7. Get the Monday alert
Every Monday the engine sends you the new customers since last week: website, the evidence and when it was first seen. A few lines in Slack or your inbox, not another dashboard. It looks like this, with made-up domains:
New Intercom customers this week: 3 example-saas.com widget.intercom.io first seen Mon 9 Nov example-shop.co.uk widget.intercom.io first seen Sat 7 Nov example-agency.io window.Intercom first seen Thu 5 Nov Possible stops: 1 (check before you trust it)
8. Check yourself every month (10 minutes)
Open 20 new customers at random. If more than one is wrong, fix the fingerprint before you trust next month’s alert. AgentData does a version of this every night, for every kind of data it shows, and hides anything that falls below its bar.
The GTM Engine
One prompt that builds an engine that checks a million websites or more a month and tells you who just started using your competitor. Paste it into Claude Code or Codex, approve six stages, and leave it running.
- 6
- stages, each approved by you
- 1M+
- websites a month, one computer
- Mon
- alert with new customers
- $0
- spent on data
- 1Writes the fingerprintfingerprints.jsonReuses one you already have, or works it out from your competitor’s known customers, and gives it a version number.
- 2Builds a million-website listgtm.dbDownloads Common Crawl’s ranked domain list and Tranco, cuts them to your market and adds your own lists.
- 3Drops the dead onesdns tableLooks up DNS for every domain, keeps the records, and skips the ones that point nowhere.
- 4Checks every live websitechecks, detectionsVisits each homepage politely, tests every fingerprint, and records every check, even the failed ones.
- 5Works out what’s newthis week’s changesCompares with earlier clean checks: new customers, first looks and possible stops.
- 6Runs itselfthe Monday alertSchedules the checks and the Monday alert to Slack or email.
You are my GTM engineer. Build me an engine that checks up to a million websites or more a month on this computer, keeps a history of every check, and alerts me when a company starts using my competitor. Public website and DNS data only. COMPETITOR: [e.g. Intercom] MARKET: [e.g. UK and Ireland, B2B software] ALERT TO: [a Slack webhook URL, or an email address] Work in six stages. At the end of each stage, show me what you found and wait for my "go" before starting the next one. STAGE 1 - FINGERPRINT
Using Claude Code or Codex?
claude "$(pbpaste)"When it’s set up you’ll have gtm.db, which grows every month, fingerprints.json with a version number, and a Monday message that tells you who’s new.
Where it goes wrong
Dead domains eat your month. Skip the DNS step and over a quarter of your time goes on websites that don’t exist.
The first run lies. Everything on run one is a first look. If your first alert says 4,000 new customers, it’s counting first looks.
You change a fingerprint. Every site with the tool now looks like it just started. That’s why each check stores the fingerprint version, and why first looks never count as new.
One container, thousands of sites. Some website builders load the same tag manager setup on every customer’s site, so a tool can suddenly appear on thousands of unrelated companies at once. If a tool jumps overnight, look at the evidence before you believe it.
Hosts that bundle tools. A hosting company can switch on email or chat for all its customers. The company didn’t choose that tool, its host did, and it won’t want to hear about switching.
Redirects. A domain that redirects to a parent brand reports the parent’s tools. Store where you ended up, not just where you started.
Your competitor leaves no trace. Browser extensions, desktop apps and anything behind a login are invisible from the outside, at any scale. Some of Drag’s competitors live entirely inside Gmail, so we never see them on a website.
Being a good citizen, at scale. At a million websites, manners matter more, not less. Name your bot, respect robots.txt, cap your DNS lookups, never get round a block, and follow the cold email rules where you sell before you contact anyone.
The money
This costs what your AI plan costs, plus electricity: about $10 a month for a Mac Mini that’s always on. Here’s how that compares with buying the same thing.
The starting numbers are an example, not our results. Move the sliders to yours.
When DIY stops being worth it
Built like this, one computer comfortably covers one market: a million websites or more, checked every month. It’s close to what we ran ourselves for months, and it found the companies Drag’s outreach agent still writes to every weekday. For most founders, it’s all you’ll need.
It stops being worth it when you want everything: every company, every tool, everywhere. Four things change at once:
- Infrastructure. Past a few million websites one computer can’t keep up. You need several machines, a real database, and someone to keep them all running.
- Revisiting. Finding a company once is cheap. Rechecking tens of millions of them every month, so the data stays true, is where the cost keeps growing.
- Volume to find the rare ones. Most tools are on a tiny share of websites. Intercom shows up on about 1 in 2,300. To find every customer of a smaller tool, you have to check an enormous number of websites that don’t have it.
- Accuracy. At that size, a small error rate means thousands of wrong rows. Knowing a detection is right before anyone acts on it becomes the main job.
That’s where we stopped doing it ourselves and built AgentData. Seven months of crawling, 137 million domains found, 87 million website checks in the last month, and a nightly check of its own accuracy. And it started as an afternoon script for Drag.
Or see who just started using your competitor, today
AgentData checks company websites and DNS records around the clock, keeps the evidence for every detection, and flags the companies that just started using a tool.
- 133M
- companies tracked
- 35M
- websites profiled
- 196M
- detections, with evidence
- 4,281
- tools tracked
Free to look up, no account needed. Full lists, downloads and new-customer alerts from $99 a month, with a 7-day trial and no card.
Ask it from Claude or ChatGPT →
Full disclosure: AgentData is ours. Everything above works without it.
Questions
How do I find every company that uses a competitor, at scale?
Write a fingerprint for the marks your competitor leaves, build a list of a million or more websites from Common Crawl and your own data, skip dead domains with a DNS lookup, and check the rest every month. The prompt builds it.
How do I get an alert when a company starts using a competitor?
Keep every check, and call it a new customer only when an earlier clean check didn’t find the tool. The engine sends those to Slack or email every Monday.
How many websites can one computer check?
In our experience, roughly 10,000 an hour once dead domains are skipped: 100,000 a night, a million or more a month.
Where can I get a big list of websites for free?
Common Crawl publishes a ranked list of 118 million domains a few times a year, and Tranco publishes the top million. Cut them by country and rank, and add your own data.
Why not just compare this month’s list with last month’s?
Because a page that failed to load, or a fingerprint you changed, looks exactly like a change. Store every check with its fingerprint version, and compare clean checks only.
Can I track more than one competitor?
Yes. Add their fingerprints to the same file. Each extra competitor costs almost nothing, because the slow part is visiting the website, not testing it.
Is it allowed?
Reading public web pages and public DNS records is how tools like Wappalyzer and BuiltWith have worked for years. Name your bot, respect robots.txt, cap your lookups and never get round a block. Emailing the people you find is covered by the email and data protection rules where you sell. We’re not lawyers, and this isn’t legal advice.
If you build this, reply and tell me how many new customers your first real alert found. I read every reply.
Sources: Common Crawl web graphs, May to July 2026, Tranco, webappanalyzer, how AgentData detects tools. AgentData numbers as of 3 November 2026.
“The cheapest lead in B2B is your competitor’s customer. Here’s how to check a million websites a month for them on one computer, and get an alert when a new one starts.”
Post on X
Share on LinkedIn