Web Reconnaissance

Cheatsheet

# Passive 
whois <DOMAIN>
 
#Certificate Transparency logs
curl -s "https://crt.sh/?q=<DOMAIN>&output=json" | jq -r '.[].name_value' | sort -u
curl -s "https://crt.sh/?q=%25.<DOMAIN>&output=json" | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u
 
# historical URLs and snapshots
waybackurls <DOMAIN> | sort -u
# or open it in browser and choose date accordingly 

# DNS record enumeration
dig <DOMAIN> any +noall +answer
dig <DOMAIN> a +short
dig <DOMAIN> mx +short
dig <DOMAIN> ns +short
dig <DOMAIN> txt +short
 
# zone transfer
dig axfr <DOMAIN> @<DNS_IP>
 
# subdomain brute force
dnsenum --enum <DOMAIN> -f /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -r
ffuf -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt -u "http://<DOMAIN>" -H "Host: FUZZ.<DOMAIN>" -ac
feroxbuster -u http://FUZZ.example.com -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt

# virtual host discovery 
gobuster vhost -u http://<IP>:<PORT> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt --append-domain
ffuf -w <WORDLIST> -u http://<IP> -H "Host: FUZZ.<DOMAIN>" -ac
 

# fingerprinting and WAF detection
curl -I http://<DOMAIN>            # server banner + redirect chain
whatweb <DOMAIN>
wafw00f <DOMAIN>
nikto -h <DOMAIN> -Tuning b        # software identification only
 
# content / metadata discovery
curl -s http://<DOMAIN>/robots.txt
curl -s http://<DOMAIN>/sitemap.xml
curl -s http://<DOMAIN>/.well-known/security.txt
curl -s http://<DOMAIN>/.well-known/openid-configuration | jq # information about the authentication server

Methodology

Phase 1: Scope and Passive Footprinting

# Capture ownership, registrar, and authoritative DNS clues
whois <DOMAIN>
whois <DOMAIN> | grep -i "iana\|registrar\|name server\|creation\|expiry"
 
# Pull historical URLs from public archives
waybackurls <DOMAIN> | sort -u

# Search indexed pages for high-value paths and documents
site:<DOMAIN> (inurl:login OR inurl:admin)
site:<DOMAIN> (filetype:pdf OR filetype:xls OR filetype:docx)
site:<DOMAIN> (ext:conf OR ext:cnf OR ext:sql)

• Confirm scope and Rules of Engagement before any query.
• Run whois <DOMAIN> to capture registrar, dates, name servers, and registrant org.
• Search engines + Google dorks for exposed files, login pages, and config (site:, filetype:, inurl:).
• Review the Wayback Machine for old endpoints, removed pages, and prior tech stacks.
• Search GitHub/GitLab for the org name and domain to find leaked credentials, keys, or internal hostnames.

Phase 2: DNS Enumeration and Subdomain Discovery

# Enumerate core DNS records
dig <DOMAIN> a +short
dig <DOMAIN> aaaa +short
dig <DOMAIN> mx +short
dig <DOMAIN> ns +short
dig <DOMAIN> txt +short
 
# Attempt AXFR against each authoritative name server
dig axfr <DOMAIN> @<DNS_IP>
 
# Pull subdomains from Certificate Transparency logs
curl -s "https://crt.sh/?q=%25.<DOMAIN>&output=json" | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u
 
# Brute force and resolve candidate subdomains
dnsenum --enum <DOMAIN> -f /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -r
dnsx -l subdomains.txt -resp -a -aaaa -cname -silent

• Enumerate records: dig <DOMAIN> any, plus explicit a, aaaa, mx, ns, txt queries.
• Attempt a zone transfer against every NS: dig axfr <DOMAIN> @<DNS_IP>.
• Pull subdomains passively from CT logs (crt.sh, Censys) before brute forcing.
• Brute force subdomains with dnsenum/ffuf and a SecLists DNS wordlist; enable recursion on hits.
• Resolve every discovered subdomain to its IP and group hosts by IP for the vhost phase.

A successful zone transfer dumps the entire zone, every subdomain, IP, and record in one request. Most servers refuse it, but a single misconfigured secondary still makes the attempt worthwhile.

Phase 3: Virtual Host Discovery

# Establish a default/wildcard baseline
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -H "Host: invalid-$(date +%s).<DOMAIN>" http://<IP>:<PORT>/
 
# name-based virtual hosts
gobuster vhost -u http://<IP>:<PORT> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt --append-domain
 
ffuf -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt -u http://<IP>:<PORT>/ -H "Host: FUZZ.<DOMAIN>" -ac
 
# Manually confirm a hit before adding it to scope notes
curl -i -H "Host: <VHOST>" http://<IP>:<PORT>/

• Establish the baseline response for an unknown Host value to set filters.
• Fuzz the Host header with gobuster vhost or ffuf -H "Host: FUZZ.<DOMAIN>".
• Auto-calibrate (-ac / size filters) to drop wildcard/default responses.
• Manually confirm each hit by requesting it with the matching Host header.
• Bind confirmed vhosts to the target IP in /etc/hosts so the browser and tools resolve them.

Phase 4: Fingerprinting and WAF Detection

# Compare HTTP and HTTPS headers and redirect behavior
curl -k -I http://<DOMAIN>
curl -k -I https://<DOMAIN>
 
# Fingerprint web technologies
whatweb -a 3 <URL>
wafw00f <URL>
 
# sfotware check with nikto
nikto -h <URL> -Tuning b
 
# Check identified products against local exploit metadata
searchsploit <PRODUCT> <VERSION>

• Grab banners and follow the full redirect chain: curl -I on http and https.
• Run whatweb / Wappalyzer to profile CMS, frameworks, and analytics.
• Detect WAFs with wafw00f before any aggressive scanning.
• Run nikto -Tuning b for software identification and obvious misconfigurations.
• Map each finding to a version and check searchsploit / advisories for matching CVEs.

Phase 5: Content and Endpoint Discovery

# ATP start burp/Zen scanner or crawler to crawl whatever sites/pages you get to reveal information like mails, usernames, comments, etc
curl -s <URL>/robots.txt
curl -s <URL>/sitemap.xml
curl -s <URL>/.well-known/security.txt
curl -s <URL>/.well-known/openid-configuration | jq
 
 
# Brute force 
feroxbuster -u <URL> -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -x php,txt,bak,zip -k
 
ffuf -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -ac -c -m 200,204,301,307,401,405,400,302 -u <URL>

• Fetch robots.txt and treat every Disallow as a hint, not a barrier.
• Fetch sitemap.xml for a quick map of intended pages.
• Probe .well-known/ URIs, especially openid-configuration and security.txt.
• Catalogue every interesting path and hand it to directory brute forcing / web attack notes.

Phase 6: Crawl and Extract Artifacts

# Crawl a discovered host with an existing crawler
python3 ReconSpider.py <URL>
 
# Inspect extracted emails, comments, and links
jq '.emails' results.json
jq '.comments' results.json
jq '.links' results.json
 
# Crawl with Katana as a modern CLI alternative
katana -u <URL> -d 3 -silent -o katana-urls.txt
 
# use bursuiite crawler to crawl the site

• Crawl only hosts that are confirmed in scope and already prioritized by earlier phases.
• Save crawler output (results.json, discovered URLs, screenshots, key responses) as evidence.
• Review emails and usernames for credential attacks, but confirm lockout policy before spraying.
• Review HTML comments and JavaScript for API keys, deprecated endpoints, and planned key rotations.
• Re-feed discovered paths and hosts into the vhost, content discovery, and fingerprinting phases.

Methodology Recovery

When every phase above is complete and no clear path remains:

1. Re-run from Phase 1 with the evidence collected, new subdomains often unlock new vhosts and endpoints.
2. Re-attempt AXFR and CT-log queries; zones and certificates change over time.
3. Swap wordlists (different SecLists DNS/content lists) and re-fuzz vhosts and directories.
4. Recheck headers and behavior on every host, not just the primary domain.
5. Re-feed crawler-extracted usernames and emails into credential attacks against the softest host.

Reference

WHOIS

Key fields to extract and why they matter:

• Registrar / Creation Date / Expiry Date : domain age and legitimacy; old, long-registered domains are established assets.
• Registrant / Admin / Tech Organization : the owning entity and points of contact (social engineering and scope confirmation).
• Domain Status (e.g. clientTransferProhibited) : locks that indicate security maturity.
• Name Servers : self-hosted vs third-party DNS, hinting at infrastructure and hosting provider.

Modern WHOIS is often privacy-redacted, so combine it with DNS and OSINT rather than relying on it alone.

DNS Record Types

For full DNS service enumeration (port 53, BIND version, AD SRV records), see DNS enumeration.

dig Common Commands

Subdomain Brute Forcing

Subdomain brute force tests pre-defined name lists against a domain to find valid subdomains. Wordlist quality drives results.

# dnsenum: -f wordlist, -r recursive on discovered subdomains
dnsenum --enum <DOMAIN> -f /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -r
 
#amass
amass enum -d target.com -brute -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -o brute.txt
 
# puredns
puredns resolve <DOMAIN> -r /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt
 
# recommended wordlists 
/usr/share/seclists/Discovery/DNS/bitquark-subdomains-top100000.txt
/usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt

Zone Transfers (AXFR)

A zone transfer is a wholesale copy of every record in a DNS zone from one name server to another, used for redundancy. A misconfigured server that allows AXFR from any client leaks the full zone: subdomains, IPs, and NS records.

dig axfr <DOMAIN> @<DNS_IP>

Virtual Hosting Types

1. Name-Based : distinguishes sites by the HTTP Host header on one IP. Most common.
2. IP-Based : one IP per site, no reliance on the Host header.
3. Port-Based : different sites on different ports of one IP. | Tool | Use for vhosts | |---|---| | gobuster  | vhost mode; needs --append-domain on newer versions | | Feroxbuster  | Rust speed, recursion, filters | | ffuf  | Fuzz the Host header with -H "Host: FUZZ.<DOMAIN>" |

# Useful gobuster flags: -t threads, -k ignore TLS errors, -o save output
gobuster vhost -u http://<DOMAIN>:<PORT> -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt --append-domain

Certificate Transparency (CT) Logs

CT logs are public, append-only records of every TLS certificate issued. Unlike brute forcing, they give a definitive historical list of subdomains including expired ones that may run outdated, vulnerable software.

# All 'dev' subdomains for facebook.com via the crt.sh JSON API
curl -s "https://crt.sh/?q=<DOMAIN>&output=json" | jq -r '.[] | select(.name_value | contains("dev")) | .name_value' | sort -u

Fingerprinting Tools

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml

Disallow entries advertise paths the owner wants hidden from crawlers,exactly the directories worth manual inspection. It is a hint source, not access control.

.well-known URIs

The .well-known/ path (RFC 8615) centralizes site metadata. IANA maintains the registry .

curl -s https://example.com/.well-known/openid-configuration | jq

openid-configuration exposes the authorization, token, userinfo, and JWKS endpoints plus supported scopes and signing algorithms a map of the auth surface for later OAuth/JWT testing.

Search Operators (Google Dorking)

Common dork patterns (see the Google Hacking Database ):

# Login / admin pages
site:example.com (inurl:login OR inurl:admin)

# Exposed documents and config
site:example.com (filetype:xls OR filetype:docx)
site:example.com inurl:config.php
site:example.com (ext:conf OR ext:cnf)

# Database backups
site:example.com filetype:sql

Common Mistakes

Attack Chains / Related Notes

• Came from: Recon & Enumeration index, Nmap network enumeration
• Feeds into: DNS enumeration, WordPress penetration testing
• Leads to: directory/content brute forcing and targeted web vulnerability testing once endpoints, subdomains, and the tech stack are mapped.