OSINT Research And Subdomain Discovery5th April 2021
Being able to discover subdomains is an essential part of researching an organisation. Behind a company’s main domain there are many other subdomains that can provide useful information about an organisation’s structure, technology, and business practices.
Subdomain discovery is a foundational skill for security researchers, but it is a very useful tool to add to your OSINT toolbox regardless of your purposes. If you can learn how to find subdomains then being able to discover information about the IP addresses, technology and services used by an organisation becomes much easier.
Finding subdomains is not always straightforward though. Common subdomains such as mail.example.com or cpanel.example.com are easy enough to find, but larger organisations can have hundreds or even thousands of subdomains as part of their digital estate. How can you even begin to find them all?
You might have made use of sites like crt.sh for subdomain discovery. These sites are excellent for discovering subdomains because they parse the details from SSL certificates as they are published. Unfortunately when working with very large domains trawling through pages of this kind of information is neither efficient nor practical. Automated tools are the only way to do this at any kind of scale though as we’ll see many of them incorporate information from crt.sh and similar sites in any case.
There are many tools available but they all work slightly differently and the level of technical expertise required varies. Some have a nice web interface while others only work from the command line. Some contain historic subdomain records in addition to current ones, and some will even allow you to brute force possible subdomain names from a list in order to try and discover parts of an organisation that you might not find otherwise.
To show the differences between some popular subdomain discovery tools I conducted a simple comparison test to assess the domain bbc.co.uk. The BBC internet footprint is huge and there are many thousands of subdomains waiting to be discovered. Each of these discovery tools works differently and returns slightly different results.
Security Trails presents results in a clear graphical format that is easy to navigate. It also shows the IP address for each subdomain so you can tell where it is hosted. It also offers historic subdomain records so you can see how subdomains have changed over time. This can often be a way to discover the true IP address of services after they have moved behind CDN services like Cloudflare.
For the bbc.co.uk domain Security Trails found 5,122 subdomains and their associated IP addresses.
Like Security Trails, DNS Dumpster runs via a web interface and presents the results in a reader-friendly format. The results also contain useful additional information like the IP address for the subdomain and allow you to pivot and explore other domains that share the same IP address.
DNS Dumpster presents the information in a useful flow chart format which makes visualisation easy and also allows export to CSV format.
It does return far fewer results though. Without an account you can only view the first 100 results but even with an account only 501 subdomains were found for bbc.co.uk. This is a much smaller number in comparison to the results returned by other tools.
DNS Recon is a powerful DNS discovery tool both for OSINT and network troubleshooting. You can use it to query the DNS records for a domain and it will usually return helpful additional information like SRV records as well as A records, MX records, and so on.
DNS Recon can find subdomains but with two methods that are slightly different to some of the other tools I’ve written about here. Once DNS Recon has identified the IP ranges used by the main domain, it can carry out a reverse IP lookup, where it checks to see which other domains (or subdomains) share the same IP address. This can take a long time but it can be extremely effective at finding subdomains that other tools miss.
The second method involves subdomain bruteforcing. This involves taking a pre-defined list of common subdomain names and checking to see if they exist on the target domain. This process is slow but it can sometimes find results that other programs don’t. If you only wanted to find subdomains in a hurry, this would not be the best tool for the job but it makes so many other DNS OSINT tasks easy that you should definitely have it in your toolbox.
For testing purposes, I pointed DNS Recon at the IP range 132[.]185.0.0 to 132[.]185.255.255, which is used by some BBC services. It discovered 2,157 subdomains on this IP range, but took almost 40 minutes to find them. With more time to assess the other BBC IP ranges it would doubtless find more. If speed is the most important thing to you for subdomain enumeration then DNS Recon isn’t ideal, but for depth and detail it is hard to beat.
Sublist3r is a fast and powerful command line tool for subdomain enumeration. It queries common sources of subdomain information like SSL certificates, search engines, and certificate transparency records like crt.sh.
It’s very fast and easy to use. It also allows subdomain bruteforcing if you’re not in a rush.
When checking bbc.co.uk, Sublist3r found 5,532 unique subdomains in just under 20 seconds. Not bad!
Findomain is my current subdomain enumeration tool of choice. It is very fast and allows you to search multiple domains at once. It queries similar sources to those used by Sublist3r but also offers premium features such as subdomain monitoring and notification. It’s also possible to increase the number of data sources through the addition of API keys.
In this exercise Findomain took 20 seconds to find 4,585 unique subdomains for bbc.co.uk (no API keys used).
Amass is a fantastically powerful command line OSINT tool developed by OWASP. It has a wide range of functions, a well-written wiki, and it also supports a large number of APIs to enrich search results. Results can also be added to a database to help with visualisation.
Amass allows subdomain discovery by both passive and active means. Most subdomain discovery methods are passive and rely on querying third party resources. This has advantages if you require a high degree of opsec since your target will be unaware of your research.
Amass also offers an active enumeration mode where it will find SSL certificates belonging to your target domain(s) and grab the subdomain names from them. This mode involves active contact with the target domain which could be a concern for some researchers but it has the advantage of ensuring that information is accurate and current.
Amass found 3,714 subdomains for bbc.co.uk. It also presents the associated IP addresses and ISP details in the results.
And The Winner Is?
Well….there is no real winner! As you can see each tool is different and has its own particular advantages and disadvantages. Each tool emphasises something different and you can choose one according to your needs.
For more articles on OSINT tools click here.