Updated: Jan 5
The dark web is a subset of the internet that is accessed via special means, such as a TOR browser, and not immediately available from the clear net. The term dark web & darknet are often used interchangeably. For reference during this article, we will refer to the darknet as the network infrastructure, such as the TOR network or I2P network, and dark web as the content aspect that is accessed & viewed by users.
There are a lot of great resources that explain what the dark web is, where it originated from & the nefarious activity that occurs there on a daily basis. This article is focused on identifying safe access options & then the multiple search options available using freely available dark web search engines that crawl the dark web.
Investigators looking to conduct traditional search techniques on the dark web need to operate in a safe manner & be aware of the variation in results that are presented by different search engines & also actors who are active in different types of darknets.
Different "Dark Nets"
Most dark web articles refer to The Onion Router (TOR) as it is the most popular & researched. However, it is important to note that there are many darknets and below is an example of three common ones:
Another common darknet is Zeronet. Each has different access requirements or methods.
Different darknet details:
Focusing on TOR, the browser bundle to connect can be downloaded here: https://www.torproject.org/download/
Simply accessing TOR from your standard machine is not advised due to possible security implications. For a lot of users, they will favor ease-of-use over security & connect directly from their standard workstation, but this has serious security considerations. The TOR browser is built on Firefox as a base, and therefore it is subject to the same vulnerabilities that Firefox has. Whilst the Firefox team might patch vulnerabilities regularly, there can be a delay for the update to reach the TOR bundle & therefore exposure users to risks. Given the nature of the content & site hosts on the dark web, this should be a critical consideration so as to not compromise your machine from both an attribution or malware perspective.
It is recommended to apply safe connection methods so as to protect your attribution & host machine from compromise.
Safe Browsing Options
There are many opinions and options for how to access darknets. Below is a simple chart for three options that you can use when connecting to a darknet to provide a safer level of protection. Each has varying barriers to entry & users will have different requirements, budgets, or considerations as part of their connection approach.
Configure a cloud virtual machine or desktop using providers such as Amazon Workspaces (https://aws.amazon.com/workspaces/), Google Cloud (https://cloud.google.com/compute), Microsoft Azure VDI (https://azure.microsoft.com/en-au/free/virtual-machines) or Paperspace (https://www.paperspace.com/). There are other providers but these are relatively cost-effective when used for small periods of time
Install TOR/darknet access on the cloud machine and use that for your research
Connect to the darknet from within the cloud virtual machine
Note: you could also configure a VPN on your cloud machine for an additional layer, however, some cloud providers make this challenging & the technical requirements can increase the barrier to entry
Install & configure a local virtual machine using a platform such as VirtualBox (https://www.virtualbox.org/) & downloading pre-configured VM's (such as the TL VM: https://www.tracelabs.org/initiatives/osint-vm) or installing an operating system from scratch
Install TOR/darknet access on the virtual machine
Configure a VPN on your standard workstation
Connect to the darknet from within the virtual machine
Provision a standalone research laptop/computer (consider using bootable operating systems such as Tails for lower-attribution)
Configure a VPN on your research laptop
Install TOR/darknet access natively on the research laptop
Connect to the darknet natively from your research laptop
Disclaimer: anything you view on the darknet that is rendered locally can still be stored in local caches on any of the options above. You must consider the legal aspects of what you are viewing in the context of your respective governing laws. OSINT Combine takes no responsibility for the content viewed or access methods detailed above.
Dark Web Searching
The dark web is crawled and indexed from numerous non-standard providers, i.e. your traditional search engines such as Google & Bing will not crawl .onion sites on the TOR network. However, proxied TOR sites, being those which use TOR2WEB type services to allow users to view dark web sites from their standard clear web browser, are regularly indexed on Google, although it is not advisable to access these through a proxy for attribution reasons.
Search engines routinely change .onion addresses or go up/down. If any of the following links become unavailable, or you are looking for search engine URLs, a great site is https://onion.live/ - simply search for the search engine name here to find its URL.
Alternatively, there are many "hidden" wiki sites that provide a catalog of resources that also include active search engine URLs. Below is a sample of some of the search engines available.
Dark Web Search Engines: (descriptions obtained from the host websites)
Not all search engines are created equal. They vary significantly with results (due to the sites that are crawled with their platforms), advanced search options (such as boolean or multilingual searching), and intent. Investigators will ideally look across multiple search engines as part of their investigation and compare results, or target their searching to search engines that provide a particular service.
Some investigators will have a requirement to identify & monitor new .onion sites as they arise. This could be to observe patterns, identify new vectors, or simply to create additional pipelines of new .onion URLs to feed into custom crawling engines for advanced users.
There are three good resources that can support this requirement:
Hunchly Dark Web Monitoring (https://www.hunch.ly/darkweb-osint/). The great folks at Hunchly provide a daily email of newly identified .onion domains from their service. This is handy for routine reporting without having to visit the dark web
H-Indexer (http://jncyepk6zbnosf4p.onion/onions.html). This TOR site provides a list of the onions it has indexed including language, title, URL & last contacted in text format
Tor66 Fresh Onions (http://tor66sewebgixwhcqfnp5inzp5x5uohhdy3kvtnyfxc2e5mxiuh34iid.onion/fresh). One of the search engines mentioned above provides their rolling set of identified .onions. This includes the time it was seen, the description & the language identified.
Investigating people on the dark web usually comes down to attribution between the surface & dark web through information slippage. This is where the same attributable markers, e.g. usernames, PGP keys, cryptocurrency addresses, are used by actors on both the surface & dark web.
When trying to attribute users participating in transactions or activity on the dark web, the information slippage is more often tied to poor habits. Below is a diagram that outlines how information propagated & shared between the surface & dark web can be used conceptually to conduct attribution:
A key consideration is the social network analysis aspect once you have identified markers on the surface web. Social groups cluster based around interests or direct associations, mapping out networks, and then conducting language & image analysis of content posted can provide valuable clues to help with attribution. There is no silver bullet and plenty of false positives, so being thorough & diligent is important in your investigation.
When trying to deanonymize & identify hosts of sites, this may require more technical investigations into SSL certificates. An introductory guide to this can be found at Hunchly (https://www.hunch.ly/resources/Hunchly-Dark-Web-Setup.pdf)
This article was designed to provide a basic introduction to dark web searching & some of the pivot points for dark web investigations that you can utilize when only free resources are available. A key takeaway is the requirement to pursue multiple search options to maximize your lead generation and set up an effective workflow so you don't get overwhelmed with the options available to you for searching.
If you are interested in more detailed dark web training, check out our online, self-paced OSINT Combine Academy at https://academy.osintcombine.com or contact us to learn about our bespoke training offerings.