Updated: Sep 24
The dark web is a subset of the internet that is accessed via special means, such as a TOR browser, and not immediately available from the clear net. The term dark web & darknet are often used interchangeably. For reference during this article, we will refer to the darknet as the network infrastructure, such as the TOR network or I2P network, and dark web as the content aspect that is accessed & viewed by users.
There are a lot of great resources that explain what the dark web is, where it originated from & the nefarious activity that occurs there on a daily basis. This article is focused on identifying safe access options & then the multiple search options available using freely available dark web search engines that crawl the dark web.
Investigators looking to conduct traditional search techniques on the dark web need to operate in a safe manner & be aware of the variation in results that are presented by different search engines & also actors who are active in different types of darknets.
Different "Dark Nets"
Most dark web articles refer to The Onion Router (TOR) as it is the most popular & researched. However, it is important to note that there are many darknets and below is an example of three common ones:
Another common darknet is Zeronet. Each has different access requirements or methods.
Different darknet details:
Focusing on TOR, the browser bundle to connect can be downloaded here: https://www.torproject.org/download/
Simply accessing TOR from your standard machine is not advised due to possible security implications. For a lot of users, they will favor ease-of-use over security & connect directly from their standard workstation, but this has serious security considerations. The TOR browser is built on Firefox as a base, and therefore it is subject to the same vulnerabilities that Firefox has. Whilst the Firefox team might patch vulnerabilities regularly, there can be a delay for the update to reach the TOR bundle & therefore exposure users to risks. Given the nature of the content & site hosts on the dark web, this should be a critical consideration so as to not compromise your machine from both an attribution or malware perspective.
It is recommended to apply safe connection methods so as to protect your attribution & host machine from compromise.
Safe Browsing Options
There are many opinions and options for how to access darknets. Below is a simple chart for three options that you can use when connecting to a darknet to provide a safer level of protection. Each has varying barriers to entry & users will have different requirements, budgets, or considerations as part of their connection approach.
Configure a cloud virtual machine or desktop using providers such as Amazon Workspaces (https://aws.amazon.com/workspaces/), Google Cloud (https://cloud.google.com/compute), Microsoft Azure VDI (https://azure.microsoft.com/en-au/free/virtual-machines) or Paperspace (https://www.paperspace.com/). There are other providers but these are relatively cost-effective when used for small periods of time
Install TOR/darknet access on the cloud machine and use that for your research
Connect to the darknet from within the cloud virtual machine
Note: you could also configure a VPN on your cloud machine for an additional layer, however, some cloud providers make this challenging & the technical requirements can increase the barrier to entry
Install & configure a local virtual machine using a platform such as VirtualBox (https://www.virtualbox.org/) & downloading pre-configured VM's (such as the TL VM: https://www.tracelabs.org/initiatives/osint-vm) or installing an operating system from scratch
Install TOR/darknet access on the virtual machine
Configure a VPN on your standard workstation
Connect to the darknet from within the virtual machine
Provision a standalone research laptop/computer (consider using bootable operating systems such as Tails for lower-attribution)
Configure a VPN on your research laptop
Install TOR/darknet access natively on the research laptop
Connect to the darknet natively from your research laptop
Disclaimer: anything you view on the darknet that is rendered locally can still be stored in local caches on any of the options above. You must consider the legal aspects of what you are viewing in the context of your respective governing laws. OSINT Combine takes no responsibility for the content viewed or access methods detailed above.
Dark Web Searching
The dark web is crawled and indexed from numerous non-standard providers, i.e. your traditional search engines such as Google & Bing will not crawl .onion sites on the TOR network. However, proxied TOR sites, being those which use TOR2WEB type services to allow users to view dark web sites from their standard clear web browser, are regularly indexed on Google, although it is not advisable to access these through a proxy for attribution reasons.
Search engines routinely change .onion addresses or go up/down. If any of the following links become unavailable, or you are looking for search engine URLs, a great site is https://onion.live/ - simply search for the search engine name here to find its URL.
Alternatively, there are many "hidden" wiki sites that provide a catalog of resources that also include active search engine URLs. Below is a sample of some of the search engines available.
Dark Web Search Engines: (descriptions obtained from the host websites)
Not all search engines are created equal. They vary significantly with results (due to the sites that are crawled with their platforms), advanced search options (such as boolean or multilingual searching), and intent. Investigators will ideally look across multiple search engines as part of their investigation and compare results, or target their searching to search engines that provide a particular service.
Some investigators will have a requirement to identify & monitor new .onion sites as they arise. This could be to observe patterns, identify new vectors, or simply to create additional pipelines of new .onion URLs to feed into custom crawling engines for advanced users.
There are three good resources that can support this requirement:
Hunchly Dark Web Monitoring (https://www.hunch.ly/darkweb-osint/). The great folks at Hunchly provide a daily email of newly identified .onion domains from their service. This is handy for routine reporting without having to visit the dark web
H-Indexer (http://jncyepk6zbnosf4p.onion/onions.html). This TOR site provides a list of the onions it has indexed including language, title, URL & last contacted in text format