Since the late 1990’s, Google has become the de-facto search engine king of the internet. More than 80% of internet users rely on it to find information that comes from sources all over the web. Google relies on a complex ranking algorithm that determines which order to present their search results to the end user. Before the pages can be ranked, however, they must be discovered.
Conventional search engines like Google rely on automated programs, called spiders, to “read” and catalog all of the information on the internet. This approach means that their findings are limited to formally created HTML pages which have been tagged to be indexed. The problem lies in the fact that the data on the internet isn’t all on neatly formatted pages, and has been estimated to be nearly 500 times the total size of the conventionally searchable internet.
What Have I Been Missing?
Most of the information that the major search engines don’t reach is contained within databases and private services that require registration or subscriptions to gain access. These locations cannot be automatically cataloged because they normally require human intervention to be used. As a result, a whole trove of information that is technically publicly “accessible” lies largely unknown and unused by the wider audience of the internet. Some of the types of information that isn’t normally indexed include public records, personal profiles, and files stored online by individuals.
What It Is And How To Get There
As a response to this massive, underutilized wealth of data, a whole new generation of tools has begun to spring up to provide new ways to access it. The type of data that a user is looking for determines which tools they’ll need to be able to find and access it. The “unindexed” resources that exist on the internet have been divided into two major categories, the deep web, and the dark web. The major difference is that the former is generally accessible to all users if they know where to look, while the latter relies on a closed network referred to as a darknet. Specific software configurations are typically required in order to access resources via these networks, and they are usually encrypted to prevent casual access.
What is The Deep Web?
The deep web refers specifically to the parts of the public internet that aren’t indexed by search engines but are publicly accessible. Originally coined as a search engine term in 2001, the phrase has gained new attention recently due to the rise in identity theft and hacking incidents. It can contain large amounts of personal information that would normally be only accessible through password-protected web portals or exist on dynamic pages that don’t have a statically linked URL.
How Can You Access It
In order to search for information located in the typically unindexed portions of the internet, you don’t need any specialized hardware or software. Since the data is already publically available, you’ll just need to find resources that can show you how to access the deep web. A good place to start would be a specialized search engine for the type of information you’re looking for. Here’s a deep web search engine list that will get you started.
- FreeBackgroundCheck.org – A public records database that utilizes hard to find data to match names to email addresses and phone numbers
- The WWW Virtual Library – The oldest search index online, which relies on volunteers to add and curate data from their fields of expertise
- DuckDuckgo – Pulling from over 400 sources, a search engine that focuses on quality deep web links over quantity
- Stumpedia – The human-powered social and web search engine
- Infoplease – Provides encyclopedic access to a variety of e-books, catalogs, bulletin boards, and mailing lists
What Is The Dark Web?
The dark web is a much smaller subset of unindexed sites that are typically accessed through specialized software. The sites themselves are located on decentralized encrypted networks that are isolated from the public internet. Many of the sites and content available in this way are either dealing in illegal activities or have some reason to attempt to remain clandestine. A high-profile example of this is the now infamous “Silk Road”, which was a hidden site on the Tor network that served as a marketplace for all manner of illegal products including drugs and falsified identification. The FBI shut down the site in 2013 and the founder was arrested and imprisoned.
What It The Tor Network?
The Tor network is a popular decentralized network client that facilitates anonymous internet access and the hosting and maintenance of hidden websites. It is one of the major encrypted peer-to-peer systems used for this purpose. In short, the way it works is that a user initiates a tor browser download, which provides the software needed to access the network. Once installed, the internet traffic passing through the software is encrypted multiple times, and then randomly routed through multiple stops on the network before reaching its’ destination. It is in this way that the system works to prevent tracking user behavior through network monitoring, as no hop along the route is ever in possession of the whole end-to-end data path. For this reason, it’s a popular method of how to access the darknet.
The other major benefit that the Tor network provides is enhanced security and anonymity. Many people use the network as a means of preventing their web activity, such as search history, browsing activity, and file download records from being recorded or viewed by others. If you’re planning to install the Tor browser for this purpose, make sure that you follow the best practices guide on the Tor Project website. It’s common that people looking for security and anonymity falsely believe that Tor will protect all of their internet traffic when in reality it only protects traffic that originates in the Tor browser itself.
You Joined The Tor Network, Now What?
Once you’ve installed the Tor software, you’ll quickly see how to access the dark web contained therein. The biggest problem is that there’s no easy roadmap to these hidden sites, and you must have a way to find the direct dark web links to take you to what you’re looking for. Fortunately, much like their less-hidden counterparts, there are catalogs that are searchable in a few places. This system will take some getting used to since unlike the public internet, the links to these hidden sites don’t have user-friendly or even easily readable names. The one thing you’ll notice is that they all end in the domain “.onion”, which is a reference to the layers of security built into the platform. While this adds additional anonymity and privacy, it also creates a major challenge for users trying to keep track of useful resources. To make this process a little easier, below is a dark web search engine list and some excellent resources. Remember, some hidden sites may contain illegal products and activities, so whatever you browse will be at your own risk.
On The Public Internet
- Ahmia – Aims to make hidden sites accessible to all users in a direct, easy to navigate fashion
- The Hidden Wiki – A listing of categorized popular hidden sites
- Deep Search – Attempts to crawl and index hidden sites for easy access
- The Reddit “Onions” Subreddit – An active user community dedicated to finding and discussing the latest hidden sites
On The Tor Network
- not Evil – A Tor-based site search service
An important thing to keep in mind when looking for hidden sites is that often, they’re kept hidden for a reason. This simple fact means that even the links that you bookmark may only exist for as little as a few days or even hours. It’ll take a fair amount of time and effort to keep yourself well versed on the most popular and useful hidden sites, but you’ll probably find resources that can’t be located in any other way. Be sure to be wary and careful when browsing these sites, because even the best-encrypted services can never be 100% secure and safe to use.
In the end, no matter how you choose to make use of the deepest darkest corners of the world wide web, you’ll never cease to be amazed by what you find. There are already more than five billion websites on the internet, with much more to come. The total size and scope are staggering, and since only a tiny fraction of it can be accessed through traditional search engines, learning how to find everything else is going to become an indispensable skill in very short order.
