LeakDB is a tool set designed to allow organizations to build and deploy their own internal plaintext “Have I Been Pwned”-like service. The LeakDB tool set can normalize, deduplicate, index, sort, and search leaked data sets on the multi-terabyte-scale, without the need to distribute large files to individual users. Once curated, LeakDB can search terabytes of data in less than a tenth of a second, and the LeakDB server exposes a simple JSON API that can be queried using the command line client or any http client. It can be deployed in a serverless configuration with a BigQuery backend (no indexes), or as an offline/traditional server with indexes.
LeakDB uses a configurable bloom filter to remove duplicate entires, sorts indexes using external parallel quicksort (i.e., memory constrained) with a k-way binary tree merge, and binary tree search to find entries in the index.
Usage
Normalizing Data
The first step is to normalize your data sets, details on how to do this can be found here. Be sure to keep all your normalized files in a single directory. This step is the same for both the serverless and server deployments.
Next you’ll need to decide if you want to setup a traditional server deployment or a serverless deployment, there are benefits and drawbacks to each as described below.
The server deployment is the cheapest way to setup LeakDB, but you’ll have to compute indexes yourself. The amount of time it takes to compute an index varies depending on the data set and your hardware. LeakDB can sort multiple terabyte data sets on almost any hardware but it could take a very long time. Domain indexes in particular can take a very long time to compute due to the high number of collisions. However, you only have to pay the computational price once. That is to say once-per-data-set, if you’ve already computed an index and want to add additional data you’ll need to re-sort the entire index (you can save bloom filter results, etc. though), so be sure to collect and normalize all your data before computing the index.
The severless deployment generally costs more money to run as the backend is BigQuery. With BigQuery you pay for data storage as well paying per-query; it isn’t particularly expensive unless you start running lots of queries against the data set. It’s also much faster to setup since you don’t have to compute indexes yourself.
Compile From Source
Just run make <platform>, files will be put in ./bin. The client, curator, and server are pure Go and should support any valid Go compiler target but you may need to modify the Makefile. The serverless binary is Linux only, since AWS Lambda only supports Linux. The easiest way to compile the Windows binaries is to cross-compile them from a better operating system like Linux or MacOS.
For example:
make macosmake linuxmake windowsThe sort command in Linux reads lines from files or standard input and writes them to standard…
The wall command in Linux sends a message to the terminals of all currently logged-in users. The…
journalctl queries logs collected by systemd-journald, the systemd logging daemon. It gives you structured access to kernel…
The stat command in Linux displays detailed metadata about files and filesystems. Where ls gives a condensed summary suitable…
In Linux, groups organize user accounts and define shared access to files and resources. Every…
The touch command in Linux does two things: it creates new empty files, and it updates the…