Duplicut : Remove Duplicates From MASSIVE Wordlist, Without Sorting It

Duplicut is a modern password wordlist creation usually implies concatenating multiple data sources.

Ideally, most probable passwords should stand at start of the wordlist, so most common passwords are cracked instantly.

With existing dedupe tools you are forced to choose if you prefer to preserve the order OR handle massive wordlists.

Unfortunately, wordlist creation requires both:

So i wrote duplicut in highly optimized C to address this very specific need

Quick start

git clone https://github.com/nil0x42/duplicut
cd duplicut/ && make
./duplicut wordlist.txt -o clean-wordlist.txt

Options

Features

  • Handle massive wordlists, even those whose size exceeds available RAM
  • Filter lines by max length (-l option)
  • Can remove lines containing non-printable ASCII chars (-p option)
  • Press any key to show program status at runtime.

Implementation

  • Written in pure C code, designed to be fast
  • Compressed hashmap items on 64 bit platforms
  • Multithreading support
  • [TODO]: Use huge memory pages to increase performance

Limitations

  • Any line longer than 255 chars is ignored
  • Heavily tested on Linux x64, mostly untested on other platforms.

Technical Details

1- Memory optimized

An uint64 is enough to index lines in hashmap, by packing size info within pointer’s extra bits:

2- Massive file handling

If whole file can’t fit in memory, it is split into  virtual chunks, then each one is tested against next chunks.

So complexity is equal to th triangle number:

Throubleshotting

If you find a bug, or something doesn’t work as expected, please compile duplicut in debug mode and post an issue with attached output:

debug level can be from 1 to 4
make debug level=1
./duplicut [OPTIONS] 2>&1 | tee /tmp/duplicut-debug.log
R K

Recent Posts

Starship : Revolutionizing Terminal Experiences Across Shells

Starship is a powerful, minimal, and highly customizable cross-shell prompt designed to enhance the terminal…

1 hour ago

Lemmy : A Decentralized Link Aggregator And Forum For The Fediverse

Lemmy is an innovative, open-source platform designed for link aggregation and discussion, providing a decentralized…

1 hour ago

Massive UX Improvements, Custom Disassemblers, And MSVC Support In ImHex v1.37.0

The latest release of ImHex v1.37.0 introduces a host of exciting features and improvements, enhancing…

3 hours ago

Ghauri : A Powerful SQL Injection Detection And Exploitation Tool

Ghauri is a cutting-edge, cross-platform tool designed to automate the detection and exploitation of SQL…

6 hours ago

Writing Tools : Revolutionizing The Art Of Writing

Writing tools have become indispensable for individuals looking to enhance their writing efficiency, accuracy, and…

6 hours ago

PatchWerk : A Tool For Cleaning NTDLL Syscall Stubs

PatchWerk is a proof-of-concept (PoC) tool designed to clean NTDLL syscall stubs by patching syscall…

1 day ago