blairjames/squeaky

Fast word list cleaner and de-duplicator.

• Filters problem characters that play havoc with UTF-8 and removes duplicates to avoid inefficient use of resources.
  
• Allows removal of words below a specified length.
  
• Very handy when optimizing and combining multiple large wordlists.
  
  

Usage:

git clone https://github.com/blairjames/squeaky.git

squeaky.py [-h] [-d] [-l LEN] [-u] input_file output_file

positional arguments:

--input_file

--output_file

options:

-h, --help - show this help message and exit.

-d, --dir - Input a directory to process for word lists. (".txt" files)

-l LEN, --len LEN - Minimum word length, words shorter than specified length will be
removed.

-u, --unique - Delete duplicate words in word list.