Cyber security

Would You Like To Know More In Tartufo – Cleaning Up Git Repositories Of Sensitive Data

If the other documentation left you wondering what to do with the results of your scans, and unsure how to get rid of those pesky leaked secrets, then look no further!

End-to-End Example

An End-to-End example walkthrough of a tartufo scan and the process of purging the dirty evil passwords that somehow ended up in your code commits. 

  1. Clone your repo!

Select and clone the repo you want to run tartufo on

Clone your repo, variables used later:
GITHUBPROJECT="yourproject"
GITHUBREPO="myrepo.git"
GITHUBADDRESS="github.com"
git clone --mirror git@${GITHUBADDRESS}:${GITHUBPROJECT}/${GITHUBREPO}

2. Use tartufo to scan your repository and find any secrets in its history!

Scan your repo!

Run Tartufo on your repo and create a list of high entropy items to remove:
tartufo --regex --output-format json scan-local-repo ${GITHUBREPO} | \
    jq -r '.found_issues[].matched_string' | \
    sort -u > remove.txt

Now you have a “bad password” file! Take a look through it, see if anything is wrong. This file will be used by BFG to replace these flagged “bad password” entries with ***REMOVED***.

3. Cleanup repo using BFG and the above remove.txt file

There’s a very slick tool designed to clean up git commit history called BFG. By default, BFG doesn’t modify the contents of your latest commit on your main (or ‘HEAD’) branch, even though it will clean all the commits before it.

This of course means if you have active code with “bad passwords”, tartufo will still fail. But let’s take the bulk of the old entries out first.

Cleanup with BFG
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.13.2/bfg-1.13.2.jar
Make a backup
cp -r ${GITHUBREPO} backup_${GITHUBREPO}
java -jar bfg-1.13.2.jar --replace-text remove.txt ${GITHUBREPO}

4. Uh Oh!

Occasionally the results will be too big to process all at once. If that happens, simply split up the results and loop through them.

occasionally the results will be to big to process all at once
split -l 200 remove.txt
for f in x*; do java -jar bfg-1.13.2.jar --replace-text $f ${GITHUBREPO}; done

5. Proceed with cleanup/audit

Now you have removed the low hanging fruit, it’s time to look at the tough stuff

run tartufo again to check for any remaining potential secrets
leftovers=`tartufo --regex -od ~/temp scan-local-repo ${GITHUBREPO}`
tmppath=`echo -e "$leftovers" | tail -n1 | awk '{print $6}'`
look through the remaining strings
if there's anything that looks like it shouldn't be there, dig into it and clear it out
cat ${tmppath}/* | jq '. | " \(.file_path) \(.matched_string) \(.signature)"' | sort -u

For more onformation click here.

Varshini

Varshini is a Cyber Security expert in Threat Analysis, Vulnerability Assessment, and Research. Passionate about staying ahead of emerging Threats and Technologies.

Recent Posts

WID_LoadLibrary : The Intricacies Of DLL Management In Windows

WID_LoadLibrary is a custom implementation inspired by the Windows API function LoadLibrary, which is used…

14 hours ago

Locksmith : A Tool For Securing Active Directory Certificate Services

Locksmith is a specialized tool designed to identify and remediate vulnerabilities in Active Directory Certificate…

14 hours ago

Uscrapper Vanta : A Cutting-Edge OSINT Tool For Advanced Data Extraction

Uscrapper Vanta is a powerful open-source intelligence (OSINT) tool designed to revolutionize web scraping and…

14 hours ago

Pake : Transforming Webpages Into Desktop Applications

Pake is an innovative tool designed to convert any webpage into a desktop application with…

19 hours ago

Bevy : Exploring The Frontier Of Game Development With Rust

Bevy is an open-source, data-driven game engine built in Rust, designed to simplify game development…

19 hours ago

AppFlowy Cloud : Enhancing Collaboration With Secure Cloud Infrastructure

AppFlowy Cloud is a robust component of the AppFlowy ecosystem, designed to provide secure user…

2 days ago