Talisman hooks into a Git repository to validate the outgoing change set for things that look suspicious, like potential SSH keys, authorisation tokens, private keys, etc. It can scan the source code to catch potential problems in embedded source code and documentation. This tool can help prevent major disasters related to security.
Version control software (VCS) is an essential part of almost all modern software development practices. Software like Git allows developers and quality analysts to save their project history to enable better collaboration, revert to previous states and recover from unintended code changes. These tools allow multiple developers to safely work on the same project and provide significant benefits even if sharing their work with others is not required. With the many benefits of keeping your code in source control, it is equally important for some assets to be kept out of your repository.
Some of the most important assets are sensitive data like passwords, secrets and private keys which should never be checked into a repository unprotected, for security reasons. Overlooking this may result in organisations having to pay out heavy compensations, and from the individual perspective, lead to a loss of privacy or more. One of the recent incidents was Uber’s secrets breach, which should give you an idea about how far this could go, if not taken care of early.
According to Bloomberg, “Uber’s 2016 breach occurred when hackers discovered that the company’s developers had published code that included their user names and passwords on a private account of the software repository GitHub. Those credentials gave the hackers immediate access to the developers’ privileged accounts on Uber’s network and with it, access to sensitive Uber servers hosted on Amazon’s servers, including the rider and driver data they stole.”
As mentioned by CNBC, “Uber has agreed to pay US$ 148 million in connection with a 2016 data breach and subsequent cover-up, according to the California Attorney General’s office.” If this convinces you about the importance of keeping secrets away from version control software, it will delight you to know that such prevention is basically a five-minute job.
In the remaining part of the article, I will walk you through a tool called Talisman, which can help you to prevent disasters, if you agree to spend just five minutes on it.
The Talisman way
Talisman is an open source tool created by Thoughtworks. It installs a hook to a repository to ensure that potential secrets or sensitive information do not leave the developer’s workstation. It validates the outgoing change set for things that look suspicious such as potential SSH keys, authorisation tokens, private keys, etc. It supports MacOSX, Linux and Windows. Talisman can be installed as a pre-commit hook or a pre-push hook. However, I strongly recommend the pre-commit hook, because it will help you to save time on editing your commit message again. Talisman sits on your machine’s home (or a parent location of your choice, some place where you keep all your Git repositories) as a Git hook. You can install it once and it will take care of any secrets being accidentally pushed to VCS from your existing or new Git repositories. Once installed, Talisman’s auto update mechanism takes care of updating new features to the installation whenever there is a new release. To install Talisman, run the following command on a terminal. This will download and install the binary at $HOME/.talisman/bin.
As a pre-commit hook (recommended), give the command:
curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash
As a pre-push hook, give the command:
curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash pre-push
Follow the simple instructions in your terminal and you are done! Let’s look at how it works. After you install Talisman, it will run checks for obvious secrets automatically, whenever a Git’s commit/push command is invoked (as chosen during installation). In case there are any potential secrets detected, Talisman will display a detailed report of the errors:
$ git commit -m “Commit Message" Talisman Report: +-----------------+--------------------------------------------------------------+ | FILE | ERRORS | +-----------------+--------------------------------------------------------------+ | danger.pem | The file name "danger.pem" | | | failed checks against the || | pattern ^.+\.pem$ | +-----------------+--------------------------------------------------------------+ | danger.pem | Expected file to not to contain hex encoded texts such as: awsSecretKey=c64e8c79aacf5ddb02f1274db2d973f363f4f553ab16 | +-----------------+--------------------------------------------------------------+
Talisman CLI support for existing Git repositories
We just figured out how Talisman prevents sensitive information from leaving the developer’s or QA’s machine and getting checked in to VCS. This works perfectly for any new repositories that you create. Now, what about the secrets that were already checked in to an existing repository? How do you look at the complete Git history and find out secrets which were accidentally checked in before, and remove them? Talisman also supports a CLI tool, which you can run from your repo. This finds out existing secrets in a Git repository. Here’s how you can use the Git History Scanner support of the Talisman CLI (you can potentially add it to your CI/CD pipeline to make secret scanning a part of your deployment process).
If you execute the command talisman from your repository, you will be presented with the following options.
--c string: Short form of checksum calculator --checksum string: Checksum calculator calculates checksum and suggests .talsimarc format --d: Short form of debug --debug: Enable debug mode (warning: very verbose) --githook string: Either pre-push or pre-commit (default ‘pre-push’) --p string: Short form of pattern --pattern string: Pattern (glob-like) of files to scan (ignores git hooks) --s: Short form of scanner --scan: Scanner scans the git commit history for potential secrets --scanWithHTML: Scans the repo and generates pretty HTML reports --v: Short form of version --version: Shows current version of talisman
Follow these steps to perform the scan:
- To get into the Git directory path to be scanned, use the command cd <directory to scan>
- To run the scan command, type the command talisman –scanWithHTML
- Running this command will create a folder named talisman_html_reports in the root of the current directory and store the report files there.
The above steps take us to the final point, which is manual analysis.
talisman –scanWithHTML creates pretty and easy-to-understand reports, which help the team to understand the errors caught by Talisman for the specific repository. Figures 1, 2, 3 and 4 show some sample reports that you can archive in your CI/CD environment.
It is important to understand that securing your secrets is absolutely essential because sometimes the stakes are really high. The consequences can destroy an organisation’s or an individual’s reputation in no time. Taking absolute care of our secrets is a practice that we need to cultivate and use the tools to make our lives easier. By now the industry has probably started to understand the importance of pushing security towards the beginning of the development life cycle to mitigate the cost of finding bugs towards the end. Taking care of sensitive data is a part of this process as well, and a shift in mindset is needed to achieve the ‘Shift left’ in security as well.
