How I Accidentally Leaked a Production API Key (And What I Learned About Secret Scanning)

It was 11:30 PM on a Tuesday. I was running on my third cup of cold coffee, pushing the final commits for a client's analytics dashboard. The code was working locally, the tests passed, and all that was left was to push the updates to the master branch of our public GitHub repository. I typed git add ., followed by git commit -m "fix: resolve dashboard data loading issues" and git push origin main. I shut down my laptop, feeling a deep sense of accomplishment.

Five minutes later, my phone started buzzing. I had received a automated email notification from GitHub: *"Security Alert: A known secret has been exposed in commit..."*. Instantly, a wave of panic washed over me. I opened my laptop and realized what had happened: in my rush to fix a local database config issue, I had temporarily hardcoded the production SendGrid and AWS API keys in my config file, planning to move them to environment variables later. I forgot, staged the changes, and pushed them to a public repo.

In this article, I want to walk you through exactly what happens when you push a secret to a public repository, the speed at which attackers exploit this, and the steps you must take to purge your git history and secure your infrastructure.

The Speed of Exploitation: The Bots are Faster Than You

Many developers think, *"Well, my repo has no stars, and I noticed the leak in five minutes. I'll just delete the file or make a new commit removing the keys, and nobody will notice."* This is a critical misconception.

There are malicious bots that monitor the global GitHub public event feed (`https://api.github.com/events`) in real time. They ingest thousands of commits per second, scanning every single diff for common key signatures (such as strings starting with `SG.` for SendGrid, AWS access key patterns `AKIA...`, or private RSA headers). They do not search repositories manually; they programmatically scan the firehose of public git activity.

Once a bot detects a potential key, it automatically tests it against the service provider's API. If the key is valid, automated scripts immediately execute actions based on the permissions of that key. In the case of mail providers like SendGrid or Mailgun, the bots immediately use the credentials to blast spam and phishing campaigns, ruining your sender reputation score instantly.

Why `git rm` Does Not Fix the Problem

When you realize you've pushed a secret, your gut reaction is probably to delete the key, commit the change, and push it again. This resolves the problem in the current file tree, but **the secret remains fully visible in the Git commit history.**

Anyone can navigate to the previous commit in your repository or view the commit diff to see the plaintext string you removed. As long as that commit hash exists in the remote repository's tree, the secret is considered exposed. Service providers like AWS and GitHub will continue to flag the key as compromised because the raw commit is still publicly accessible.

The Proper Remediation Protocol

If you find yourself in this situation, do not panic. Follow this step-by-step containment and remediation protocol immediately.

Step 1: Revoke the Compromised Secret Immediately

Do not try to fix Git first. **Go straight to the source service provider and deactivate the credential.**

• For AWS, log into the IAM console, set the compromised Access Key to "Inactive", and then delete it.
• For Stripe, Mailgun, SendGrid, or other APIs, log into the dashboard, delete the compromised key, and generate a new one.

Revoking the key immediately cuts off the attacker's access, neutralizing the threat regardless of what is in your Git history.

Step 2: Clean Your Git History

Once the key is disabled, you must remove all traces of the secret from your repository's history. There are two main tools for this: **Git Filter-repo** (recommended) and the older **BFG Repo-Cleaner**.

Never use `git filter-branch` as it is slow, error-prone, and deprecated. Instead, install `git-filter-repo` and run it to search and destroy the specific key strings across all commits, branches, and tags.

# 1. Install git-filter-repo (using Homebrew on macOS)
brew install git-filter-repo

# 2. Create a file containing the text to replace (e.g., secrets.txt)
# Format: original_secret==>replacement_placeholder
SG.xyz123apiKeyHere==>REPLACED_SECRET

# 3. Run git-filter-repo to scrub the secret from all history
git filter-repo --replace-text secrets.txt

# 4. Force-push the cleaned history back to your remote repository
git push origin --force --all
git push origin --force --tags

This command completely rewrites your Git history, replacing every occurrence of the secret string with your placeholder across every single commit you’ve ever made.

How to Prevent Leaks: Environment Variables and Pre-Commit Hooks

The only true way to handle secrets is to make it impossible to push them in the first place. You can achieve this using two industry-standard tools: **dotenv** and **pre-commit hooks**.

1. Never Hardcode Keys — Use Environment Variables

In your Node.js or Python backend, always pull credentials from the environment. Create a `.env` file for local development and make sure it is added to your `.gitignore` file so it is never committed.

# .env (Never commit this file!)
PORT=3000
DATABASE_URL=mongodb://localhost:27017/app
SENDGRID_API_KEY=SG.your_actual_key_here

// Read configuration from env variables in your JS code
require('dotenv').config();

const mailer = require('@sendgrid/mail');
mailer.setApiKey(process.env.SENDGRID_API_KEY); // Safe and clean

2. Implement Local Secret Scanning via git Secrets or Gitleaks

You can set up local git hooks that automatically scan your changes before they are committed. If a key is found, the commit is blocked.

We configure **Gitleaks** as a pre-commit hook in our pipelines. Gitleaks checks your staged files against a ruleset of known API key formats and regexes, saving you from accidental additions.

Summary Checklist for Devs

• Never commit credentials: Store all keys in a `.env` file, and double-check that `.env` is listed in your `.gitignore`.
• Revoke first, scrub second: If you leak a secret, disable it instantly. Do not waste time rewiring git commits while the key is active.
• Use git-filter-repo: Purge files and text strings from your git history using modern, safe tools rather than deleting lines.
• Scan early: Set up automated secret scanners locally (Gitleaks) and in your CI/CD pipelines (GitHub Actions) to catch leaks before they hit production.