Adding detectors (#46)
* rename secret * Add supporting docs and tooling for adding new detectors
This commit is contained in:
Dustin Decker
@@ -0,0 +1,125 @@
|
||||
# Secret Detectors
|
||||
|
||||
Secret Detectors have these two major functions:
|
||||
|
||||
1. Given some bytes, extract possible secrets, typically using a regex.
|
||||
2. Validate the secrets against the target API, typically using a HTTP client.
|
||||
|
||||
The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Secret Detectors](#secret-detectors)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Sourcing Guidelines](#sourcing-guidelines)
|
||||
- [Development Guidelines](#development-guidelines)
|
||||
- [Development Dependencies](#development-dependencies)
|
||||
- [Creating a new Secret Scanner](#creating-a-new-secret-detector)
|
||||
- [Addendum](#addendum)
|
||||
- [Managing Test Secrets](#managing-test-secrets)
|
||||
- [Setting up Google Cloud SDK](#setting-up-google-cloud-sdk)
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Sourcing Guidelines
|
||||
|
||||
We are interested in detectors for services that meet at least one of these criteria
|
||||
- host data (they store any sort of data provided)
|
||||
- have paid services (having a free or trial tier is okay though)
|
||||
|
||||
If you think that something should be included outside of these guidelines, please let us know.
|
||||
|
||||
### Development Guidelines
|
||||
|
||||
- When reasonable, favor using the `net/http` library to make requests instead of bringing in another library.
|
||||
- Use the [`common.SaneHttpClient`](pkg/common/http.go) for the `http.Client` whenever possible.
|
||||
- We recommend an editor with gopls integration (such as Vscode with Go plugin) for benefits like easily running tests, autocompletion, linting, type checking, etc.
|
||||
|
||||
### Development Dependencies
|
||||
|
||||
- A GitLab account
|
||||
- A Google account
|
||||
- [Google Cloud SDK installed](#setting-up-google-cloud-sdk)
|
||||
- Go 1.16
|
||||
- Make
|
||||
|
||||
### Creating a new Secret Scanner
|
||||
|
||||
1. Identify the Secret Scanner name from the [proto/secrets.proto](proto/secrets.proto) `DetectorType` enum.
|
||||
|
||||
2. Generate the SecretScanner
|
||||
|
||||
```bash
|
||||
go run hack/generate/generate.go detector <DetectorType enum name>
|
||||
```
|
||||
|
||||
3. Complete the secret detector.
|
||||
|
||||
The previous step templated a boilerplate + some example code as a package in the `pkg/secrets` folder for you to work on.
|
||||
The secret detector can be completed with these general steps:
|
||||
|
||||
1. Add the test secret to GCP Secrets. See [managing test secrets](#managing-test-secrets)
|
||||
2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
|
||||
3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
|
||||
4. Update the tests with these test cases at minimum:
|
||||
1. Found and verified (using a credential loaded from GCP Secrets)
|
||||
2. Found and unverified
|
||||
3. Not found
|
||||
4. Any false positive cases that you come across
|
||||
5. Create a merge request for review. CI tests must be passing.
|
||||
|
||||
## Addendum
|
||||
|
||||
### Managing Test Secrets
|
||||
|
||||
Do not embed test credentials in the test code. Instead, use GCP Secrets Manager.
|
||||
|
||||
1. Access the latest secret version for modification.
|
||||
|
||||
Note: `/tmp/s` is a valid path on Linux. You will need to change that for Windows or OSX, otherwise you will see an error. On Windows you will also need to install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
|
||||
|
||||
```bash
|
||||
gcloud secrets versions access --project thog-contractor-tests --secret detectors3 latest > /tmp/s
|
||||
```
|
||||
|
||||
2. Add the secret that you need for testing.
|
||||
|
||||
The command above saved it to `/tmp/s`.
|
||||
|
||||
The format is standard env file format,
|
||||
|
||||
```bash
|
||||
SECRET_TYPE_ONE=value
|
||||
SECRET_TYPE_ONE_INACTIVE=v@lue
|
||||
```
|
||||
|
||||
3. Update the secret version with your modification.
|
||||
|
||||
```bash
|
||||
gcloud secrets versions add --project thog-contractor-tests detectors3 --data-file /tmp/s
|
||||
```
|
||||
|
||||
4. Access the secret value as shown in the [example code](pkg/secrets/heroku/heroku_test.go).
|
||||
|
||||
### Setting up Google Cloud SDK
|
||||
|
||||
1. Install the Google Cloud SDK: https://cloud.google.com/sdk/docs/install
|
||||
2. Authenticate with `gcloud auth login --update-adc` using your Google account
|
||||
|
||||
### Adding Protos in Windows
|
||||
|
||||
1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
|
||||
2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
|
||||
3. Open Ubuntu cli and install `dos2unix`.
|
||||
```bash
|
||||
sudo apt install dos2unix
|
||||
```
|
||||
4. Identify the `trufflehog` local directory and convert `scripts/gen_proto.sh` file in Unix format.
|
||||
```bash
|
||||
dos2unix ./scripts/gen_proto.sh
|
||||
```
|
||||
5. Open [proto/secrets.proto](proto/secrets.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
|
||||
```bash
|
||||
make protos
|
||||
```
|
||||
@@ -0,0 +1,126 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"text/template"
|
||||
|
||||
"github.com/go-errors/errors"
|
||||
"gopkg.in/alecthomas/kingpin.v2"
|
||||
)
|
||||
|
||||
var (
|
||||
app = kingpin.New("generate", "Generate is used to write new features.")
|
||||
kind = app.Arg("kind", "Kind of thing to generate.").Required().Enum("detector")
|
||||
name = app.Arg("name", "Name of the Source/Detector to generate.").Required().String()
|
||||
nameTitle, nameLower, nameUpper string
|
||||
)
|
||||
|
||||
func main() {
|
||||
log.SetFlags(log.Lmsgprefix)
|
||||
log.SetPrefix("😲 [generate] ")
|
||||
|
||||
kingpin.MustParse(app.Parse(os.Args[1:]))
|
||||
nameTitle = strings.Title(*name)
|
||||
nameLower = strings.ToLower(*name)
|
||||
nameUpper = strings.ToUpper(*name)
|
||||
|
||||
switch *kind {
|
||||
case "detector":
|
||||
mustWriteTemplates([]templateJob{
|
||||
{
|
||||
TemplatePath: "pkg/detectors/heroku/heroku.go",
|
||||
WritePath: filepath.Join(folderPath(), nameLower+".go"),
|
||||
ReplaceString: []string{"heroku"},
|
||||
},
|
||||
{
|
||||
TemplatePath: "pkg/detectors/heroku/heroku_test.go",
|
||||
WritePath: filepath.Join(folderPath(), nameLower+"_test.go"),
|
||||
ReplaceString: []string{"heroku"},
|
||||
},
|
||||
})
|
||||
// case "source":
|
||||
// mustWriteTemplates([]templateJob{
|
||||
// {
|
||||
// TemplatePath: "pkg/sources/filesystem/filesystem.go",
|
||||
// WritePath: filepath.Join(folderPath(), nameLower+".go"),
|
||||
// ReplaceString: []string{"filesystem"},
|
||||
// },
|
||||
// {
|
||||
// TemplatePath: "pkg/sources/filesystem/filesystem_test.go",
|
||||
// WritePath: filepath.Join(folderPath(), nameLower+"_test.go"),
|
||||
// ReplaceString: []string{"filesystem"},
|
||||
// },
|
||||
// })
|
||||
}
|
||||
}
|
||||
|
||||
type templateJob struct {
|
||||
TemplatePath string
|
||||
WritePath string
|
||||
ReplaceString []string
|
||||
}
|
||||
|
||||
func mustWriteTemplates(jobs []templateJob) {
|
||||
log.Printf("Generating %s %s\n", strings.Title(*kind), nameTitle)
|
||||
|
||||
// Make the folder.
|
||||
log.Printf("Creating folder %s\n", folderPath())
|
||||
err := makeFolder(folderPath())
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Write the files from templates.
|
||||
for _, job := range jobs {
|
||||
tmplBytes, err := ioutil.ReadFile(job.TemplatePath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
tmplRaw := string(tmplBytes)
|
||||
|
||||
for _, rplString := range job.ReplaceString {
|
||||
tmplRaw = strings.ReplaceAll(tmplRaw, strings.ToLower(rplString), "<<.NameLower>>")
|
||||
tmplRaw = strings.ReplaceAll(tmplRaw, strings.Title(rplString), "<<.NameTitle>>")
|
||||
tmplRaw = strings.ReplaceAll(tmplRaw, strings.ToUpper(rplString), "<<.NameUpper>>")
|
||||
}
|
||||
|
||||
tmpl := template.Must(template.New("main").Delims("<<", ">>").Parse(tmplRaw))
|
||||
|
||||
log.Printf("Writing file %s\n", job.WritePath)
|
||||
f, err := os.OpenFile(job.WritePath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
tmpl.Execute(f, templateData{
|
||||
NameTitle: nameTitle,
|
||||
NameLower: nameLower,
|
||||
NameUpper: nameUpper,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type templateData struct {
|
||||
NameTitle string
|
||||
NameLower string
|
||||
NameUpper string
|
||||
}
|
||||
|
||||
func folderPath() string {
|
||||
return filepath.Join("pkg/", *kind+"s", nameLower)
|
||||
}
|
||||
|
||||
func makeFolder(path string) error {
|
||||
_, err := os.Stat(path)
|
||||
if os.IsNotExist(err) {
|
||||
err := os.MkdirAll(path, 0755)
|
||||
if err != nil {
|
||||
return errors.New(err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return errors.Errorf("%s %s already exists", *kind, *name)
|
||||
}
|
||||
Executable
+16
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eu
|
||||
|
||||
function cleanup {
|
||||
rm -rf pkg/detectors/test
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
export CGO_ENABLED=0
|
||||
|
||||
export FORCE_PASS_DIFF=true
|
||||
|
||||
echo "████████████ Testing generate Detector"
|
||||
go run hack/generate/generate.go detector Test
|
||||
go test ./pkg/detectors/test -benchmem -bench .
|
||||
echo ""
|
||||
Reference in New Issue
Block a user