re-add actions CLI switch to action yml

* Add GitHub Actions output

Co-authored-by: Mike Vanbuskirk <mike.vanbuskirk@trufflesec.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,31 +3,56 @@ name: Bug report
 about: Create a report to help us improve
 title: ""
 labels: bug, needs triage
-assignees: dustin-decker, bill-rich
+assignees: trufflesecurity/product-eng
 ---
 
-**Describe the bug**
-A clear and concise description of what the bug is.
+### Community Note
 
-**To Reproduce**
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
+* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
+* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
+* If you are interested in working on this issue or have submitted a pull request, please leave a comment
 
-**Expected behavior**
-A clear and concise description of what you expected to happen.
+### TruffleHog Version
+<!--- Please run `trufflehog --version` to show the version. If you are not running the latest version, please upgrade because your issue may have already been fixed. --->
 
-**Output**
-Please include relevant log output, and run with the `--debug` flag.
+### Trace Output
 
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
+<!---
+Please provide a link to a GitHub Gist containing the complete debug output. Please do NOT paste the debug output in the issue; just paste a link to the Gist.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Version [e.g. 22]
+To obtain the trace output, run trufflehog with the --trace flag.
+--->
+
+### Expected Behavior
+
+<!--- What should have happened? --->
+
+### Actual Behavior
+
+<!--- What actually happened? --->
+
+### Steps to Reproduce
+
+<!--- Please list the steps required to reproduce the issue. --->
+ 1. Go to '...'
+ 2. Click on '....'
+ 3. Scroll down to '....'
+ 4. See error
+
+## Environment
+ * OS: [e.g. iOS]
+ * Version [e.g. 22]
+
+## Additional Context
+<!--- Add any other context about the problem here. --->
+
+### References
+
+<!---
+Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
+
+Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor documentation? For example:
+--->
+
+* #0000
 
-**Additional context**
-Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -3,17 +3,36 @@ name: Feature request
 about: Suggest an idea for this project
 title: ""
 labels: enhancement, needs triage
-assignees: dustin-decker, bill-rich
+assignees: trufflesecurity/product-eng
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+### Community Note
 
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
+* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
+* If you are interested in working on this issue or have submitted a pull request, please leave a comment
 
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+### Description
 
-**Additional context**
-Add any other context or screenshots about the feature request here.
+<!--- Please leave a helpful description of the feature request here. --->
+
+## Problem to be Addressed
+<!--- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] --->
+
+## Description of the Preferred Solution
+<!--- A clear and concise description of what you want to happen. What 
+information may be required and what would be the preferred way to provide it? 
+What should the output include? --->
+
+## Additional Context
+<!--- Add any other context or screenshots about the feature request here. --->
+
+### References
+
+<!---
+Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
+
+Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
+--->
+
+* #0000

.github/workflows/TESTING.md

@@ -0,0 +1,19 @@
+# Testing
+
+Most testing is handled automatically by our GitHub Actions workflows.
+
+## Local GitHub Action Testing
+
+In some cases you may wish to submit changes to the Trufflehog GitHub Action. Unfortunately GitHub does not provide a 1st-party testing environment for testing actions outside of GitHub Actions.
+
+Fortunately [nektos/act](https://github.com/nektos/act) enables local testing of GitHub Actions.
+
+### Instructions
+
+1. Please follow [the installation instructions](http://https://github.com/nektos/act#installation) for your OS.
+2. The first run of `act` will ask you to specify an image. `Medium` should suffice.
+3. You'll need to configure a personal-access-token(PAT) with: `repo:status`, `repo_deployment`, and `public_repo` permissions.
+4. Set an environment variable named `GITHUB_TOKEN` with the PAT from the previous step as the value: `$ export GITHUB_TOKEN=<your_PAT>`
+5. Run the following command from the repository root: `act pull_request -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`
+6. If the job was successful, you should expect to see output from the scanner showing several detected secrets.
+7. If you want to omit the context of a pull request event and just test that the action starts successfully, run: `act -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`

.github/workflows/codeql-analysis.yml

@@ -38,11 +38,11 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v1
+      uses: github/codeql-action/init@v2
       with:
         languages: ${{ matrix.language }}
         # If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,7 +53,7 @@ jobs:
     # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v1
+      uses: github/codeql-action/autobuild@v2
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 https://git.io/JvXDl
@@ -67,4 +67,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v1
+      uses: github/codeql-action/analyze@v2

.github/workflows/dogfood.yml

@@ -1,30 +0,0 @@
-
-name: Dogfood
-
-on:
-  push:
-    tags:
-      - v*
-    branches:
-      - main
-  pull_request:
-
-jobs:
-  test:
-    strategy:
-      matrix:
-        go-version: [1.17.x]
-        os: [ubuntu-latest]
-    runs-on: ${{ matrix.os }}
-    steps:
-    - name: Install Go
-      uses: actions/setup-go@v2
-      with:
-        go-version: ${{ matrix.go-version }}
-    - name: Checkout code
-      uses: actions/checkout@v3
-      with:
-        fetch-depth: 0
-        ref: ${{ github.head_ref }}
-    - name: Dogfood
-      run:  echo $GITHUB_BASE_REF; CGO_ENABLED=0 go run . git file://. --since_commit $GITHUB_BASE_REF

.github/workflows/lint.yml

@@ -17,15 +17,15 @@ jobs:
     name: lint
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/setup-go@v2
+      - uses: actions/setup-go@v4
         with:
-          go-version: '1.17'
+          go-version: '1.18'
       - uses: actions/checkout@v3
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
           # Optional: version of golangci-lint to use in form of v1.2 or v1.2.3 or `latest` to use the latest version
-          version: v1.43
+          version: latest
           # Optional: working directory, useful for monorepos
           # working-directory: somedir
 

.github/workflows/release.yml

@@ -1,17 +1,9 @@
 name: Release
 
-# on:
-#   push:
-#     tags:
-#       - 'v*'
-
 on:
   push:
     tags:
       - v*
-    branches:
-      - main
-  pull_request:
 
 permissions:
   contents: write
@@ -28,22 +20,28 @@ jobs:
         with:
           fetch-depth: 0
       - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
-      - name: Docker Login
-        uses: docker/login-action@v1
+        uses: docker/setup-qemu-action@v2
+      - name: Docker Login to DockerHub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+      - name: Docker Login to GitHub Container Registry
+        uses: docker/login-action@v2
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v4
         with:
-          go-version: 1.17
+          go-version: '1.18'
       - name: Run GoReleaser
-        uses: goreleaser/goreleaser-action@v2
+        uses: goreleaser/goreleaser-action@v4
         with:
           distribution: goreleaser
           version: latest
           args: release --rm-dist
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          HOMEBREW_TAP_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }}

.github/workflows/secrets.yml

@@ -0,0 +1,31 @@
+
+name: Scan for secrets
+
+on:
+  push:
+    tags:
+      - v*
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.18'
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+        ref: ${{ github.head_ref }}
+    - name: Dogfood
+      uses: ./
+      id: dogfood
+      with:
+        path: ./
+        base: ${{ github.event.repository.default_branch }}
+        head: HEAD

.github/workflows/smoke.yml

@@ -0,0 +1,24 @@
+
+name: Smoke
+
+on:
+  pull_request:
+
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.18'
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+        ref: ${{ github.head_ref }}
+    - name: Smoke
+      run: |
+        set -e
+        go run . git https://github.com/dustin-decker/secretsandstuff.git
+        go run . github --repo https://github.com/dustin-decker/secretsandstuff.git

.github/workflows/snifftest.yml

@@ -6,7 +6,6 @@ on:
       - v*
     branches:
       - main
-  pull_request:
 
 permissions:
   contents: read
@@ -17,9 +16,9 @@ jobs:
     name: Run Snifftest
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/setup-go@v2
+      - uses: actions/setup-go@v4
         with:
-          go-version: '1.17'
+          go-version: '1.18'
       - uses: actions/checkout@v3
       - name: Run Snifftest
         run: make snifftest

.github/workflows/test.yml

@@ -11,46 +11,38 @@ on:
 
 jobs:
   test:
-    strategy:
-      matrix:
-        go-version: [1.17.x]
-        os: [ubuntu-latest]
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     permissions:
       contents: 'read'
       id-token: 'write'
     steps:
     - name: Install Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: '1.18'
     - name: Checkout code
       uses: actions/checkout@v3
     - id: 'auth'
-      uses: 'google-github-actions/auth@v0'
+      uses: 'google-github-actions/auth@v1'
       with:
         workload_identity_provider: 'projects/811013774421/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
         service_account: 'github-ci-external@trufflehog-testing.iam.gserviceaccount.com'
     - name: Test
-      run: make test
+      run: make test-integration
   test-detectors:
-    strategy:
-      matrix:
-        go-version: [1.17.x]
-        os: [ubuntu-latest]
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     permissions:
       contents: 'read'
       id-token: 'write'
     steps:
     - name: Install Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: '1.18'
     - name: Checkout code
       uses: actions/checkout@v3
     - id: 'auth'
-      uses: 'google-github-actions/auth@v0'
+      uses: 'google-github-actions/auth@v1'
       with:
         workload_identity_provider: 'projects/811013774421/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
         service_account: 'github-ci-external@trufflehog-testing.iam.gserviceaccount.com'

.gitignore

@@ -1,2 +1,7 @@
 .idea
 dist
+.env
+*.test
+
+# binary
+trufflehog

.goreleaser.yml

@@ -11,8 +11,39 @@ builds:
     - amd64
     - arm64
 dockers:
+  - image_templates: ["trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64"]
+    dockerfile: Dockerfile.goreleaser
+    extra_files:
+    - entrypoint.sh
+    use: buildx
+    build_flag_templates:
+    - --platform=linux/amd64
+    - --label=org.opencontainers.image.title={{ .ProjectName }}
+    - --label=org.opencontainers.image.description={{ .ProjectName }}
+    - --label=org.opencontainers.image.url=https://github.com/trufflesecurity/{{ .ProjectName }}
+    - --label=org.opencontainers.image.source=https://github.com/trufflesecurity/{{ .ProjectName }}
+    - --label=org.opencontainers.image.version={{ .Version }}
+    - --label=org.opencontainers.image.revision={{ .FullCommit }}
+    - --label=org.opencontainers.image.licenses=AGPL-3.0
+  - image_templates: ["trufflesecurity/{{ .ProjectName }}:{{ .Version }}-arm64v8"]
+    goarch: arm64
+    dockerfile: Dockerfile.goreleaser
+    extra_files:
+    - entrypoint.sh
+    use: buildx
+    build_flag_templates:
+    - --platform=linux/arm64/v8
+    - --label=org.opencontainers.image.title={{ .ProjectName }}
+    - --label=org.opencontainers.image.description={{ .ProjectName }}
+    - --label=org.opencontainers.image.url=https://github.com/trufflesecurity/{{ .ProjectName }}
+    - --label=org.opencontainers.image.source=https://github.com/trufflesecurity/{{ .ProjectName }}
+    - --label=org.opencontainers.image.version={{ .Version }}
+    - --label=org.opencontainers.image.revision={{ .FullCommit }}
+    - --label=org.opencontainers.image.licenses=AGPL-3.0
   - image_templates: ["ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64"]
     dockerfile: Dockerfile.goreleaser
+    extra_files:
+    - entrypoint.sh
     use: buildx
     build_flag_templates:
     - --platform=linux/amd64
@@ -26,6 +57,8 @@ dockers:
   - image_templates: ["ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}-arm64v8"]
     goarch: arm64
     dockerfile: Dockerfile.goreleaser
+    extra_files:
+    - entrypoint.sh
     use: buildx
     build_flag_templates:
     - --platform=linux/arm64/v8
@@ -37,6 +70,14 @@ dockers:
     - --label=org.opencontainers.image.revision={{ .FullCommit }}
     - --label=org.opencontainers.image.licenses=AGPL-3.0
 docker_manifests:
+  - name_template: trufflesecurity/{{ .ProjectName }}:{{ .Version }}
+    image_templates:
+    - trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64
+    - trufflesecurity/{{ .ProjectName }}:{{ .Version }}-arm64v8
+  - name_template: trufflesecurity/{{ .ProjectName }}:latest
+    image_templates:
+    - trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64
+    - trufflesecurity/{{ .ProjectName }}:{{ .Version }}-arm64v8
   - name_template: ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}
     image_templates:
     - ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64
@@ -45,3 +86,13 @@ docker_manifests:
     image_templates:
     - ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}-amd64
     - ghcr.io/trufflesecurity/{{ .ProjectName }}:{{ .Version }}-arm64v8
+brews:
+  - tap:
+      owner: trufflesecurity
+      name: homebrew-trufflehog
+      token: "{{ .Env.HOMEBREW_TAP_TOKEN }}"
+    description: "Find credentials all over the place"
+    name: "trufflehog"
+    homepage: "https://github.com/trufflesecurity/trufflehog"
+    install: |
+      bin.install "trufflehog"

.pre-commit-hooks.yaml

@@ -0,0 +1,6 @@
+- id: trufflehog
+  name: TruffleHog
+  description: Detect secrets in your data with TruffleHog.
+  entry: trufflehog git file://. --since-commit HEAD --only-verified --fail
+  language: golang
+  pass_filenames: false

CODEOWNERS

@@ -0,0 +1,23 @@
+# catch-all
+* @trufflesecurity/product-eng
+
+# teams
+pkg/sources/ @trufflesecurity/backend
+pkg/detectors/ @trufflesecurity/detection
+
+# critical detectors
+pkg/detectors/aws/ @trufflesecurity/backend
+pkg/detectors/gcp/ @trufflesecurity/backend
+pkg/detectors/azure/ @trufflesecurity/backend
+pkg/detectors/okta/ @trufflesecurity/backend
+pkg/detectors/privatekey/ @trufflesecurity/backend
+pkg/detectors/slack/ @trufflesecurity/backend
+pkg/detectors/slackwebhook/ @trufflesecurity/backend
+pkg/detectors/microsoftteamswebhook/ @trufflesecurity/backend
+pkg/detectors/twilio/ @trufflesecurity/backend
+pkg/detectors/sendgrid/ @trufflesecurity/backend
+pkg/detectors/gitlab/ @trufflesecurity/backend
+pkg/detectors/gitlabv2/ @trufflesecurity/backend
+pkg/detectors/github/ @trufflesecurity/backend
+pkg/detectors/github_old/ @trufflesecurity/backend
+pkg/detectors/githubapp/ @trufflesecurity/backend

CONTRIBUTING.md

@@ -1,7 +1,13 @@
 # Contribution guidelines
 
-Please create an issue to collect feedback prior to feature additions. If possible try to keep PR's scoped to one feature, and add tests for new features.
+Please create an issue to collect feedback prior to feature additions. If possible try to keep PRs scoped to one feature, and add tests for new features.
 
-When showing intereste, in a bug, enhancement, PR, or issue, please use the thumbs up/thumbs down emoji on the original message rather than adding comments expressing the same.
+When showing interest in a bug, enhancement, PR, or issue, please use the thumbs up/thumbs down emoji on the original message rather than adding comments expressing the same.
 
-Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/trufflehog) before we are able to accept contributions.
+Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/trufflehog) before we are able to accept contributions.
+
+# Resources
+
+## Adding new secret detectors
+
+We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!

Dockerfile

@@ -1,12 +1,18 @@
-FROM golang:bullseye as builder
-RUN mkdir /build
-COPY . /build
-WORKDIR /build
-RUN CGO_ENABLED=0 go build -a -o trufflehog main.go
-RUN mkdir /empty
+FROM --platform=${BUILDPLATFORM} golang:bullseye as builder
 
-FROM alpine
-RUN apk add git
-COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
+WORKDIR /build
+COPY . . 
+ENV CGO_ENABLED=0
+ARG TARGETOS TARGETARCH
+RUN  --mount=type=cache,target=/go/pkg/mod \
+     --mount=type=cache,target=/root/.cache/go-build \
+     GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o trufflehog .
+
+FROM alpine:3.15
+RUN apk add --no-cache bash git openssh-client ca-certificates \
+    && rm -rf /var/cache/apk/* && \
+    update-ca-certificates
 COPY --from=builder /build/trufflehog /usr/bin/trufflehog
-ENTRYPOINT ["/usr/bin/trufflehog"]
+COPY entrypoint.sh /etc/entrypoint.sh
+RUN chmod +x /etc/entrypoint.sh
+ENTRYPOINT ["/etc/entrypoint.sh"]

Dockerfile.goreleaser

@@ -1,4 +1,10 @@
-FROM alpine
-RUN apk add git
-COPY trufflehog /usr/bin/trufflehog
-ENTRYPOINT ["/usr/bin/trufflehog"]
+FROM alpine:3.15
+
+RUN apk add --no-cache bash git openssh-client ca-certificates \
+    && rm -rf /var/cache/apk/* && \
+    update-ca-certificates
+WORKDIR /usr/bin/
+COPY trufflehog .
+COPY entrypoint.sh /etc/entrypoint.sh
+RUN chmod +x /etc/entrypoint.sh
+ENTRYPOINT ["/etc/entrypoint.sh"]

Makefile

@@ -1,6 +1,7 @@
-PROTOS_IMAGE ?= us-docker.pkg.dev/thog-artifacts/public/go-ci-1.17-1
+PROTOS_IMAGE ?= trufflesecurity/protos:1.18-0
 
 .PHONY: check
+.PHONY: lint
 .PHONY: test
 .PHONY: test-race
 .PHONY: run
@@ -20,17 +21,23 @@ check:
 	go fmt $(shell go list ./... | grep -v /vendor/)
 	go vet $(shell go list ./... | grep -v /vendor/)
 
+lint:
+	golangci-lint run --enable bodyclose --out-format=colored-line-number --timeout 10m
+
 test-failing:
 	CGO_ENABLED=0 go test -timeout=5m $(shell go list ./... | grep -v /vendor/) | grep FAIL
 
 test:
-	CGO_ENABLED=0 go test -timeout=5m $(shell go list ./... | grep -v /vendor/ | grep -v /pkg/detectors)
+	CGO_ENABLED=0 go test -timeout=5m $(shell go list ./... | grep -v /vendor/ | grep -v pkg/detectors)
+
+test-integration:
+	CGO_ENABLED=0 go test -timeout=5m -tags=integration $(shell go list ./... | grep -v /vendor/ | grep -v pkg/detectors)
 
 test-race:
-	CGO_ENABLED=1 go test -timeout=5m -race $(shell go list ./... | grep -v /vendor/ | grep -v /pkg/detectors)
+	CGO_ENABLED=1 go test -timeout=5m -race $(shell go list ./... | grep -v /vendor/ | grep -v pkg/detectors)
 
 test-detectors:
-	CGO_ENABLED=0 go test -timeout=5m $(shell go list ./... | grep /pkg/detectors)
+	CGO_ENABLED=0 go test -tags=detectors -timeout=5m $(shell go list ./... | grep pkg/detectors)
 
 bench:
 	CGO_ENABLED=0 go test $(shell go list ./pkg/secrets/... | grep -v /vendor/) -benchmem -run=xxx -bench .
@@ -47,5 +54,12 @@ protos:
 protos-windows:
 	docker run -v "$(shell cygpath -w $(shell pwd))":/pwd "${PROTOS_IMAGE}" bash -c "cd /pwd; ./scripts/gen_proto.sh"
 
+release-protos-image:
+	docker buildx build --push --platform=linux/amd64,linux/arm64 \
+	-t trufflesecurity/protos:1.18-0 -f hack/Dockerfile.protos .
+
 snifftest:
 	./hack/snifftest/snifftest.sh
+
+test-release:
+	goreleaser release --rm-dist --skip-publish --snapshot

README.md

@@ -6,11 +6,14 @@
 
 ---
 
+<div align="center">
 
 [![CI Status](https://github.com/trufflesecurity/trufflehog/actions/workflows/release.yml/badge.svg)](https://github.com/trufflesecurity/trufflehog/actions)
-[![Go Report Card](https://goreportcard.com/badge/github.com/trufflesecurity/trufflehog)](https://goreportcard.com/report/github.com/trufflesecurity/trufflehog)
-[![Docker Hub Build Status](https://img.shields.io/docker/cloud/build/trufflesecurity/trufflehog.svg)](https://hub.docker.com/r/trufflesecurity/trufflehog/)
-![GitHub](https://img.shields.io/github/license/trufflesecurity/trufflehog)
+[![Go Report Card](https://goreportcard.com/badge/github.com/trufflesecurity/trufflehog/v3)](https://goreportcard.com/report/github.com/trufflesecurity/trufflehog/v3)
+[![License](https://img.shields.io/badge/license-AGPL--3.0-brightgreen)](/LICENSE)
+[![Total Detectors](https://img.shields.io/github/directory-file-count/trufflesecurity/truffleHog/pkg/detectors?label=Total%20Detectors&type=dir)](/pkg/detectors)
+
+</div>
 
 ---
 
@@ -24,45 +27,134 @@ https://join.slack.com/t/trufflehog-community/shared_invite/zt-pw2qbi43-Aa86hkii
 
 ![GitHub scanning demo](https://storage.googleapis.com/truffle-demos/non-interactive.svg)
 
+```bash
+docker run --rm -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --org=trufflesecurity
+```
+
+## Examples
+
+### Example 1: Scan a repo for only verified secrets
+
+Command:
+
+```bash
+trufflehog git https://github.com/trufflesecurity/test_keys --only-verified
+```
+
+Expected output:
+```
+🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷
+
+Found verified result 🐷🔑
+Detector Type: AWS
+Decoder Type: PLAIN
+Raw result: AKIAYVP4CIPPERUVIFXG
+Line: 4
+Commit: fbc14303ffbf8fb1c2c1914e8dda7d0121633aca
+File: keys
+Email: counter <counter@counters-MacBook-Air.local>
+Repository: https://github.com/trufflesecurity/test_keys
+Timestamp: 2022-06-16 10:17:40 -0700 PDT
+...
+```
+
+### Example 2: Scan a GitHub Org for only verified secrets
+
+```bash
+trufflehog github --org=trufflesecurity --only-verified
+```
+
+### Example 3: Scan a GitHub Repo for only verified keys and get JSON output
+
+Command:
+
+```bash
+trufflehog git https://github.com/trufflesecurity/test_keys --only-verified --json
+```
+
+Expected output:
+```
+{"SourceMetadata":{"Data":{"Git":{"commit":"fbc14303ffbf8fb1c2c1914e8dda7d0121633aca","file":"keys","email":"counter \u003ccounter@counters-MacBook-Air.local\u003e","repository":"https://github.com/trufflesecurity/test_keys","timestamp":"2022-06-16 10:17:40 -0700 PDT","line":4}}},"SourceID":0,"SourceType":16,"SourceName":"trufflehog - git","DetectorType":2,"DetectorName":"AWS","DecoderName":"PLAIN","Verified":true,"Raw":"AKIAYVP4CIPPERUVIFXG","Redacted":"AKIAYVP4CIPPERUVIFXG","ExtraData":{"account":"595918472158","arn":"arn:aws:iam::595918472158:user/canarytokens.com@@mirux23ppyky6hx3l6vclmhnj","user_id":"AIDAYVP4CIPPJ5M54LRCY"},"StructuredData":null}
+...
+```
+
+### Example 4: Scan an S3 bucket for verified keys
+
+```bash
+trufflehog s3 --bucket=<bucket name> --only-verified
+```
+
+### Example 5: Scan a Github Repo using SSH authentication in docker
+
+```bash
+docker run --rm -v "$HOME/.ssh:/root/.ssh:ro" trufflesecurity/trufflehog:latest git ssh://github.com/trufflesecurity/test_keys
+```
+
+### Example 6: Scan individual files or directories
+
+```bash
+trufflehog filesystem path/to/file1.txt path/to/file2.txt path/to/dir
+```
+
+### Example 7: Scan GCS buckets for verified secrets.
+
+```bash
+trufflehog gcs --project-id=<project-ID> --cloud-environment --only-verified
+```
+
+### FAQ
+
++ All I see is `🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷` and the program exits, what gives?
+  + That means no secrets were detected
++ Why is the scan is taking a long time when I scan a GitHub org
+  + Unauthenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
++ It says a private key was verified, what does that mean?
+  + Check out our Driftwood blog post to learn how to do this, in short we've confirmed the key can be used live for SSH or SSL [Blog post](https://trufflesecurity.com/blog/driftwood-know-if-private-keys-are-sensitive/)
+
 
 # What's new in v3?
 
 TruffleHog v3 is a complete rewrite in Go with many new powerful features.
 
-- We've **added over 600 credential detectors that support active verification against their respective APIs**.
-- We've also added native **support for scanning GitHub, GitLab, filesystems, and S3**.
+- We've **added over 700 credential detectors that support active verification against their respective APIs**.
+- We've also added native **support for scanning GitHub, GitLab, filesystems, S3, and Circle CI**.
+- **Instantly verify private keys** against millions of github users and **billions** of TLS certificates using our [Driftwood](https://trufflesecurity.com/blog/driftwood) technology.
 
 
 ## What is credential verification?
-For every potential credential that is detected, we've painstakingly implemented programatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
+For every potential credential that is detected, we've painstakingly implemented programmatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
 
 ## Installation
 
 Several options:
 
 ### 1. Go
-`go install github.com/trufflesecurity/trufflehog.git@latest`
+```bash
+git clone https://github.com/trufflesecurity/trufflehog.git
+
+cd trufflehog; go install
+```
 
 ### 2. [Release binaries](https://github.com/trufflesecurity/trufflehog/releases)
 
 ### 3. Docker
 
 
-> Note: Apple M1 hardware users should run with `docker run --platform linux/arm64` for better performance.
+> Note: Apple M1 hardware users should run with `docker run --rm --platform linux/arm64` for better performance.
 
 #### **Most users**
 
 ```bash
-docker run -it -v "$PWD:/pwd" ghcr.io/trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys --debug 
+docker run --rm -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys
 ```
 
 #### **Apple M1 users**
 
 The `linux/arm64` image is better to run on the M1 than the amd64 image.
-Even better is running the native darwin binary avilable, but there is not container image for that.
+Even better is running the native darwin binary available, but there is no container image for that.
 
 ```bash
-docker run --platform linux/arm64 -it -v "$PWD:/pwd" ghcr.io/trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys 
+docker run --rm --platform linux/arm64 -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys
 ```
 
 ### 4. Pip (help wanted)
@@ -73,9 +165,11 @@ Here is an example of a [project that does it](https://github.com/Yelp/dumb-init
 
 Help with setting up this packaging would be appreciated!
 
-### 5. Brew (help wanted)
+### 5. Brew
 
-We'd love to distribute via brew and could use your help.
+```bash
+brew install trufflesecurity/trufflehog/trufflehog
+```
 
 ## Usage
 
@@ -85,10 +179,13 @@ TruffleHog has a sub-command for each source of data that you may want to scan:
 - github
 - gitlab
 - S3
-- filesystem
-- file and stdin
+- filesystem (files and directories)
+- syslog
+- circleci
+- GCS (Google Cloud Storage)
+- stdin (coming soon)
 
-Each subcommand can have options that you can see with the `-h` flag provided to the sub command:
+Each subcommand can have options that you can see with the `--help` flag provided to the sub command:
 
 ```
 $ trufflehog git --help
@@ -97,22 +194,24 @@ usage: TruffleHog git [<flags>] <uri>
 Find credentials in git repositories.
 
 Flags:
-      --help           Show context-sensitive help (also try --help-long and --help-man).
-      --debug          Run in debug mode
-      --json           Output in JSON format.
-      --concurrency=8  Number of concurrent workers.
-      --verification   Verify the results.
-  -i, --include_paths=INCLUDE_PATHS  
-                       Path to file with newline separated regexes for files to include in scan.
-  -x, --exclude_paths=EXCLUDE_PATHS  
-                       Path to file with newline separated regexes for files to exclude in scan.
-      --branch=BRANCH  Branch to scan.
-      --allow          No-op flag for backwards compat.
-      --entropy        No-op flag for backwards compat.
-      --regex          No-op flag for backwards compat.
+      --help                     Show context-sensitive help (also try --help-long and --help-man).
+      --debug                    Run in debug mode.
+      --trace                    Run in trace mode.
+      --profile                  Enables profiling and sets a pprof and fgprof server on :18066.
+  -j, --json                     Output in JSON format.
+      --json-legacy              Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.
+      --concurrency=10           Number of concurrent workers.
+      --no-verification          Don't verify the results.
+      --only-verified            Only output verified results.
+      --filter-unverified        Only output first unverified result per chunk per detector if there are more than one results.
+      --config=CONFIG            Path to configuration file.
+      --print-avg-detector-time  Print the average time spent on each detector.
+      --no-update                Don't check for updates.
+      --fail                     Exit with code 183 if results are found.
+      --version                  Show application version.
 
 Args:
-  <uri>  Git repository URL. https:// or file:// schema expected.
+  <uri>  Git repository URL. https://, file://, or ssh:// schema expected.
 ```
 
 For example, to scan a  `git` repository, start with
@@ -121,6 +220,176 @@ For example, to scan a  `git` repository, start with
 $ trufflehog git https://github.com/trufflesecurity/trufflehog.git
 ```
 
+Exit Codes:
+- 0: No errors and no results were found.
+- 1: An error was encountered. Sources may not have completed scans.
+- 183: No errors were encountered, but results were found. Will only be returned if `--fail` flag is used.
+
+#### Scanning an organization
+
+Try scanning an entire GitHub organization with the following:
+
+```bash
+docker run --rm -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --org=trufflesecurity
+```
+
+### TruffleHog OSS Github Action
+
+```yaml
+- name: TruffleHog OSS
+  uses: trufflesecurity/trufflehog@main
+  with:
+    # Repository path
+    path:
+    # Start scanning from here (usually main branch).
+    base:
+    # Scan commits until here (usually dev branch).
+    head: # optional
+    # Extra args to be passed to the trufflehog cli.
+    extra_args: --debug --only-verified
+```
+
+The TruffleHog OSS Github Action can be used to scan a range of commits for leaked credentials. The action will fail if
+any results are found.
+
+For example, to scan the contents of pull requests you could use the following workflow:
+```yaml
+name: Leaked Secrets Scan
+on: [pull_request]
+jobs:
+  TruffleHog:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: TruffleHog OSS
+        uses: trufflesecurity/trufflehog@main
+        with:
+          path: ./
+          base: ${{ github.event.repository.default_branch }}
+          head: HEAD
+          extra_args: --debug --only-verified
+```
+
+### Precommit Hook
+
+Trufflehog can be used in a precommit hook to prevent credentials from leaking before they ever leave your computer.
+An example `.pre-commit-config.yaml` is provided (see [pre-commit.com](https://pre-commit.com/) for installation).
+
+```yaml
+repos:
+- repo: local
+  hooks:
+    - id: trufflehog
+      name: TruffleHog
+      description: Detect secrets in your data.
+      entry: bash -c 'trufflehog git file://. --since-commit HEAD --only-verified --fail'
+      # For running trufflehog in docker, use the following entry instead:
+      # entry: bash -c 'docker run --rm -v "$(pwd):/workdir" -i --rm trufflesecurity/trufflehog:latest git file:///workdir --since-commit HEAD --only-verified --fail'
+      language: system
+      stages: ["commit", "push"]
+```
+
+## Regex Detector (alpha)
+
+Trufflehog supports detection and verification of custom regular expressions.
+For detection, at least one **regular expression** and **keyword** is required.
+A **keyword** is a fixed literal string identifier that appears in or around
+the regex to be detected. To allow maximum flexibility for verification, a
+webhook is used containing the regular expression matches.
+
+Trufflehog will send a JSON POST request containing the regex matches to a
+configured webhook endpoint. If the endpoint responds with a `200 OK` response
+status code, the secret is considered verified.
+
+**NB:** This feature is alpha and subject to change.
+
+### Regex Detector Example
+
+```yaml
+# config.yaml
+detectors:
+- name: hog detector
+  keywords:
+  - hog
+  regex:
+    adjective: hogs are (\S+)
+  verify:
+  - endpoint: http://localhost:8000/
+    # unsafe must be set if the endpoint is HTTP
+    unsafe: true
+    headers:
+    - 'Authorization: super secret authorization header'
+```
+
+```
+$ trufflehog filesystem /tmp --config config.yaml --only-verified
+🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷
+
+Found verified result 🐷🔑
+Detector Type: CustomRegex
+Decoder Type: PLAIN
+Raw result: hogs are cool
+File: /tmp/hog-facts.txt
+```
+
+#### Verification Server Example (Python)
+
+Unless you run a verification server, secrets found by the custom regex
+detector will be unverified. Here is an example Python implementation of a
+verification server for the above `config.yaml` file.
+
+```python
+import json
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+AUTH_HEADER = 'super secret authorization header'
+
+
+class Verifier(BaseHTTPRequestHandler):
+    def do_GET(self):
+        self.send_response(405)
+        self.end_headers()
+
+    def do_POST(self):
+        try:
+            if self.headers['Authorization'] != AUTH_HEADER:
+                self.send_response(401)
+                self.end_headers()
+                return
+
+            # read the body
+            length = int(self.headers['Content-Length'])
+            request = json.loads(self.rfile.read(length))
+            self.log_message("%s", request)
+
+            # check the match
+            if request['hog detector']['adjective'][-1] == 'cool':
+                self.send_response(200)
+                self.end_headers()
+            else:
+                # any other response besides 200
+                self.send_response(406)
+                self.end_headers()
+        except Exception:
+            self.send_response(400)
+            self.end_headers()
+
+
+with HTTPServer(('', 8000), Verifier) as server:
+    try:
+        server.serve_forever()
+    except KeyboardInterrupt:
+        pass
+```
+
+## Use as a library
+
+Currently, trufflehog is in heavy development and no guarantees can be made on
+the stability of the public APIs at this time.
+
 ## Contributors
 
 This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
@@ -137,6 +406,17 @@ Contributions are very welcome! Please see our [contribution guidelines first](C
 
 We no longer accept contributions to TruffleHog v2, but that code is available in the `v2` branch.
 
+### Adding new secret detectors
+
+We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
+
 ## License Change
 
 Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.
+
+
+## Enterprise product
+
+Are you interested in continously monitoring your Git, Jira, Slack, Confluence, etc.. for credentials? We have an enterprise product that can help. Reach out here to learn more https://trufflesecurity.com/contact/
+
+We take the revenue from the enterprise product to fund more awesome open source projects that the whole community can benefit from.

action.yml

@@ -0,0 +1,39 @@
+name: 'TruffleHog OSS'
+description: 'Scan Github Actions with TruffleHog'
+author: Truffle Security Co. <support@trufflesec.com>
+
+inputs:
+  path:
+    description: Repository path
+    required: true
+  base:
+    description: Start scanning from here (usually main branch).
+    required: false
+    default: ''
+  head:
+    description: Scan commits until here (usually dev branch).
+    required: false
+  extra_args:
+    default: ''
+    description: Extra args to be passed to the trufflehog cli.
+    required: false
+outputs:
+  results:
+    description: 'Trufflehog scan results'
+branding:
+  icon: "shield"
+  color: "green"
+runs:
+  using: "docker"
+  image: "docker://ghcr.io/trufflesecurity/trufflehog:latest"
+  args:
+    - git
+    - file://${{ inputs.path }}
+    - --since-commit
+    - ${{ inputs.base }}
+    - --branch
+    - ${{ inputs.head }}
+    - --fail
+    - --no-update
+    - --github-actions
+    - ${{ inputs.extra_args }}

entrypoint.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+# Parse the last argument into an array of extra_args.
+mapfile -t extra_args < <(bash -c "for arg in ${*: -1}; do echo \$arg; done")
+
+results=$(/usr/bin/trufflehog "${@: 1: $#-1}" "${extra_args[@]}" | tee /dev/tty)
+echo "results=$results" >> $GITHUB_OUTPUT

go.mod

@@ -1,58 +1,78 @@
 module github.com/trufflesecurity/trufflehog/v3
 
-go 1.17
+go 1.18
 
-replace github.com/gitleaks/go-gitdiff => github.com/bill-rich/go-gitdiff v0.7.6-custom1
+replace github.com/jpillora/overseer => github.com/trufflesecurity/overseer v1.1.7-custom5
 
-replace github.com/jpillora/overseer => github.com/dustin-decker/overseer v1.1.7-custom2
+replace github.com/zricethezav/gitleaks/v8 => github.com/trufflesecurity/gitleaks/v8 v8.6.1-custom10
+
+replace github.com/gitleaks/go-gitdiff => github.com/trufflesecurity/go-gitdiff v0.7.6-zombies2
 
 require (
-	cloud.google.com/go/secretmanager v1.3.0
+	cloud.google.com/go/secretmanager v1.10.0
+	cloud.google.com/go/storage v1.30.1
 	github.com/Azure/go-autorest/autorest/azure/auth v0.5.11
-	github.com/aliyun/alibaba-cloud-sdk-go v1.61.1465
-	github.com/aws/aws-sdk-go v1.43.27
-	github.com/aws/aws-sdk-go-v2/credentials v1.11.1
-	github.com/aws/aws-sdk-go-v2/service/sts v1.16.2
+	github.com/TheZeroSlave/zapsentry v1.14.0
+	github.com/aws/aws-sdk-go v1.44.83
+	github.com/bill-rich/disk-buffer-reader v0.1.7
+	github.com/bill-rich/go-syslog v0.0.0-20220413021637-49edb52a574c
 	github.com/bitfinexcom/bitfinex-api-go v0.0.0-20210608095005-9e0b26f200fb
-	github.com/bradleyfalzon/ghinstallation/v2 v2.0.4
-	github.com/denisbrodbeck/machineid v1.0.1
-	github.com/envoyproxy/protoc-gen-validate v0.6.7
-	github.com/fatih/color v1.13.0
-	github.com/felixge/fgprof v0.9.2
-	github.com/gitleaks/go-gitdiff v0.7.5
+	github.com/bradleyfalzon/ghinstallation/v2 v2.2.0
+	github.com/crewjam/rfc5424 v0.1.0
+	github.com/denisenkom/go-mssqldb v0.12.3
+	github.com/envoyproxy/protoc-gen-validate v0.9.1
+	github.com/fatih/color v1.15.0
+	github.com/felixge/fgprof v0.9.3
+	github.com/getsentry/sentry-go v0.19.0
 	github.com/go-errors/errors v1.4.2
-	github.com/go-git/go-git/v5 v5.4.2
+	github.com/go-git/go-git/v5 v5.6.1
+	github.com/go-ldap/ldap/v3 v3.4.4
+	github.com/go-logr/logr v1.2.3
+	github.com/go-logr/zapr v1.2.3
+	github.com/go-redis/redis v6.15.9+incompatible
+	github.com/go-sql-driver/mysql v1.7.0
+	github.com/gobwas/glob v0.2.3
 	github.com/golang-jwt/jwt v3.2.2+incompatible
+	github.com/google/go-cmp v0.5.9
 	github.com/google/go-github/v42 v42.0.0
+	github.com/googleapis/gax-go/v2 v2.8.0
 	github.com/gorilla/mux v1.8.0
-	github.com/h2non/filetype v1.1.3
-	github.com/hashicorp/go-retryablehttp v0.7.0
-	github.com/joho/godotenv v1.4.0
+	github.com/hashicorp/go-retryablehttp v0.7.2
+	github.com/jlaffaye/ftp v0.1.0
+	github.com/joho/godotenv v1.5.1
 	github.com/jpillora/overseer v1.1.6
 	github.com/kylelemons/godebug v1.1.0
-	github.com/mattn/go-colorable v0.1.12
+	github.com/lib/pq v1.10.7
+	github.com/mattn/go-sqlite3 v1.14.16
+	github.com/mholt/archiver/v4 v4.0.0-alpha.7
+	github.com/patrickmn/go-cache v2.1.0+incompatible
 	github.com/paulbellamy/ratecounter v0.2.0
+	github.com/petar-dambovaliev/aho-corasick v0.0.0-20211021192214-5ab2d9280aa9
 	github.com/pkg/errors v0.9.1
-	github.com/razorpay/razorpay-go v0.0.0-20210728161131-0341409a6ab2
-	github.com/rs/zerolog v1.26.1
-	github.com/sergi/go-diff v1.2.0
-	github.com/sirupsen/logrus v1.8.1
+	github.com/rabbitmq/amqp091-go v1.8.0
+	github.com/sergi/go-diff v1.3.1
+	github.com/stretchr/testify v1.8.2
 	github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502
-	github.com/xanzy/go-gitlab v0.60.0
-	github.com/zricethezav/gitleaks/v8 v8.5.2
-	golang.org/x/crypto v0.0.0-20211215165025-cf75a172585e
-	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
-	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
-	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
-	google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf
-	google.golang.org/protobuf v1.28.0
+	github.com/xanzy/go-gitlab v0.81.0
+	go.mongodb.org/mongo-driver v1.11.3
+	go.uber.org/zap v1.24.0
+	golang.org/x/crypto v0.7.0
+	golang.org/x/exp v0.0.0-20221018205818-5c77f4b2bbd7
+	golang.org/x/oauth2 v0.6.0
+	golang.org/x/sync v0.1.0
+	golang.org/x/text v0.8.0
+	google.golang.org/api v0.114.0
+	google.golang.org/protobuf v1.30.0
 	gopkg.in/alecthomas/kingpin.v2 v2.2.6
+	gopkg.in/h2non/gock.v1 v1.1.2
+	sigs.k8s.io/yaml v1.3.0
 )
 
 require (
-	cloud.google.com/go v0.100.2 // indirect
-	cloud.google.com/go/compute v1.3.0 // indirect
-	cloud.google.com/go/iam v0.1.0 // indirect
+	cloud.google.com/go v0.110.0 // indirect
+	cloud.google.com/go/compute v1.18.0 // indirect
+	cloud.google.com/go/compute/metadata v0.2.3 // indirect
+	cloud.google.com/go/iam v0.12.0 // indirect
 	github.com/Azure/go-autorest v14.2.0+incompatible // indirect
 	github.com/Azure/go-autorest/autorest v0.11.24 // indirect
 	github.com/Azure/go-autorest/autorest/adal v0.9.18 // indirect
@@ -60,53 +80,78 @@ require (
 	github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
 	github.com/Azure/go-autorest/logger v0.2.1 // indirect
 	github.com/Azure/go-autorest/tracing v0.6.0 // indirect
-	github.com/Microsoft/go-winio v0.4.16 // indirect
-	github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7 // indirect
-	github.com/acomagu/bufpipe v1.0.3 // indirect
+	github.com/Azure/go-ntlmssp v0.0.0-20220621081337-cb9428e4ac1e // indirect
+	github.com/Microsoft/go-winio v0.5.2 // indirect
+	github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 // indirect
+	github.com/acomagu/bufpipe v1.0.4 // indirect
 	github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
 	github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.16.1 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.8 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.2 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.2 // indirect
-	github.com/aws/smithy-go v1.11.2 // indirect
+	github.com/andybalholm/brotli v1.0.4 // indirect
+	github.com/benbjohnson/clock v1.1.0 // indirect
+	github.com/cloudflare/circl v1.1.0 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dimchansky/utfbom v1.1.1 // indirect
-	github.com/emirpasic/gods v1.12.0 // indirect
+	github.com/dsnet/compress v0.0.1 // indirect
+	github.com/emirpasic/gods v1.18.1 // indirect
+	github.com/go-asn1-ber/asn1-ber v1.5.4 // indirect
 	github.com/go-git/gcfg v1.5.0 // indirect
-	github.com/go-git/go-billy/v5 v5.3.1 // indirect
+	github.com/go-git/go-billy/v5 v5.4.1 // indirect
 	github.com/go-ole/go-ole v1.2.6 // indirect
-	github.com/golang-jwt/jwt/v4 v4.2.0 // indirect
+	github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
+	github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe // indirect
+	github.com/golang-sql/sqlexp v0.1.0 // indirect
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
-	github.com/google/go-cmp v0.5.7 // indirect
-	github.com/google/go-github/v41 v41.0.0 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
+	github.com/google/go-github/v50 v50.1.0 // indirect
 	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/google/pprof v0.0.0-20211214055906-6f57359322fd // indirect
-	github.com/googleapis/gax-go/v2 v2.1.1 // indirect
-	github.com/hashicorp/go-cleanhttp v0.5.1 // indirect
-	github.com/imdario/mergo v0.3.12 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
+	github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 // indirect
+	github.com/hashicorp/errwrap v1.0.0 // indirect
+	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
+	github.com/hashicorp/go-multierror v1.1.1 // indirect
+	github.com/imdario/mergo v0.3.13 // indirect
 	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/jpillora/s3 v1.1.4 // indirect
-	github.com/json-iterator/go v1.1.11 // indirect
-	github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 // indirect
-	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/kevinburke/ssh_config v1.2.0 // indirect
+	github.com/klauspost/compress v1.15.11 // indirect
+	github.com/klauspost/pgzip v1.2.5 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.17 // indirect
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
-	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe // indirect
+	github.com/nwaples/rardecode/v2 v2.0.0-beta.2 // indirect
+	github.com/onsi/ginkgo v1.16.5 // indirect
+	github.com/onsi/gomega v1.23.0 // indirect
+	github.com/pierrec/lz4/v4 v4.1.14 // indirect
+	github.com/pjbgf/sha1cd v0.3.0 // indirect
 	github.com/pkg/diff v0.0.0-20200914180035-5b29258ca4f7 // indirect
-	github.com/xanzy/ssh-agent v0.3.0 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/skeema/knownhosts v1.1.0 // indirect
+	github.com/therootcompany/xz v1.0.1 // indirect
+	github.com/ulikunitz/xz v0.5.10 // indirect
+	github.com/xanzy/ssh-agent v0.3.3 // indirect
+	github.com/xdg-go/pbkdf2 v1.0.0 // indirect
+	github.com/xdg-go/scram v1.1.1 // indirect
+	github.com/xdg-go/stringprep v1.0.3 // indirect
+	github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
 	github.com/yusufpapurcu/wmi v1.2.2 // indirect
-	go.opencensus.io v0.23.0 // indirect
-	golang.org/x/mod v0.5.0 // indirect
-	golang.org/x/sys v0.0.0-20220209214540-3681064d5158 // indirect
-	golang.org/x/text v0.3.7 // indirect
-	golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect
-	golang.org/x/tools v0.1.7 // indirect
-	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
-	google.golang.org/api v0.70.0 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	go.uber.org/atomic v1.7.0 // indirect
+	go.uber.org/multierr v1.6.0 // indirect
+	golang.org/x/mod v0.8.0 // indirect
+	golang.org/x/net v0.8.0 // indirect
+	golang.org/x/sys v0.6.0 // indirect
+	golang.org/x/time v0.3.0 // indirect
+	golang.org/x/tools v0.6.0 // indirect
+	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/grpc v1.44.0 // indirect
-	gopkg.in/ini.v1 v1.66.2 // indirect
+	google.golang.org/genproto v0.0.0-20230320184635-7606e756e683 // indirect
+	google.golang.org/grpc v1.53.0 // indirect
 	gopkg.in/warnings.v0 v0.1.2 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

hack/Dockerfile.protos

@@ -0,0 +1,23 @@
+# trufflesecurity/protos:1.18-0
+
+FROM golang:1.18-buster
+
+ARG TARGETARCH
+ARG TARGETOS
+ENV PROTOC_VER=3.20.0
+ENV GORELEASER_VER=v1.7.0
+ENV SQLBOILER_VER=v4.9.0
+ENV PROTOC_GEN_GO_VER=v1.5.2
+
+RUN echo "building $TARGETARCH"
+RUN apt-get update; apt-get install -y --no-install-recommends git unzip && rm -rf /var/lib/apt/lists/*
+RUN set -e; \
+	arch=$(echo $TARGETARCH | sed -e s/amd64/x86_64/ -e s/arm64/aarch_64/); \
+	wget -q https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VER}/protoc-${PROTOC_VER}-${TARGETOS}-${arch}.zip && unzip protoc-${PROTOC_VER}-${TARGETOS}-${arch}.zip -d /usr/local
+RUN go install "github.com/golang/protobuf/protoc-gen-go@${PROTOC_GEN_GO_VER}"
+RUN GO111MODULE=off go get -d "github.com/envoyproxy/protoc-gen-validate" && \
+    cd $GOPATH/src/github.com/envoyproxy/protoc-gen-validate && \
+    git checkout v0.6.7 && \
+    ln -s /usr/local/protoc/include/google google && \
+    make build
+CMD ["bash"]

hack/docs/Adding_Detectors_Internal.md

@@ -41,7 +41,7 @@ If you think that something should be included outside of these guidelines, plea
 - A GitLab account
 - A Google account
 - [Google Cloud SDK installed](#setting-up-google-cloud-sdk)
-- Go 1.16
+- Go 1.17+
 - Make
 
 ### Creating a new Secret Scanner
@@ -100,7 +100,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
    gcloud secrets versions add --project trufflehog-testing detectors3 --data-file /tmp/s
    ```
 
-4. Access the secret value as shown in the [example code](pkg/secrets/heroku/heroku_test.go).
+4. Access the secret value as shown in the [example code](pkg/detectors/heroku/heroku_test.go).
 
 ### Setting up Google Cloud SDK
 
@@ -123,3 +123,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
    ```bash
    make protos
    ```
+### Testing a detector
+```bash
+   go test ./pkg/detectors/<detector> -tags=detectors
+   ```

hack/docs/Adding_Detectors_external.md

@@ -0,0 +1,111 @@
+# Secret Detectors
+
+Secret Detectors have these two major functions:
+
+1. Given some bytes, extract possible secrets, typically using a regex.
+2. Validate the secrets against the target API, typically using a HTTP client.
+
+The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
+
+## Table of Contents
+
+- [Secret Detectors](#secret-detectors)
+  * [Table of Contents](#table-of-contents)
+  * [Getting Started](#getting-started)
+    + [Sourcing Guidelines](#sourcing-guidelines)
+    + [Development Guidelines](#development-guidelines)
+    + [Development Dependencies](#development-dependencies)
+    + [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
+  * [Addendum](#addendum)
+    + [Using a test secret file](#using-a-test-secret-file)
+    + [Adding Protos in Windows](#adding-protos-in-windows)
+
+## Getting Started
+
+### Sourcing Guidelines
+
+We are interested in detectors for services that meet at least one of these criteria
+- host data (they store any sort of data provided)
+- have paid services (having a free or trial tier is okay though)
+
+If you think that something should be included outside of these guidelines, please let us know.
+
+### Development Guidelines
+
+- When reasonable, favor using the `net/http` library to make requests instead of bringing in another library.
+- Use the [`common.SaneHttpClient`](/pkg/common/http.go) for the `http.Client` whenever possible.
+
+### Development Dependencies
+
+- Go 1.17+
+- Make
+
+### Creating a new Secret Scanner
+
+1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum. If necessary, run `make protos` when adding new ones.
+
+2. Generate the Secret Detector
+
+   ```bash
+   go run hack/generate/generate.go detector <DetectorType enum name>
+   ```
+3. Add Secret Scanner
+
+   Add the secret scanner to the `pkg/engine/defaults.go` file like `github.com/trufflesecurity/trufflehog/v3/pkg/detectors/<detector_name>` and 
+   `<detector_name>.Scanner{},`
+
+4. Complete the secret detector.
+
+   The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
+   The secret detector can be completed with these general steps:
+
+   1. Create a [test secrets file, and export the variable](#using-a-test-secret-file)
+   2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
+   3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
+   4. Update the tests with these test cases at minimum:
+      1. Found and verified (using a credential loaded from GCP Secrets)
+      2. Found and unverified
+      3. Not found
+      4. Any false positive cases that you come across
+   5. Create a pull request for review.
+
+## Addendum
+
+### Using a test secret file
+
+1. Create a file called `.env` with this env file format:
+
+   ```bash
+   SECRET_TYPE_ONE=value
+   SECRET_TYPE_ONE_INACTIVE=v@lue
+   ```
+
+2. Export the `TEST_SECRET_FILE` variable, pointing to the env file:
+
+   ```bash
+   export TEST_SECRET_FILE=".env"
+   ```
+
+Now, the detector test should attempt to load the given env key from that file.
+
+### Adding Protos in Windows
+
+1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
+2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
+3. Open Ubuntu cli and install `dos2unix`.
+   ```bash
+   sudo apt install dos2unix
+   ```
+4. Identify the `trufflehog` local directory and convert `scripts/gen_proto.sh` file in Unix format.
+   ```bash
+   dos2unix ./scripts/gen_proto.sh
+   ```
+5. Open [/proto/detectors.proto](/proto/detectors.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
+   ```bash
+   make protos
+   ```
+
+### Testing a detector
+```bash
+   go test ./pkg/detectors/<detector> -tags=detectors
+   ```

hack/generate/generate.go

@@ -1,7 +1,7 @@
 package main
 
 import (
-	"io/ioutil"
+	"fmt"
 	"log"
 	"os"
 	"path/filepath"
@@ -9,6 +9,8 @@ import (
 	"text/template"
 
 	"github.com/go-errors/errors"
+	"golang.org/x/text/cases"
+	"golang.org/x/text/language"
 	"gopkg.in/alecthomas/kingpin.v2"
 )
 
@@ -24,7 +26,7 @@ func main() {
 	log.SetPrefix("😲 [generate] ")
 
 	kingpin.MustParse(app.Parse(os.Args[1:]))
-	nameTitle = strings.Title(*name)
+	nameTitle = cases.Title(language.AmericanEnglish).String(*name)
 	nameLower = strings.ToLower(*name)
 	nameUpper = strings.ToUpper(*name)
 
@@ -65,7 +67,7 @@ type templateJob struct {
 }
 
 func mustWriteTemplates(jobs []templateJob) {
-	log.Printf("Generating %s %s\n", strings.Title(*kind), nameTitle)
+	log.Printf("Generating %s %s\n", cases.Title(language.AmericanEnglish).String(*kind), nameTitle)
 
 	// Make the folder.
 	log.Printf("Creating folder %s\n", folderPath())
@@ -76,7 +78,7 @@ func mustWriteTemplates(jobs []templateJob) {
 
 	// Write the files from templates.
 	for _, job := range jobs {
-		tmplBytes, err := ioutil.ReadFile(job.TemplatePath)
+		tmplBytes, err := os.ReadFile(job.TemplatePath)
 		if err != nil {
 			log.Fatal(err)
 		}
@@ -84,7 +86,7 @@ func mustWriteTemplates(jobs []templateJob) {
 
 		for _, rplString := range job.ReplaceString {
 			tmplRaw = strings.ReplaceAll(tmplRaw, strings.ToLower(rplString), "<<.NameLower>>")
-			tmplRaw = strings.ReplaceAll(tmplRaw, strings.Title(rplString), "<<.NameTitle>>")
+			tmplRaw = strings.ReplaceAll(tmplRaw, cases.Title(language.AmericanEnglish).String(rplString), "<<.NameTitle>>")
 			tmplRaw = strings.ReplaceAll(tmplRaw, strings.ToUpper(rplString), "<<.NameUpper>>")
 		}
 
@@ -95,11 +97,14 @@ func mustWriteTemplates(jobs []templateJob) {
 		if err != nil {
 			log.Fatal(err)
 		}
-		tmpl.Execute(f, templateData{
+		err = tmpl.Execute(f, templateData{
 			NameTitle: nameTitle,
 			NameLower: nameLower,
 			NameUpper: nameUpper,
 		})
+		if err != nil {
+			log.Fatal(fmt.Errorf("failed to execute template: %w", err))
+		}
 	}
 }
 

hack/snifftest/main.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"context"
 	"fmt"
 	"os"
 	"reflect"
@@ -12,13 +11,14 @@ import (
 	"time"
 
 	"github.com/paulbellamy/ratecounter"
-	log "github.com/sirupsen/logrus"
 	"golang.org/x/sync/semaphore"
 	"gopkg.in/alecthomas/kingpin.v2"
 
+	"github.com/trufflesecurity/trufflehog/v3/pkg/context"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/decoders"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/engine"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/log"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/source_metadatapb"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/sourcespb"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
@@ -42,6 +42,20 @@ var (
 )
 
 func main() {
+	// setup logger
+	logger, flush := log.New("trufflehog", log.WithConsoleSink(os.Stderr))
+	// make it the default logger for contexts
+	context.SetDefaultLogger(logger)
+	defer func() { _ = flush() }()
+	logFatal := func(err error, message string, keyAndVals ...any) {
+		logger.Error(err, message, keyAndVals...)
+		if err != nil {
+			os.Exit(1)
+			return
+		}
+		os.Exit(0)
+	}
+
 	ctx, cancel := context.WithTimeout(context.Background(), time.Hour*2)
 	var cancelOnce sync.Once
 	defer cancelOnce.Do(cancel)
@@ -68,19 +82,19 @@ func main() {
 		} else {
 			_, ok := allScanners[input]
 			if !ok {
-				log.Fatal("could not find scanner by that name")
+				logFatal(fmt.Errorf("invalid input"), "could not find scanner by that name")
 			}
 			selectedScanners[input] = allScanners[input]
 		}
 		if len(selectedScanners) == 0 {
-			log.Fatal("no detectors selected")
+			logFatal(fmt.Errorf("invalid input"), "no detectors selected")
 		}
 
 		for _, excluded := range *scanCmdExclude {
 			delete(selectedScanners, excluded)
 		}
 
-		log.Infof("loaded %d secret detectors", len(selectedScanners)+3)
+		logger.Info("loaded secret detectors", "count", len(selectedScanners)+3)
 
 		var wgScanners sync.WaitGroup
 
@@ -92,7 +106,7 @@ func main() {
 				time.Sleep(60 * time.Second)
 				counter.Incr(int64(chunkCounter - prev))
 				prev = chunkCounter
-				log.Infof("chunk scan rate: %d/sec", counter.Rate()/60)
+				logger.Info("chunk scan rate per second", "rate", counter.Rate()/60)
 			}
 		}()
 
@@ -122,7 +136,7 @@ func main() {
 
 								res, err := scanner.FromData(ctx, *scanVerify, decoded.Data)
 								if err != nil {
-									log.Fatal(err)
+									logFatal(err, "error scanning chunk")
 								}
 								if len(res) > 0 {
 									if resCounter[name] == nil {
@@ -131,16 +145,19 @@ func main() {
 									}
 									atomic.AddUint64(resCounter[name], uint64(len(res)))
 									if *scanThreshold != 0 && int(*resCounter[name]) > *scanThreshold {
-										log.WithField("scanner", name).Errorf("exceeded result threshold of %d", *scanThreshold)
+										logger.Error(
+											fmt.Errorf("exceeded result threshold"), "snifftest failed",
+											"scanner", name, "threshold", *scanThreshold,
+										)
 										failed = true
 										os.Exit(1)
 									}
 
 									if *scanPrintRes {
 										for _, r := range res {
-											logger := log.WithField("secret", name).WithField("meta", chunk.SourceMetadata).WithField("result", string(r.Raw))
+											logger := logger.WithValues("secret", name, "meta", chunk.SourceMetadata, "result", string(r.Raw))
 											if *scanPrintChunkRes {
-												logger = logger.WithField("chunk", string(decoded.Data))
+												logger = logger.WithValues("chunk", string(decoded.Data))
 											}
 											logger.Info("result")
 										}
@@ -156,18 +173,20 @@ func main() {
 		}
 
 		for _, repo := range strings.Split(*scanCmdRepo, ",") {
-			sem.Acquire(ctx, 1)
+			if err := sem.Acquire(ctx, 1); err != nil {
+				logFatal(err, "timed out waiting for semaphore")
+			}
 			wgChunkers.Add(1)
 			go func(r string) {
 				defer sem.Release(1)
 				defer wgChunkers.Done()
-				log.Infof("cloning %s", r)
-				path, repo, err := git.CloneRepoUsingUnauthenticated(r)
+				logger.Info("cloning repo", "repo", r)
+				path, repo, err := git.CloneRepoUsingUnauthenticated(ctx, r)
 				if err != nil {
-					log.Fatal(err)
+					logFatal(err, "error cloning repo", "repo", r)
 				}
 
-				log.Infof("cloned %s", r)
+				logger.Info("cloned repo", "repo", r)
 
 				s := git.NewGit(sourcespb.SourceType_SOURCE_TYPE_GIT, 0, 0, "snifftest", false, runtime.NumCPU(),
 					func(file, email, commit, timestamp, repository string, line int64) *source_metadatapb.MetaData {
@@ -184,12 +203,12 @@ func main() {
 						}
 					})
 
-				log.Infof("scanning %s", r)
+				logger.Info("scanning repo", "repo", r)
 				err = s.ScanRepo(ctx, repo, path, git.NewScanOptions(), chunksChan)
 				if err != nil {
-					log.Fatal(err)
+					logFatal(err, "error scanning repo")
 				}
-				log.Infof("scanned %s", r)
+				logger.Info("scanned repo", "repo", r)
 				defer os.RemoveAll(path)
 			}(repo)
 		}
@@ -201,9 +220,9 @@ func main() {
 
 		wgScanners.Wait()
 
-		log.WithField("chunks", chunkCounter).Info("completed")
+		logger.Info("completed snifftest", "chunks", chunkCounter)
 		for scanner, resultsCount := range resCounter {
-			log.WithField("results", *resultsCount).Info(scanner)
+			logger.Info(scanner, "results", *resultsCount)
 		}
 
 		if failed {

main.go

@@ -1,10 +1,7 @@
 package main
 
 import (
-	"context"
-	"encoding/json"
 	"fmt"
-	"log"
 	"net/http"
 	_ "net/http/pprof"
 	"os"
@@ -14,40 +11,57 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/felixge/fgprof"
+	"github.com/go-logr/logr"
+	"github.com/gorilla/mux"
 	"github.com/jpillora/overseer"
-	"github.com/trufflesecurity/trufflehog/v3/pkg/updater"
-	"github.com/trufflesecurity/trufflehog/v3/pkg/version"
 	"gopkg.in/alecthomas/kingpin.v2"
 
-	"github.com/felixge/fgprof"
-
-	"github.com/gorilla/mux"
-	"github.com/sirupsen/logrus"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/config"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/context"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/decoders"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/engine"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/handlers"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/log"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/output"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/sources/git"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/updater"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/version"
 )
 
 var (
-	cli            = kingpin.New("TruffleHog", "TruffleHog is a tool for finding credentials.")
-	cmd            string
-	debug          = cli.Flag("debug", "Run in debug mode").Bool()
-	versionFlag    = cli.Flag("version", "Prints trufflehog version.").Bool()
-	jsonOut        = cli.Flag("json", "Output in JSON format.").Short('j').Bool()
-	jsonLegacy     = cli.Flag("json-legacy", "Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.").Bool()
-	concurrency    = cli.Flag("concurrency", "Number of concurrent workers.").Default(strconv.Itoa(runtime.NumCPU())).Int()
-	noVerification = cli.Flag("no-verification", "Don't verify the results.").Bool()
-	onlyVerified   = cli.Flag("only-verified", "Only output verified results.").Bool()
+	cli                 = kingpin.New("TruffleHog", "TruffleHog is a tool for finding credentials.")
+	cmd                 string
+	debug               = cli.Flag("debug", "Run in debug mode.").Bool()
+	trace               = cli.Flag("trace", "Run in trace mode.").Bool()
+	profile             = cli.Flag("profile", "Enables profiling and sets a pprof and fgprof server on :18066.").Bool()
+	jsonOut             = cli.Flag("json", "Output in JSON format.").Short('j').Bool()
+	jsonLegacy          = cli.Flag("json-legacy", "Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.").Bool()
+	gitHubActionsFormat = cli.Flag("github-actions", "Output in GitHub Actions format.").Bool()
+	concurrency         = cli.Flag("concurrency", "Number of concurrent workers.").Default(strconv.Itoa(runtime.NumCPU())).Int()
+	noVerification      = cli.Flag("no-verification", "Don't verify the results.").Bool()
+	onlyVerified        = cli.Flag("only-verified", "Only output verified results.").Bool()
+	filterUnverified    = cli.Flag("filter-unverified", "Only output first unverified result per chunk per detector if there are more than one results.").Bool()
+	configFilename      = cli.Flag("config", "Path to configuration file.").ExistingFile()
 	// rules = cli.Flag("rules", "Path to file with custom rules.").String()
 	printAvgDetectorTime = cli.Flag("print-avg-detector-time", "Print the average time spent on each detector.").Bool()
 	noUpdate             = cli.Flag("no-update", "Don't check for updates.").Bool()
+	fail                 = cli.Flag("fail", "Exit with code 183 if results are found.").Bool()
+	archiveMaxSize       = cli.Flag("archive-max-size", "Maximum size of archive to scan. (Byte units eg. 512B, 2KB, 4MB)").Bytes()
+	archiveMaxDepth      = cli.Flag("archive-max-depth", "Maximum depth of archive to scan.").Int()
+	archiveTimeout       = cli.Flag("archive-timeout", "Maximum time to spend extracting an archive.").Duration()
+	includeDetectors     = cli.Flag("include-detectors", "Comma separated list of detector types to include. Protobuf name or IDs may be used, as well as ranges.").Default("all").String()
+	excludeDetectors     = cli.Flag("exclude-detectors", "Comma separated list of detector types to exclude. Protobuf name or IDs may be used, as well as ranges. IDs defined here take precedence over the include list.").String()
 
 	gitScan             = cli.Command("git", "Find credentials in git repositories.")
-	gitScanURI          = gitScan.Arg("uri", "Git repository URL. https:// or file:// schema expected.").Required().String()
+	gitScanURI          = gitScan.Arg("uri", "Git repository URL. https://, file://, or ssh:// schema expected.").Required().String()
 	gitScanIncludePaths = gitScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
 	gitScanExcludePaths = gitScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
+	gitScanExcludeGlobs = gitScan.Flag("exclude-globs", "Comma separated list of globs to exclude in scan. This option filters at the `git log` level, resulting in faster scans.").String()
 	gitScanSinceCommit  = gitScan.Flag("since-commit", "Commit to start scan from.").String()
 	gitScanBranch       = gitScan.Flag("branch", "Branch to scan.").String()
 	gitScanMaxDepth     = gitScan.Flag("max-depth", "Maximum depth of commits to scan.").Int()
@@ -55,45 +69,96 @@ var (
 	_                   = gitScan.Flag("entropy", "No-op flag for backwards compat.").Bool()
 	_                   = gitScan.Flag("regex", "No-op flag for backwards compat.").Bool()
 
-	githubScan           = cli.Command("github", "Find credentials in GitHub repositories.")
-	githubScanEndpoint   = githubScan.Flag("endpoint", "GitHub endpoint.").Default("https://api.github.com").String()
-	githubScanRepos      = githubScan.Flag("repo", `GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
-	githubScanOrgs       = githubScan.Flag("org", `GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
-	githubScanToken      = githubScan.Flag("token", "GitHub token.").String()
-	githubIncludeForks   = githubScan.Flag("include-forks", "Include forks in scan.").Bool()
-	githubIncludeMembers = githubScan.Flag("include-members", "Include organization member repositories in scan.").Bool()
+	githubScan             = cli.Command("github", "Find credentials in GitHub repositories.")
+	githubScanEndpoint     = githubScan.Flag("endpoint", "GitHub endpoint.").Default("https://api.github.com").String()
+	githubScanRepos        = githubScan.Flag("repo", `GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
+	githubScanOrgs         = githubScan.Flag("org", `GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
+	githubScanToken        = githubScan.Flag("token", "GitHub token. Can be provided with environment variable GITHUB_TOKEN.").Envar("GITHUB_TOKEN").String()
+	githubIncludeForks     = githubScan.Flag("include-forks", "Include forks in scan.").Bool()
+	githubIncludeMembers   = githubScan.Flag("include-members", "Include organization member repositories in scan.").Bool()
+	githubIncludeRepos     = githubScan.Flag("include-repos", `Repositories to include in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/trufflehog", "trufflesecurity/t*"`).Strings()
+	githubExcludeRepos     = githubScan.Flag("exclude-repos", `Repositories to exclude in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/driftwood", "trufflesecurity/d*"`).Strings()
+	githubScanIncludePaths = githubScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
+	githubScanExcludePaths = githubScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
 
 	gitlabScan = cli.Command("gitlab", "Find credentials in GitLab repositories.")
 	// TODO: Add more GitLab options
-	gitlabScanEndpoint = gitlabScan.Flag("endpoint", "GitLab endpoint.").Default("https://gitlab.com").String()
-	gitlabScanRepos    = gitlabScan.Flag("repo", "GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
-	gitlabScanToken    = gitlabScan.Flag("token", "GitLab token.").Required().String()
+	gitlabScanEndpoint     = gitlabScan.Flag("endpoint", "GitLab endpoint.").Default("https://gitlab.com").String()
+	gitlabScanRepos        = gitlabScan.Flag("repo", "GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
+	gitlabScanToken        = gitlabScan.Flag("token", "GitLab token. Can be provided with environment variable GITLAB_TOKEN.").Envar("GITLAB_TOKEN").Required().String()
+	gitlabScanIncludePaths = gitlabScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
+	gitlabScanExcludePaths = gitlabScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
 
-	filesystemScan        = cli.Command("filesystem", "Find credentials in a filesystem.")
-	filesystemDirectories = filesystemScan.Flag("directory", "Path to directory to scan. You can repeat this flag.").Required().Strings()
+	filesystemScan  = cli.Command("filesystem", "Find credentials in a filesystem.")
+	filesystemPaths = filesystemScan.Arg("path", "Path to file or directory to scan.").Strings()
+	// DEPRECATED: --directory is deprecated in favor of arguments.
+	filesystemDirectories = filesystemScan.Flag("directory", "Path to directory to scan. You can repeat this flag.").Strings()
 	// TODO: Add more filesystem scan options. Currently only supports scanning a list of directories.
 	// filesystemScanRecursive = filesystemScan.Flag("recursive", "Scan recursively.").Short('r').Bool()
-	// filesystemScanIncludePaths = filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
-	// filesystemScanExcludePaths = filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
+	filesystemScanIncludePaths = filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
+	filesystemScanExcludePaths = filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
 
 	s3Scan         = cli.Command("s3", "Find credentials in S3 buckets.")
-	s3ScanKey      = s3Scan.Flag("key", "S3 key used to authenticate.").String()
-	s3ScanSecret   = s3Scan.Flag("secret", "S3 secret used to authenticate.").String()
+	s3ScanKey      = s3Scan.Flag("key", "S3 key used to authenticate. Can be provided with environment variable AWS_ACCESS_KEY_ID.").Envar("AWS_ACCESS_KEY_ID").String()
+	s3ScanSecret   = s3Scan.Flag("secret", "S3 secret used to authenticate. Can be provided with environment variable AWS_SECRET_ACCESS_KEY.").Envar("AWS_SECRET_ACCESS_KEY").String()
 	s3ScanCloudEnv = s3Scan.Flag("cloud-environment", "Use IAM credentials in cloud environment.").Bool()
 	s3ScanBuckets  = s3Scan.Flag("bucket", "Name of S3 bucket to scan. You can repeat this flag.").Strings()
+
+	gcsScan           = cli.Command("gcs", "Find credentials in GCS buckets.")
+	gcsProjectID      = gcsScan.Flag("project-id", "GCS project ID used to authenticate. Can NOT be used with unauth scan. Can be provided with environment variable GOOGLE_CLOUD_PROJECT.").Envar("GOOGLE_CLOUD_PROJECT").String()
+	gcsCloudEnv       = gcsScan.Flag("cloud-environment", "Use Application Default Credentials, IAM credentials to authenticate.").Bool()
+	gcsServiceAccount = gcsScan.Flag("service-account", "Path to GCS service account JSON file.").ExistingFile()
+	gcsWithoutAuth    = gcsScan.Flag("without-auth", "Scan GCS buckets without authentication. This will only work for public buckets").Bool()
+	gcsAPIKey         = gcsScan.Flag("api-key", "GCS API key used to authenticate. Can be provided with environment variable GOOGLE_API_KEY.").Envar("GOOGLE_API_KEY").String()
+	gcsIncludeBuckets = gcsScan.Flag("include-buckets", "Buckets to scan. Comma seperated list of buckets. You can repeat this flag. Globs are supported").Short('I').Strings()
+	gcsExcludeBuckets = gcsScan.Flag("exclude-buckets", "Buckets to exclude from scan. Comma separated list of buckets. Globs are supported").Short('X').Strings()
+	gcsIncludeObjects = gcsScan.Flag("include-objects", "Objects to scan. Comma separated list of objects. you can repeat this flag. Globs are supported").Short('i').Strings()
+	gcsExcludeObjects = gcsScan.Flag("exclude-objects", "Objects to exclude from scan. Comma separated list of objects. You can repeat this flag. Globs are supported").Short('x').Strings()
+	gcsMaxObjectSize  = gcsScan.Flag("max-object-size", "Maximum size of objects to scan. Objects larger than this will be skipped. (Byte units eg. 512B, 2KB, 4MB)").Default("10MB").Bytes()
+
+	syslogScan     = cli.Command("syslog", "Scan syslog")
+	syslogAddress  = syslogScan.Flag("address", "Address and port to listen on for syslog. Example: 127.0.0.1:514").String()
+	syslogProtocol = syslogScan.Flag("protocol", "Protocol to listen on. udp or tcp").String()
+	syslogTLSCert  = syslogScan.Flag("cert", "Path to TLS cert.").String()
+	syslogTLSKey   = syslogScan.Flag("key", "Path to TLS key.").String()
+	syslogFormat   = syslogScan.Flag("format", "Log format. Can be rfc3164 or rfc5424").String()
+
+	circleCiScan      = cli.Command("circleci", "Scan CircleCI")
+	circleCiScanToken = circleCiScan.Flag("token", "CircleCI token. Can also be provided with environment variable").Envar("CIRCLECI_TOKEN").Required().String()
 )
 
 func init() {
 	for i, arg := range os.Args {
 		if strings.HasPrefix(arg, "--") {
-			os.Args[i] = strings.ReplaceAll(arg, "_", "-")
+			split := strings.SplitN(arg, "=", 2)
+			split[0] = strings.ReplaceAll(split[0], "_", "-")
+			os.Args[i] = strings.Join(split, "=")
 		}
 	}
 
+	cli.Version("trufflehog " + version.BuildVersion)
 	cmd = kingpin.MustParse(cli.Parse(os.Args[1:]))
+
+	switch {
+	case *trace:
+		log.SetLevel(5)
+	case *debug:
+		log.SetLevel(2)
+	}
 }
 
 func main() {
+	// setup logger
+	logFormat := log.WithConsoleSink
+	if *jsonOut {
+		logFormat = log.WithJSONSink
+	}
+	logger, sync := log.New("trufflehog", logFormat(os.Stderr))
+	// make it the default logger for contexts
+	context.SetDefaultLogger(logger)
+	defer func() { _ = sync() }()
+	logFatal := logFatalFunc(logger)
+
 	updateCfg := overseer.Config{
 		Program:       run,
 		Debug:         *debug,
@@ -105,73 +170,27 @@ func main() {
 	if !*noUpdate {
 		updateCfg.Fetcher = updater.Fetcher(version.BuildVersion)
 	}
+	if version.BuildVersion == "dev" {
+		updateCfg.Fetcher = nil
+	}
+
 	err := overseer.RunErr(updateCfg)
 	if err != nil {
-		logrus.WithError(err).Fatal("error occured with trufflehog updater 🐷")
+		logFatal(err, "error occured with trufflehog updater 🐷")
 	}
 }
 
 func run(state overseer.State) {
-	cli := kingpin.New("TruffleHog", "TruffleHog is a tool for finding credentials.")
-	debug := cli.Flag("debug", "Run in debug mode").Bool()
-	versionFlag := cli.Flag("version", "Prints trufflehog version.").Bool()
-	jsonOut := cli.Flag("json", "Output in JSON format.").Short('j').Bool()
-	jsonLegacy := cli.Flag("json-legacy", "Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.").Bool()
-	concurrency := cli.Flag("concurrency", "Number of concurrent workers.").Default(strconv.Itoa(runtime.NumCPU())).Int()
-	noVerification := cli.Flag("no-verification", "Don't verify the results.").Bool()
-	onlyVerified := cli.Flag("only-verified", "Only output verified results.").Bool()
-	// rules := cli.Flag("rules", "Path to file with custom rules.").String()
-	printAvgDetectorTime := cli.Flag("print-avg-detector-time", "Print the average time spent on each detector.").Bool()
+	ctx := context.Background()
+	logger := ctx.Logger()
+	logFatal := logFatalFunc(logger)
 
-	gitScan := cli.Command("git", "Find credentials in git repositories.")
-	gitScanURI := gitScan.Arg("uri", "Git repository URL. https:// or file:// schema expected.").Required().String()
-	gitScanIncludePaths := gitScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
-	gitScanExcludePaths := gitScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
-	gitScanSinceCommit := gitScan.Flag("since-commit", "Commit to start scan from.").String()
-	gitScanBranch := gitScan.Flag("branch", "Branch to scan.").String()
-	gitScanMaxDepth := gitScan.Flag("max-depth", "Maximum depth of commits to scan.").Int()
-	gitScan.Flag("allow", "No-op flag for backwards compat.").Bool()
-	gitScan.Flag("entropy", "No-op flag for backwards compat.").Bool()
-	gitScan.Flag("regex", "No-op flag for backwards compat.").Bool()
+	logger.V(2).Info(fmt.Sprintf("trufflehog %s", version.BuildVersion))
 
-	githubScan := cli.Command("github", "Find credentials in GitHub repositories.")
-	githubScanEndpoint := githubScan.Flag("endpoint", "GitHub endpoint.").Default("https://api.github.com").String()
-	githubScanRepos := githubScan.Flag("repo", `GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
-	githubScanOrgs := githubScan.Flag("org", `GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
-	githubScanToken := githubScan.Flag("token", "GitHub token.").String()
-	githubIncludeForks := githubScan.Flag("include-forks", "Include forks in scan.").Bool()
-	githubIncludeMembers := githubScan.Flag("include-members", "Include organization member repositories in scan.").Bool()
-
-	gitlabScan := cli.Command("gitlab", "Find credentials in GitLab repositories.")
-	// TODO: Add more GitLab options
-	gitlabScanEndpoint := gitlabScan.Flag("endpoint", "GitLab endpoint.").Default("https://gitlab.com").String()
-	gitlabScanRepos := gitlabScan.Flag("repo", "GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
-	gitlabScanToken := gitlabScan.Flag("token", "GitLab token.").Required().String()
-
-	filesystemScan := cli.Command("filesystem", "Find credentials in a filesystem.")
-	filesystemDirectories := filesystemScan.Flag("directory", "Path to directory to scan. You can repeat this flag.").Required().Strings()
-	// TODO: Add more filesystem scan options. Currently only supports scanning a list of directories.
-	// filesystemScanRecursive := filesystemScan.Flag("recursive", "Scan recursively.").Short('r').Bool()
-	// filesystemScanIncludePaths := filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
-	// filesystemScanExcludePaths := filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
-
-	s3Scan := cli.Command("s3", "Find credentials in S3 buckets.")
-	s3ScanKey := s3Scan.Flag("key", "S3 key used to authenticate.").String()
-	s3ScanSecret := s3Scan.Flag("secret", "S3 secret used to authenticate.").String()
-	s3ScanCloudEnv := s3Scan.Flag("cloud-environment", "Use IAM credentials in cloud environment.").Bool()
-	s3ScanBuckets := s3Scan.Flag("bucket", "Name of S3 bucket to scan. You can repeat this flag.").Strings()
-
-	for i, arg := range os.Args {
-		if strings.HasPrefix(arg, "--") {
-			os.Args[i] = strings.ReplaceAll(arg, "_", "-")
-		}
-	}
-
-	cmd := kingpin.MustParse(cli.Parse(os.Args[1:]))
-
-	if *versionFlag {
-		fmt.Println("trufflehog " + version.BuildVersion)
-		return
+	if *githubScanToken != "" {
+		// NOTE: this kludge is here to do an authenticated shallow commit
+		// TODO: refactor to better pass credentials
+		os.Setenv("GITHUB_TOKEN", *githubScanToken)
 	}
 
 	// When setting a base commit, chunks must be scanned in order.
@@ -179,83 +198,237 @@ func run(state overseer.State) {
 		*concurrency = 1
 	}
 
-	if *jsonOut {
-		logrus.SetFormatter(&logrus.JSONFormatter{})
-	}
-	if *debug {
-		logrus.SetLevel(logrus.DebugLevel)
-	} else {
-		logrus.SetLevel(logrus.InfoLevel)
-	}
-
-	if *debug {
+	if *profile {
 		go func() {
 			router := mux.NewRouter()
 			router.PathPrefix("/debug/pprof").Handler(http.DefaultServeMux)
 			router.PathPrefix("/debug/fgprof").Handler(fgprof.Handler())
-			logrus.Info("starting pprof and fgprof server on :18066 /debug/pprof and /debug/fgprof")
+			logger.Info("starting pprof and fgprof server on :18066 /debug/pprof and /debug/fgprof")
 			if err := http.ListenAndServe(":18066", router); err != nil {
-				logrus.Error(err)
+				logger.Error(err, "error serving pprof and fgprof")
 			}
 		}()
 	}
 
-	ctx := context.TODO()
+	conf := &config.Config{}
+	if *configFilename != "" {
+		var err error
+		conf, err = config.Read(*configFilename)
+		if err != nil {
+			logFatal(err, "error parsing the provided configuration file")
+		}
+	}
+
+	if *archiveMaxSize != 0 {
+		handlers.SetArchiveMaxSize(int(*archiveMaxSize))
+	}
+	if *archiveMaxDepth != 0 {
+		handlers.SetArchiveMaxDepth(*archiveMaxDepth)
+	}
+	if *archiveTimeout != 0 {
+		handlers.SetArchiveMaxTimeout(*archiveTimeout)
+	}
+
+	// Build include and exclude detector filter sets.
+	var includeDetectorTypes, excludeDetectorTypes map[detectorspb.DetectorType]config.DetectorID
+	{
+		includeList, err := config.ParseDetectors(*includeDetectors)
+		if err != nil {
+			// Exit if there was an error to inform the user of the misconfiguration.
+			logFatal(err, "invalid include list detector configuration")
+		}
+		excludeList, err := config.ParseDetectors(*excludeDetectors)
+		if err != nil {
+			// Exit if there was an error to inform the user of the misconfiguration.
+			logFatal(err, "invalid exclude list detector configuration")
+		}
+		includeDetectorTypes = detectorTypeToMap(includeList)
+		excludeDetectorTypes = detectorTypeToMap(excludeList)
+	}
+	includeFilter := func(d detectors.Detector) bool {
+		id, ok := includeDetectorTypes[d.Type()]
+		if id.Version == 0 {
+			return ok
+		}
+		versionD, ok := d.(detectors.Versioner)
+		if !ok {
+			// Error: version provided but not a detectors.Versioner
+			logFatal(
+				fmt.Errorf("version provided but detector does not have a version"),
+				"invalid include list detector configuration",
+				"detector", id,
+			)
+		}
+		return versionD.Version() == id.Version
+	}
+	excludeFilter := func(d detectors.Detector) bool {
+		id, ok := excludeDetectorTypes[d.Type()]
+		if id.Version == 0 {
+			return !ok
+		}
+		versionD, ok := d.(detectors.Versioner)
+		if !ok {
+			// Error: version provided but not a detectors.Versioner
+			logFatal(
+				fmt.Errorf("version provided but detector does not have a version"),
+				"invalid exclude list detector configuration",
+				"detector", id,
+			)
+		}
+		return versionD.Version() != id.Version
+	}
+
 	e := engine.Start(ctx,
 		engine.WithConcurrency(*concurrency),
 		engine.WithDecoders(decoders.DefaultDecoders()...),
 		engine.WithDetectors(!*noVerification, engine.DefaultDetectors()...),
+		engine.WithDetectors(!*noVerification, conf.Detectors...),
+		engine.WithFilterDetectors(includeFilter),
+		engine.WithFilterDetectors(excludeFilter),
+		engine.WithFilterUnverified(*filterUnverified),
 	)
 
-	filter, err := common.FilterFromFiles(*gitScanIncludePaths, *gitScanExcludePaths)
-	if err != nil {
-		logrus.WithError(err).Fatal("could not create filter")
-	}
-
 	var repoPath string
+	var remote bool
 	switch cmd {
 	case gitScan.FullCommand():
-		var remote bool
-		repoPath, remote, err = git.PrepareRepo(*gitScanURI)
+		filter, err := common.FilterFromFiles(*gitScanIncludePaths, *gitScanExcludePaths)
+		if err != nil {
+			logFatal(err, "could not create filter")
+		}
+		repoPath, remote, err = git.PrepareRepoSinceCommit(ctx, *gitScanURI, *gitScanSinceCommit)
 		if err != nil || repoPath == "" {
-			logrus.WithError(err).Fatal("error preparing git repo for scanning")
+			logFatal(err, "error preparing git repo for scanning")
 		}
 		if remote {
 			defer os.RemoveAll(repoPath)
 		}
-		err = e.ScanGit(ctx, repoPath, *gitScanBranch, *gitScanSinceCommit, *gitScanMaxDepth, filter)
-		if err != nil {
-			logrus.WithError(err).Fatal("Failed to scan git.")
+		excludedGlobs := []string{}
+		if *gitScanExcludeGlobs != "" {
+			excludedGlobs = strings.Split(*gitScanExcludeGlobs, ",")
+		}
+
+		cfg := sources.GitConfig{
+			RepoPath:     repoPath,
+			HeadRef:      *gitScanBranch,
+			BaseRef:      *gitScanSinceCommit,
+			MaxDepth:     *gitScanMaxDepth,
+			Filter:       filter,
+			ExcludeGlobs: excludedGlobs,
+		}
+		if err = e.ScanGit(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan Git.")
 		}
 	case githubScan.FullCommand():
-		if len(*githubScanOrgs) == 0 && len(*githubScanRepos) == 0 {
-			log.Fatal("You must specify at least one organization or repository.")
-		}
-		err = e.ScanGitHub(ctx, *githubScanEndpoint, *githubScanRepos, *githubScanOrgs, *githubScanToken, *githubIncludeForks, filter, *concurrency, *githubIncludeMembers)
+		filter, err := common.FilterFromFiles(*githubScanIncludePaths, *githubScanExcludePaths)
 		if err != nil {
-			logrus.WithError(err).Fatal("Failed to scan git.")
+			logFatal(err, "could not create filter")
+		}
+		if len(*githubScanOrgs) == 0 && len(*githubScanRepos) == 0 {
+			logFatal(fmt.Errorf("invalid config"), "You must specify at least one organization or repository.")
+		}
+
+		cfg := sources.GithubConfig{
+			Endpoint:       *githubScanEndpoint,
+			Token:          *githubScanToken,
+			IncludeForks:   *githubIncludeForks,
+			IncludeMembers: *githubIncludeMembers,
+			Concurrency:    *concurrency,
+			ExcludeRepos:   *githubExcludeRepos,
+			IncludeRepos:   *githubIncludeRepos,
+			Repos:          *githubScanRepos,
+			Orgs:           *githubScanOrgs,
+			Filter:         filter,
+		}
+		if err := e.ScanGitHub(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan Github.")
 		}
 	case gitlabScan.FullCommand():
-		err := e.ScanGitLab(ctx, *gitlabScanEndpoint, *gitlabScanToken, *gitlabScanRepos)
+		filter, err := common.FilterFromFiles(*gitlabScanIncludePaths, *gitlabScanExcludePaths)
 		if err != nil {
-			logrus.WithError(err).Fatal("Failed to scan GitLab.")
+			logFatal(err, "could not create filter")
+		}
+
+		cfg := sources.GitlabConfig{
+			Endpoint: *gitlabScanEndpoint,
+			Token:    *gitlabScanToken,
+			Repos:    *gitlabScanRepos,
+			Filter:   filter,
+		}
+		if err := e.ScanGitLab(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan GitLab.")
 		}
 	case filesystemScan.FullCommand():
-		err := e.ScanFileSystem(ctx, *filesystemDirectories)
+		filter, err := common.FilterFromFiles(*filesystemScanIncludePaths, *filesystemScanExcludePaths)
 		if err != nil {
-			logrus.WithError(err).Fatal("Failed to scan filesystem")
+			logFatal(err, "could not create filter")
+		}
+		if len(*filesystemDirectories) > 0 {
+			ctx.Logger().Info("--directory flag is deprecated, please pass directories as arguments")
+		}
+		paths := make([]string, 0, len(*filesystemPaths)+len(*filesystemDirectories))
+		paths = append(paths, *filesystemPaths...)
+		paths = append(paths, *filesystemDirectories...)
+		cfg := sources.FilesystemConfig{
+			Paths:  paths,
+			Filter: filter,
+		}
+		if err = e.ScanFileSystem(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan filesystem")
 		}
 	case s3Scan.FullCommand():
-		err := e.ScanS3(ctx, *s3ScanKey, *s3ScanSecret, *s3ScanCloudEnv, *s3ScanBuckets)
-		if err != nil {
-			logrus.WithError(err).Fatal("Failed to scan S3.")
+		cfg := sources.S3Config{
+			Key:       *s3ScanKey,
+			Secret:    *s3ScanSecret,
+			Buckets:   *s3ScanBuckets,
+			CloudCred: *s3ScanCloudEnv,
+		}
+		if err := e.ScanS3(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan S3.")
+		}
+	case syslogScan.FullCommand():
+		cfg := sources.SyslogConfig{
+			Address:     *syslogAddress,
+			Format:      *syslogFormat,
+			Protocol:    *syslogProtocol,
+			CertPath:    *syslogTLSCert,
+			KeyPath:     *syslogTLSKey,
+			Concurrency: *concurrency,
+		}
+		if err := e.ScanSyslog(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan syslog.")
+		}
+	case circleCiScan.FullCommand():
+		if err := e.ScanCircleCI(ctx, *circleCiScanToken); err != nil {
+			logFatal(err, "Failed to scan CircleCI.")
+		}
+	case gcsScan.FullCommand():
+		cfg := sources.GCSConfig{
+			ProjectID:      *gcsProjectID,
+			CloudCred:      *gcsCloudEnv,
+			ServiceAccount: *gcsServiceAccount,
+			WithoutAuth:    *gcsWithoutAuth,
+			ApiKey:         *gcsAPIKey,
+			IncludeBuckets: commaSeperatedToSlice(*gcsIncludeBuckets),
+			ExcludeBuckets: commaSeperatedToSlice(*gcsExcludeBuckets),
+			IncludeObjects: commaSeperatedToSlice(*gcsIncludeObjects),
+			ExcludeObjects: commaSeperatedToSlice(*gcsExcludeObjects),
+			Concurrency:    *concurrency,
+			MaxObjectSize:  int64(*gcsMaxObjectSize),
+		}
+		if err := e.ScanGCS(ctx, cfg); err != nil {
+			logFatal(err, "Failed to scan GCS.")
 		}
 	}
+	// asynchronously wait for scanning to finish and cleanup
+	go e.Finish(ctx)
 
 	if !*jsonLegacy && !*jsonOut {
 		fmt.Fprintf(os.Stderr, "🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷\n\n")
 	}
 
+	// NOTE: this loop will terminate when the results channel is closed in
+	// e.Finish()
 	foundResults := false
 	for r := range e.ResultsChan() {
 		if *onlyVerified && !r.Verified {
@@ -263,35 +436,50 @@ func run(state overseer.State) {
 		}
 		foundResults = true
 
+		var err error
 		switch {
 		case *jsonLegacy:
-			legacy := output.ConvertToLegacyJSON(&r, repoPath)
-			out, err := json.Marshal(legacy)
-			if err != nil {
-				logrus.WithError(err).Fatal("could not marshal result")
-			}
-			fmt.Println(string(out))
+			err = output.PrintLegacyJSON(ctx, &r)
 		case *jsonOut:
-			out, err := json.Marshal(r)
-			if err != nil {
-				logrus.WithError(err).Fatal("could not marshal result")
-			}
-			fmt.Println(string(out))
+			err = output.PrintJSON(&r)
+		case *gitHubActionsFormat:
+			err = output.PrintGitHubActionsOutput(&r)
 		default:
-			output.PrintPlainOutput(&r)
+			err = output.PrintPlainOutput(&r)
+		}
+		if err != nil {
+			logFatal(err, "error printing results")
 		}
 	}
-	logrus.Debugf("scanned %d chunks", e.ChunksScanned())
+	logger.V(2).Info("finished scanning",
+		"chunks", e.ChunksScanned(),
+		"bytes", e.BytesScanned(),
+	)
 
 	if *printAvgDetectorTime {
 		printAverageDetectorTime(e)
 	}
 
-	if foundResults {
-		os.Exit(1)
+	if foundResults && *fail {
+		logger.V(2).Info("exiting with code 183 because results were found")
+		os.Exit(183)
 	}
 }
 
+func commaSeperatedToSlice(s []string) []string {
+	var result []string
+	for _, items := range s {
+		for _, item := range strings.Split(items, ",") {
+			item = strings.TrimSpace(item)
+			if item == "" {
+				continue
+			}
+			result = append(result, item)
+		}
+	}
+	return result
+}
+
 func printAverageDetectorTime(e *engine.Engine) {
 	fmt.Fprintln(os.Stderr, "Average detector time is the measurement of average time spent on each detector when results are returned.")
 	for detectorName, durations := range e.DetectorAvgTime() {
@@ -303,3 +491,24 @@ func printAverageDetectorTime(e *engine.Engine) {
 		fmt.Fprintf(os.Stderr, "%s: %s\n", detectorName, avgDuration)
 	}
 }
+
+// logFatalFunc returns a log.Fatal style function. Calling the returned
+// function will terminate the program without cleanup.
+func logFatalFunc(logger logr.Logger) func(error, string, ...any) {
+	return func(err error, message string, keyAndVals ...any) {
+		logger.Error(err, message, keyAndVals...)
+		if err != nil {
+			os.Exit(1)
+			return
+		}
+		os.Exit(0)
+	}
+}
+
+func detectorTypeToMap(detectors []config.DetectorID) map[detectorspb.DetectorType]config.DetectorID {
+	output := make(map[detectorspb.DetectorType]config.DetectorID, len(detectors))
+	for _, d := range detectors {
+		output[d.ID] = d
+	}
+	return output
+}

pkg/cache/cache.go

@@ -0,0 +1,20 @@
+// Package cache provides an interface which can be implemented by different cache types.
+package cache
+
+// Cache is used to store key/value pairs.
+type Cache interface {
+	// Set stores the given key/value pair.
+	Set(string, string)
+	// Get returns the value for the given key and a boolean indicating if the key was found.
+	Get(string) (string, bool)
+	// Exists returns true if the given key exists in the cache.
+	Exists(string) bool
+	// Delete the given key from the cache.
+	Delete(string)
+	// Clear all key/value pairs from the cache.
+	Clear()
+	// Count the number of key/value pairs in the cache.
+	Count() int
+	// Contents returns all keys in the cache encoded as a string.
+	Contents() string
+}

pkg/cache/memory/memory.go

@@ -0,0 +1,85 @@
+package memory
+
+import (
+	"strings"
+	"time"
+
+	"github.com/patrickmn/go-cache"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/context"
+)
+
+const (
+	expirationInterval = 12 * time.Hour
+	purgeInterval      = 13 * time.Hour
+	defaultExpiration  = cache.DefaultExpiration
+)
+
+// Cache is a wrapper around the go-cache library.
+type Cache struct {
+	c *cache.Cache
+}
+
+// New constructs a new in-memory cache.
+func New() *Cache {
+	c := cache.New(expirationInterval, purgeInterval)
+	return &Cache{c: c}
+}
+
+// NewWithData constructs a new in-memory cache with existing data.
+func NewWithData(ctx context.Context, data []string) *Cache {
+	ctx.Logger().V(3).Info("Loading cache", "num-items", len(data))
+
+	items := make(map[string]cache.Item, len(data))
+	for _, d := range data {
+		items[d] = cache.Item{Object: d, Expiration: int64(defaultExpiration)}
+	}
+
+	c := cache.NewFrom(expirationInterval, purgeInterval, items)
+	return &Cache{c: c}
+}
+
+// Set adds a key-value pair to the cache.
+func (c *Cache) Set(key, value string) {
+	c.c.Set(key, value, defaultExpiration)
+}
+
+// Get returns the value for the given key.
+func (c *Cache) Get(key string) (string, bool) {
+	res, ok := c.c.Get(key)
+	if !ok {
+		return "", ok
+	}
+	return res.(string), ok
+}
+
+// Exists returns true if the given key exists in the cache.
+func (c *Cache) Exists(key string) bool {
+	_, ok := c.c.Get(key)
+	return ok
+}
+
+// Delete removes the key-value pair from the cache.
+func (c *Cache) Delete(key string) {
+	c.c.Delete(key)
+}
+
+// Clear removes all key-value pairs from the cache.
+func (c *Cache) Clear() {
+	c.c.Flush()
+}
+
+// Count returns the number of key-value pairs in the cache.
+func (c *Cache) Count() int {
+	return c.c.ItemCount()
+}
+
+// Contents returns all key-value pairs in the cache encodes as a string.
+func (c *Cache) Contents() string {
+	items := c.c.Items()
+	res := make([]string, 0, len(items))
+	for k := range items {
+		res = append(res, k)
+	}
+	return strings.Join(res, ",")
+}

pkg/cache/memory/memory_test.go

@@ -0,0 +1,153 @@
+package memory
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+
+	logContext "github.com/trufflesecurity/trufflehog/v3/pkg/context"
+)
+
+func TestCache(t *testing.T) {
+	c := New()
+
+	// Test set and get.
+	c.Set("key1", "key1")
+	v, ok := c.Get("key1")
+	if !ok || v != "key1" {
+		t.Fatalf("Unexpected value for key1: %v, %v", v, ok)
+	}
+
+	// Test exists.
+	if !c.Exists("key1") {
+		t.Fatalf("Expected key1 to exist")
+	}
+
+	// Test the count.
+	if c.Count() != 1 {
+		t.Fatalf("Unexpected count: %d", c.Count())
+	}
+
+	// Test delete.
+	c.Delete("key1")
+	v, ok = c.Get("key1")
+	if ok || v != "" {
+		t.Fatalf("Unexpected value for key1 after delete: %v, %v", v, ok)
+	}
+
+	// Test clear.
+	c.Set("key10", "key10")
+	c.Clear()
+	v, ok = c.Get("key10")
+	if ok || v != "" {
+		t.Fatalf("Unexpected value for key10 after clear: %v, %v", v, ok)
+	}
+
+	// Test contents.
+	keys := []string{"key1", "key2", "key3"}
+	for _, k := range keys {
+		c.Set(k, k)
+	}
+
+	items := c.Contents()
+	sort.Strings(keys)
+	res := strings.Split(items, ",")
+	sort.Strings(res)
+
+	if len(keys) != len(res) {
+		t.Fatalf("Unexpected length of items: %d", len(res))
+	}
+	if !cmp.Equal(keys, res) {
+		t.Fatalf("Unexpected items: %v", res)
+	}
+}
+
+func TestCache_NewWithData(t *testing.T) {
+	c := NewWithData(logContext.Background(), []string{"key1", "key2", "key3"})
+
+	// Test the count.
+	if c.Count() != 3 {
+		t.Fatalf("Unexpected count: %d", c.Count())
+	}
+
+	// Test contents.
+	keys := []string{"key1", "key2", "key3"}
+	items := c.Contents()
+	sort.Strings(keys)
+	res := strings.Split(items, ",")
+	sort.Strings(res)
+
+	if len(keys) != len(res) {
+		t.Fatalf("Unexpected length of items: %d", len(res))
+	}
+	if !cmp.Equal(keys, res) {
+		t.Fatalf("Unexpected items: %v", res)
+	}
+}
+
+func setupBenchmarks(b *testing.B) *Cache {
+	b.Helper()
+
+	c := New()
+
+	for i := 0; i < 500_000; i++ {
+		key := fmt.Sprintf("key%d", i)
+		c.Set(key, key)
+	}
+
+	return c
+}
+
+func BenchmarkSet(b *testing.B) {
+	c := New()
+
+	for i := 0; i < b.N; i++ {
+		key := fmt.Sprintf("key%d", i)
+		c.Set(key, key)
+	}
+}
+
+func BenchmarkGet(b *testing.B) {
+	c := setupBenchmarks(b)
+	b.ResetTimer()
+
+	for i := 0; i < b.N; i++ {
+		key := fmt.Sprintf("key%d", i)
+		c.Get(key)
+	}
+}
+
+func BenchmarkDelete(b *testing.B) {
+	c := setupBenchmarks(b)
+	b.ResetTimer()
+
+	for i := 0; i < b.N; i++ {
+		key := fmt.Sprintf("key%d", i)
+		c.Delete(key)
+	}
+}
+
+func BenchmarkCount(b *testing.B) {
+	c := setupBenchmarks(b)
+	b.ResetTimer()
+
+	for i := 0; i < b.N; i++ {
+		c.Count()
+	}
+}
+
+func BenchmarkContents(b *testing.B) {
+	c := setupBenchmarks(b)
+	b.ResetTimer()
+
+	var s string
+
+	for i := 0; i < b.N; i++ {
+		s = c.Contents()
+	}
+
+	_ = s
+}

pkg/common/filter.go

@@ -6,7 +6,7 @@ import (
 	"os"
 	"regexp"
 
-	log "github.com/sirupsen/logrus"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/context"
 )
 
 type Filter struct {
@@ -20,7 +20,8 @@ type FilterRuleSet []regexp.Regexp
 func FilterEmpty() *Filter {
 	filter, err := FilterFromFiles("", "")
 	if err != nil {
-		log.WithError(err).Fatalf("could not create empty filter")
+		context.Background().Logger().Error(err, "could not create empty filter")
+		os.Exit(1)
 	}
 	return filter
 }
@@ -57,15 +58,19 @@ func FilterRulesFromFile(source string) (*FilterRuleSet, error) {
 	}
 
 	commentPattern := regexp.MustCompile(`^\s*#`)
+	emptyLinePattern := regexp.MustCompile(`^\s*$`)
 
 	file, err := os.Open(source)
+	logger := context.Background().Logger().WithValues("file", source)
 	if err != nil {
-		log.WithError(err).Fatalf("unable to open filter file: %s", source)
+		logger.Error(err, "unable to open filter file", "file", source)
+		os.Exit(1)
 	}
 	defer func(file *os.File) {
 		err := file.Close()
 		if err != nil {
-			log.WithError(err).Fatalf("unable to close filter file: %s", source)
+			logger.Error(err, "unable to close filter file")
+			os.Exit(1)
 		}
 	}(file)
 
@@ -75,6 +80,9 @@ func FilterRulesFromFile(source string) (*FilterRuleSet, error) {
 		if commentPattern.MatchString(line) {
 			continue
 		}
+		if emptyLinePattern.MatchString(line) {
+			continue
+		}
 		pattern, err := regexp.Compile(line)
 		if err != nil {
 			return nil, fmt.Errorf("can not compile regular expression: %s", line)
@@ -86,6 +94,9 @@ func FilterRulesFromFile(source string) (*FilterRuleSet, error) {
 
 // Pass returns true if the include FilterRuleSet matches the pattern and the exclude FilterRuleSet does not match.
 func (filter *Filter) Pass(object string) bool {
+	if filter == nil {
+		return true
+	}
 	excluded := filter.exclude.Matches(object)
 	included := filter.include.Matches(object)
 	return !excluded && included

pkg/common/filter_test.go

@@ -129,6 +129,13 @@ func TestFilterFromFile(t *testing.T) {
 			pattern:             "test",
 			pass:                false,
 		},
+		"EmptyLinesAreIgnored": {
+			includeFile:         false,
+			excludeFile:         true,
+			excludeFileContents: " \ntest.txt",
+			pattern:             "hello world.txt",
+			pass:                true,
+		},
 	}
 	for name, test := range tests {
 		var includeTestFile, excludeTestFile string
@@ -163,6 +170,9 @@ func testFilterWriteFile(filename string, content []byte) error {
 	if err != nil {
 		return err
 	}
-	f.Write(content)
+	_, err = f.Write(content)
+	if err != nil {
+		return err
+	}
 	return f.Close()
 }

pkg/common/http.go

@@ -121,14 +121,37 @@ func RetryableHttpClient() *http.Client {
 	return httpClient.StandardClient()
 }
 
+func RetryableHttpClientTimeout(timeOutSeconds int64) *http.Client {
+	httpClient := retryablehttp.NewClient()
+	httpClient.RetryMax = 3
+	httpClient.Logger = nil
+	httpClient.HTTPClient.Timeout = time.Duration(timeOutSeconds) * time.Second
+	httpClient.HTTPClient.Transport = NewCustomTransport(nil)
+	return httpClient.StandardClient()
+}
+
+const DefaultResponseTimeout = 5 * time.Second
+
+var saneTransport = &http.Transport{
+	Proxy: http.ProxyFromEnvironment,
+	DialContext: (&net.Dialer{
+		Timeout:   2 * time.Second,
+		KeepAlive: 5 * time.Second,
+	}).DialContext,
+	MaxIdleConns:          5,
+	IdleConnTimeout:       5 * time.Second,
+	TLSHandshakeTimeout:   3 * time.Second,
+	ExpectContinueTimeout: 1 * time.Second,
+}
+
 func SaneHttpClient() *http.Client {
 	httpClient := &http.Client{}
-	httpClient.Timeout = time.Second * 2
-	httpClient.Transport = NewCustomTransport(nil)
+	httpClient.Timeout = DefaultResponseTimeout
+	httpClient.Transport = NewCustomTransport(saneTransport)
 	return httpClient
 }
 
-//custom timeout for some scanners
+// SaneHttpClientTimeOut adds a custom timeout for some scanners
 func SaneHttpClientTimeOut(timeOutSeconds int64) *http.Client {
 	httpClient := &http.Client{}
 	httpClient.Timeout = time.Second * time.Duration(timeOutSeconds)

pkg/common/patterns.go

@@ -0,0 +1,39 @@
+package common
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+const EmailPattern = `\b(?:[a-z0-9!#$%&'*+/=?^_\x60{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_\x60{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])\b`
+const SubDomainPattern = `\b([A-Za-z0-9](?:[A-Za-z0-9\-]{0,61}[A-Za-z0-9])?)\b`
+const UUIDPattern = `\b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b`
+const UUIDPatternUpperCase = `\b([0-9A-Z]{8}-[0-9A-Z]{4}-[0-9A-Z]{4}-[0-9A-Z]{4}-[0-9A-Z]{12})\b`
+
+const RegexPattern = "0-9a-z"
+const AlphaNumPattern = "0-9a-zA-Z"
+const HexPattern = "0-9a-f"
+
+//Custom Regex functions
+func BuildRegex(pattern string, specialChar string, length int) string {
+	return fmt.Sprintf(`\b([%s%s]{%s})\b`, pattern, specialChar, strconv.Itoa(length))
+}
+
+func BuildRegexJWT(firstRange, secondRange, thirdRange string) string {
+	if RangeValidation(firstRange) || RangeValidation(secondRange) || RangeValidation(thirdRange) {
+		panic("Min value should not be greater than or equal to max")
+	}
+	return fmt.Sprintf(`\b(ey[%s]{%s}.ey[%s-\/_]{%s}.[%s-\/_]{%s})\b`, AlphaNumPattern, firstRange, AlphaNumPattern, secondRange, AlphaNumPattern, thirdRange)
+}
+
+func RangeValidation(rangeInput string) bool {
+	range_split := strings.Split(rangeInput, ",")
+	range_min, _ := strconv.ParseInt(strings.TrimSpace(range_split[0]), 10, 0)
+	range_max, _ := strconv.ParseInt(strings.TrimSpace(range_split[1]), 10, 0)
+	return range_min >= range_max
+}
+
+func ToUpperCase(input string) string {
+	return strings.ToUpper(input)
+}

pkg/common/recover.go

@@ -0,0 +1,47 @@
+package common
+
+import (
+	"fmt"
+	"os"
+	"runtime/debug"
+	"time"
+
+	"github.com/getsentry/sentry-go"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/context"
+)
+
+// Recover handles panics and reports to Sentry.
+func Recover(ctx context.Context) {
+	if err := recover(); err != nil {
+		panicStack := string(debug.Stack())
+		if eventID := sentry.CurrentHub().Recover(err); eventID != nil {
+			ctx.Logger().Info("panic captured", "event_id", *eventID)
+		}
+		ctx.Logger().Error(fmt.Errorf("panic"), "recovered from panic",
+			"stack-trace", panicStack,
+			"recover", err,
+		)
+		if !sentry.Flush(time.Second * 5) {
+			ctx.Logger().Info("sentry flush failed")
+		}
+	}
+}
+
+// RecoverWithExit handles panics and reports to Sentry before exiting.
+func RecoverWithExit(ctx context.Context) {
+	if err := recover(); err != nil {
+		panicStack := string(debug.Stack())
+		if eventID := sentry.CurrentHub().Recover(err); eventID != nil {
+			ctx.Logger().Info("panic captured", "event_id", *eventID)
+		}
+		ctx.Logger().Error(fmt.Errorf("panic"), "recovered from panic before exiting",
+			"stack-trace", panicStack,
+			"recover", err,
+		)
+		if !sentry.Flush(time.Second * 5) {
+			ctx.Logger().Info("sentry flush failed")
+		}
+		os.Exit(1)
+	}
+}

pkg/common/secrets.go

@@ -3,12 +3,13 @@ package common
 import (
 	"context"
 	"fmt"
+	"os"
 	"time"
 
 	secretmanager "cloud.google.com/go/secretmanager/apiv1"
+	"cloud.google.com/go/secretmanager/apiv1/secretmanagerpb"
 	"github.com/joho/godotenv"
 	"github.com/pkg/errors"
-	secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
 )
 
 type Secret struct{ kv map[string]string }
@@ -21,7 +22,20 @@ func (s *Secret) MustGetField(name string) string {
 	return val
 }
 
+func GetSecretFromEnv(filename string) (secret *Secret, err error) {
+	data, err := godotenv.Read(filename)
+	if err != nil {
+		return nil, err
+	}
+	return &Secret{kv: data}, nil
+}
+
 func GetTestSecret(ctx context.Context) (secret *Secret, err error) {
+	filename := os.Getenv("TEST_SECRET_FILE")
+	if len(filename) > 0 {
+		return GetSecretFromEnv(filename)
+	}
+
 	return GetSecret(ctx, "trufflehog-testing", "test")
 }
 
@@ -29,6 +43,11 @@ func GetSecret(ctx context.Context, gcpProject, name string) (secret *Secret, er
 	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
 	defer cancel()
 
+	filename := os.Getenv("TEST_SECRET_FILE")
+	if len(filename) > 0 {
+		return GetSecretFromEnv(filename)
+	}
+
 	parent := fmt.Sprintf("projects/%s/secrets/%s/versions/latest", gcpProject, name)
 
 	client, err := secretmanager.NewClient(ctx)

pkg/common/vars.go

@@ -2,24 +2,17 @@ package common
 
 import (
 	"path/filepath"
-
-	"github.com/h2non/filetype"
+	"strings"
 )
 
 var (
 	KB, MB, GB, TB, PB = 1e3, 1e6, 1e9, 1e12, 1e15
-	IGNORED_EXTENSIONS = []string{"pdf", "mp4", "avi", "mpeg", "mpg", "mov", "wmv", "m4p", "swf", "mp2", "flv", "vob", "webm", "hdv", "3gp", "ogg", "mp3", "wav", "flac", "tif", "tiff", "jpg", "jpeg", "png", "gif", "zip", "webp"}
+	IgnoredExtensions  = []string{"mp4", "avi", "mpeg", "mpg", "mov", "wmv", "m4p", "swf", "mp2", "flv", "vob", "webm", "hdv", "3gp", "ogg", "mp3", "wav", "flac", "webp"}
 )
 
-func SkipFile(filename string, data []byte) bool {
-	if filepath.Ext(filename) == "" {
-		//no sepcified extension, check mimetype
-		if filetype.IsArchive(data[:256]) {
-			return true
-		}
-	}
-	for _, ext := range IGNORED_EXTENSIONS {
-		if filepath.Ext(filename) == ext {
+func SkipFile(filename string) bool {
+	for _, ext := range IgnoredExtensions {
+		if strings.TrimPrefix(filepath.Ext(filename), ".") == ext {
 			return true
 		}
 	}

pkg/common/vars_test.go

@@ -0,0 +1,36 @@
+package common
+
+import "testing"
+
+func TestSkipFile(t *testing.T) {
+	type testCase struct {
+		file string
+		want bool
+	}
+
+	// Add a test case for each ignored extension.
+	testCases := make([]testCase, 0, len(IgnoredExtensions)+1)
+	for _, ext := range IgnoredExtensions {
+		testCases = append(testCases, testCase{
+			file: "test." + ext,
+			want: true,
+		})
+	}
+
+	// Add a test case for a file that should not be skipped.
+	testCases = append(testCases, testCase{file: "test.txt", want: false})
+
+	for _, tt := range testCases {
+		t.Run(tt.file, func(t *testing.T) {
+			if got := SkipFile(tt.file); got != tt.want {
+				t.Errorf("SkipFile(%v) got %v, want %v", tt.file, got, tt.want)
+			}
+		})
+	}
+}
+
+func BenchmarkSkipFile(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		SkipFile("test.mp4")
+	}
+}

pkg/config/config.go

@@ -0,0 +1,45 @@
+package config
+
+import (
+	"os"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/custom_detectors"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/custom_detectorspb"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/protoyaml"
+)
+
+// Config holds user supplied configuration.
+type Config struct {
+	Detectors []detectors.Detector
+}
+
+// Read parses a given filename into a Config.
+func Read(filename string) (*Config, error) {
+	input, err := os.ReadFile(filename)
+	if err != nil {
+		return nil, err
+	}
+	return NewYAML(input)
+}
+
+// NewYAML parses the given YAML data into a Config.
+func NewYAML(input []byte) (*Config, error) {
+	// Parse the raw YAML into a structure.
+	var messages custom_detectorspb.CustomDetectors
+	if err := protoyaml.UnmarshalStrict(input, &messages); err != nil {
+		return nil, err
+	}
+	// Convert the structured YAML into detectors.
+	var detectors []detectors.Detector
+	for _, detectorConfig := range messages.Detectors {
+		detector, err := custom_detectors.NewWebhookCustomRegex(detectorConfig)
+		if err != nil {
+			return nil, err
+		}
+		detectors = append(detectors, detector)
+	}
+	return &Config{
+		Detectors: detectors,
+	}, nil
+}

pkg/config/detectors.go

@@ -0,0 +1,183 @@
+package config
+
+import (
+	"fmt"
+	"sort"
+	"strconv"
+	"strings"
+
+	dpb "github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+)
+
+var (
+	specialGroups = map[string][]DetectorID{
+		"all": allDetectors(),
+	}
+
+	detectorTypeValue = make(map[string]dpb.DetectorType, len(dpb.DetectorType_value))
+	validDetectors    = make(map[dpb.DetectorType]struct{}, len(dpb.DetectorType_value))
+	maxDetectorType   dpb.DetectorType
+)
+
+// Setup package local global variables.
+func init() {
+	for k, v := range dpb.DetectorType_value {
+		dt := dpb.DetectorType(v)
+		detectorTypeValue[strings.ToLower(k)] = dt
+		validDetectors[dt] = struct{}{}
+		if dt > maxDetectorType {
+			maxDetectorType = dt
+		}
+	}
+}
+
+type DetectorID struct {
+	ID      dpb.DetectorType
+	Version int
+}
+
+// ParseDetectors parses user supplied string into a list of detectors types.
+// "all" will return the list of all available detectors. The input is comma
+// separated and may use the case-insensitive detector name defined in the
+// protobuf, or the protobuf enum number. A range may be used as well in the
+// form "start-end". Order is preserved and duplicates are ignored.
+func ParseDetectors(input string) ([]DetectorID, error) {
+	var output []DetectorID
+	seenDetector := map[DetectorID]struct{}{}
+	for _, item := range strings.Split(input, ",") {
+		item = strings.TrimSpace(item)
+		if item == "" {
+			continue
+		}
+		detectors, ok := specialGroups[strings.ToLower(item)]
+		if !ok {
+			var err error
+			detectors, err = asRange(item)
+			if err != nil {
+				return nil, err
+			}
+		}
+		for _, d := range detectors {
+			if _, ok := seenDetector[d]; ok {
+				continue
+			}
+			seenDetector[d] = struct{}{}
+			output = append(output, d)
+		}
+	}
+	return output, nil
+}
+
+func (id DetectorID) String() string {
+	name := dpb.DetectorType_name[int32(id.ID)]
+	if name == "" {
+		name = "<invalid ID>"
+	}
+	if id.Version == 0 {
+		return name
+	}
+	return fmt.Sprintf("%s.v%d", name, id.Version)
+}
+
+// allDetectors reutrns an ordered slice of all detector types.
+func allDetectors() []DetectorID {
+	all := make([]DetectorID, 0, len(dpb.DetectorType_name))
+	for id := range dpb.DetectorType_name {
+		all = append(all, DetectorID{ID: dpb.DetectorType(id)})
+	}
+	sort.Slice(all, func(i, j int) bool { return all[i].ID < all[j].ID })
+	return all
+}
+
+// asRange converts a single input into a slice of detector types. If the input
+// is not in range format, a slice of length 1 is returned. Unbounded ranges
+// are allowed.
+func asRange(input string) ([]DetectorID, error) {
+	// Check if it's a single detector type.
+	dt, err := asDetectorID(input)
+	if err == nil {
+		return []DetectorID{dt}, nil
+	}
+
+	// Check if it's a range; if not return the error from above.
+	start, end, found := strings.Cut(input, "-")
+	if !found {
+		return nil, err
+	}
+	start, end = strings.TrimSpace(start), strings.TrimSpace(end)
+
+	// Convert the range start and end to a DetectorType.
+	dtStart, err := asDetectorID(start)
+	if err != nil {
+		return nil, err
+	}
+	dtEnd, err := asDetectorID(end)
+	// If end is empty it's an unbounded range.
+	if err != nil && end != "" {
+		return nil, err
+	}
+	if end == "" {
+		dtEnd.ID = maxDetectorType
+	}
+
+	// Ensure these ranges don't have versions.
+	if dtEnd.Version != 0 || dtStart.Version != 0 {
+		return nil, fmt.Errorf("versions within ranges are not supported: %s", input)
+	}
+
+	step := dpb.DetectorType(1)
+	if dtStart.ID > dtEnd.ID {
+		step = -1
+	}
+	var output []DetectorID
+	for dt := dtStart.ID; dt != dtEnd.ID; dt += step {
+		if _, ok := validDetectors[dt]; !ok {
+			continue
+		}
+		output = append(output, DetectorID{ID: dt})
+	}
+	return append(output, dtEnd), nil
+}
+
+// asDetectorID converts the case-insensitive input into a DetectorID.
+// Name or ID may be used.
+func asDetectorID(input string) (DetectorID, error) {
+	if input == "" {
+		return DetectorID{}, fmt.Errorf("empty detector")
+	}
+	var detectorID DetectorID
+	// Separate the version if there is one.
+	if detector, version, hasVersion := strings.Cut(input, "."); hasVersion {
+		parsedVersion, err := parseVersion(version)
+		if err != nil {
+			return DetectorID{}, fmt.Errorf("invalid version for input: %q error: %w", input, err)
+		}
+		detectorID.Version = parsedVersion
+		// Because there was a version, the detector type input is the part before the '.'
+		input = detector
+	}
+
+	// Check if it's a named detector.
+	if dt, ok := detectorTypeValue[strings.ToLower(input)]; ok {
+		detectorID.ID = dt
+		return detectorID, nil
+	}
+	// Check if it's a detector ID.
+	if i, err := strconv.ParseInt(input, 10, 32); err == nil {
+		dt := dpb.DetectorType(i)
+		if _, ok := validDetectors[dt]; !ok {
+			return DetectorID{}, fmt.Errorf("invalid detector ID: %s", input)
+		}
+		detectorID.ID = dt
+		return detectorID, nil
+	}
+	return DetectorID{}, fmt.Errorf("unrecognized detector type: %s", input)
+}
+
+func parseVersion(v string) (int, error) {
+	if !strings.HasPrefix(strings.ToLower(v), "v") {
+		return 0, fmt.Errorf("version must start with 'v'")
+	}
+	version := strings.TrimLeft(v, "vV")
+	return strconv.Atoi(version)
+}

pkg/config/detectors_test.go

@@ -0,0 +1,47 @@
+package config
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	dpb "github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+)
+
+func TestDetectorParsing(t *testing.T) {
+	tests := map[string]struct {
+		input    string
+		expected []DetectorID
+	}{
+		"all":                       {"AlL", allDetectors()},
+		"trailing range":            {"0-", allDetectors()},
+		"all after 1":               {"1-", allDetectors()[1:]},
+		"named and valid range":     {"aWs,8-9", []DetectorID{{ID: dpb.DetectorType_AWS}, {ID: dpb.DetectorType_Github}, {ID: dpb.DetectorType_Gitlab}}},
+		"duplicate order preserved": {"9, 8, 9", []DetectorID{{ID: 9}, {ID: 8}}},
+		"named range":               {"github - gitlab", []DetectorID{{ID: dpb.DetectorType_Github}, {ID: dpb.DetectorType_Gitlab}}},
+		"range preserved":           {"8-9, 7-10", []DetectorID{{ID: 8}, {ID: 9}, {ID: 7}, {ID: 10}}},
+		"reverse range":             {"9-8", []DetectorID{{ID: 9}, {ID: 8}}},
+		"range preserved with all":  {"10-,all", append(allDetectors()[10:], allDetectors()[:10]...)},
+		"empty list item":           {"8, ,9", []DetectorID{{ID: 8}, {ID: 9}}},
+		"invalid end range":         {"0-1337", nil},
+		"invalid name":              {"foo", nil},
+		"negative":                  {"-1", nil},
+		"github.v1":                 {"github.v1", []DetectorID{{ID: dpb.DetectorType_Github, Version: 1}}},
+		"gitlab.v100":               {"gitlab.v100", []DetectorID{{ID: dpb.DetectorType_Gitlab, Version: 100}}},
+		"range with versions":       {"github.v2 - gitlab.v1", nil},
+		"invalid version no v":      {"gitlab.2", nil},
+		"invalid version no number": {"gitlab.github", nil},
+		"capital V is fine":         {"GiTlAb.V2", []DetectorID{{ID: dpb.DetectorType_Gitlab, Version: 2}}},
+		"id number with version":    {"8.v2", []DetectorID{{ID: 8, Version: 2}}},
+	}
+
+	for name, tt := range tests {
+		t.Run(name, func(t *testing.T) {
+			got, gotErr := ParseDetectors(tt.input)
+			if tt.expected == nil {
+				assert.Error(t, gotErr)
+				return
+			}
+			assert.Equal(t, tt.expected, got)
+		})
+	}
+}

pkg/context/context.go

@@ -0,0 +1,174 @@
+package context
+
+import (
+	"context"
+	"fmt"
+	"runtime/debug"
+	"time"
+
+	"github.com/go-logr/logr"
+)
+
+var (
+	// defaultLogger can be set via SetDefaultLogger.
+	defaultLogger logr.Logger = logr.Discard()
+)
+
+// Context wraps context.Context and includes an additional Logger() method.
+type Context interface {
+	context.Context
+	Logger() logr.Logger
+	Parent() context.Context
+	SetParent(ctx context.Context) Context
+}
+
+// Parent returns the parent context.
+func (l logCtx) Parent() context.Context {
+	return l.Context
+}
+
+// SetParent sets the parent context on the context.
+func (l logCtx) SetParent(ctx context.Context) Context {
+	l.Context = ctx
+	return l
+}
+
+type CancelFunc context.CancelFunc
+
+// logCtx implements Context.
+type logCtx struct {
+	// Embed context.Context to get all methods for free.
+	context.Context
+	log logr.Logger
+	err *error
+}
+
+// Logger returns a structured logger.
+func (l logCtx) Logger() logr.Logger {
+	return l.log
+}
+
+func (l logCtx) Err() error {
+	if l.err != nil && *l.err != nil {
+		return *l.err
+	}
+	return l.Context.Err()
+}
+
+// Background returns context.Background with a default logger.
+func Background() Context {
+	return logCtx{
+		log:     defaultLogger,
+		Context: context.Background(),
+	}
+}
+
+// TODO returns context.TODO with a default logger.
+func TODO() Context {
+	return logCtx{
+		log:     defaultLogger,
+		Context: context.TODO(),
+	}
+}
+
+// WithCancel returns context.WithCancel with the log object propagated.
+func WithCancel(parent Context) (Context, context.CancelFunc) {
+	ctx, cancel := context.WithCancel(parent)
+	lCtx := logCtx{
+		log:     parent.Logger(),
+		Context: ctx,
+	}
+	return captureCancelCallstack(lCtx, cancel)
+}
+
+// WithDeadline returns context.WithDeadline with the log object propagated and
+// the deadline added to the structured log values.
+func WithDeadline(parent Context, d time.Time) (Context, context.CancelFunc) {
+	ctx, cancel := context.WithDeadline(parent, d)
+	lCtx := logCtx{
+		log:     parent.Logger().WithValues("deadline", d),
+		Context: ctx,
+	}
+	return captureCancelCallstack(lCtx, cancel)
+}
+
+// WithTimeout returns context.WithTimeout with the log object propagated and
+// the timeout added to the structured log values.
+func WithTimeout(parent Context, timeout time.Duration) (Context, context.CancelFunc) {
+	ctx, cancel := context.WithTimeout(parent, timeout)
+	lCtx := logCtx{
+		log:     parent.Logger().WithValues("timeout", timeout),
+		Context: ctx,
+	}
+	return captureCancelCallstack(lCtx, cancel)
+}
+
+// WithValue returns context.WithValue with the log object propagated and
+// the value added to the structured log values (if the key is a string).
+func WithValue(parent Context, key, val any) Context {
+	logger := parent.Logger()
+	if k, ok := key.(string); ok {
+		logger = logger.WithValues(k, val)
+	}
+	return logCtx{
+		log:     logger,
+		Context: context.WithValue(parent, key, val),
+	}
+}
+
+// WithValues returns context.WithValue with the log object propagated and
+// the values added to the structured log values (if the key is a string).
+func WithValues(parent Context, keyAndVals ...any) Context {
+	ctx := parent
+	for i := 0; i < len(keyAndVals)-1; i += 2 {
+		ctx = WithValue(ctx, keyAndVals[i], keyAndVals[i+1])
+	}
+	return ctx
+}
+
+// WithLogger converts a context.Context into a Context by adding a logger.
+func WithLogger(parent context.Context, logger logr.Logger) Context {
+	return logCtx{
+		log:     logger,
+		Context: parent,
+	}
+}
+
+// AddLogger converts a context.Context into a Context. If the underlying type
+// is already a Context, that will be returned, otherwise a default logger will
+// be added.
+func AddLogger(parent context.Context) Context {
+	if loggerCtx, ok := parent.(Context); ok {
+		return loggerCtx
+	}
+	return WithLogger(parent, defaultLogger)
+}
+
+// SetupDefaultLogger sets the package-level global default logger that will be
+// used for Background and TODO contexts.
+func SetDefaultLogger(l logr.Logger) {
+	defaultLogger = l
+}
+
+// captureCancelCallstack is a helper function to capture the callstack where
+// the cancel function was first called.
+func captureCancelCallstack(ctx logCtx, f context.CancelFunc) (Context, context.CancelFunc) {
+	if ctx.err == nil {
+		var err error
+		ctx.err = &err
+	}
+	return ctx, func() {
+		// We must check Err() before calling f() since f() sets the error.
+		// If there's already an error, do nothing special.
+		if ctx.Err() != nil {
+			f()
+			return
+		}
+		f()
+		// Set the error with the stacktrace if the err pointer is non-nil.
+		*ctx.err = fmt.Errorf(
+			"%w (canceled at %v\n%s)",
+			ctx.Err(), time.Now(), string(debug.Stack()),
+		)
+	}
+}

pkg/context/context_test.go

@@ -0,0 +1,214 @@
+package context
+
+import (
+	"bytes"
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/go-logr/logr"
+	"github.com/go-logr/zapr"
+	"github.com/stretchr/testify/assert"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/log"
+	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
+	"go.uber.org/zap/zaptest"
+)
+
+// testLogger is a helper function to create a logger with a closure callback.
+func testLogger(t *testing.T, f func(zapcore.Entry)) logr.Logger {
+	return zapr.NewLogger(zaptest.NewLogger(t,
+		zaptest.WrapOptions(zap.Hooks(func(e zapcore.Entry) error {
+			f(e)
+			return nil
+		}))))
+}
+
+// infoCounterContext is a helper function to create a Context that will count
+// the number of Info messages logged.
+func infoCounterContext(t *testing.T) (Context, *int) {
+	var infoCount int
+	logger := testLogger(t, func(e zapcore.Entry) {
+		if e.Level == zap.InfoLevel {
+			infoCount++
+		}
+	})
+	return WithLogger(context.Background(), logger), &infoCount
+}
+
+func TestWithCancel(t *testing.T) {
+	parentCtx, infoCount := infoCounterContext(t)
+	ctx, cancel := WithCancel(parentCtx)
+	cancel()
+	assert.Equal(t, 0, *infoCount)
+	select {
+	case <-ctx.Done():
+		ctx.Logger().Info("yay")
+	case <-time.After(1 * time.Second):
+		assert.Fail(t, "context should be done")
+	}
+	assert.Equal(t, 1, *infoCount)
+}
+
+func TestWithTimeout(t *testing.T) {
+	parentCtx, infoCount := infoCounterContext(t)
+	ctx, cancel := WithTimeout(parentCtx, 10*time.Millisecond)
+	defer cancel()
+
+	assert.Equal(t, 0, *infoCount)
+	select {
+	case <-ctx.Done():
+		ctx.Logger().Info("yay")
+	case <-time.After(1 * time.Second):
+		assert.Fail(t, "context should be done")
+	}
+	assert.Equal(t, 1, *infoCount)
+
+	ctx, cancel = WithTimeout(parentCtx, 1*time.Second)
+	defer cancel()
+	select {
+	case <-ctx.Done():
+		assert.Fail(t, "context should not be done")
+	case <-time.After(10 * time.Millisecond):
+		ctx.Logger().Info("yay")
+	}
+	assert.Equal(t, 2, *infoCount)
+}
+
+func TestWithLogger(t *testing.T) {
+	var infoCount int
+	logger := testLogger(t, func(e zapcore.Entry) {
+		if e.Level == zap.InfoLevel {
+			infoCount++
+		}
+	})
+
+	ctx := WithLogger(context.Background(), logger)
+	assert.Equal(t, logger, ctx.Logger())
+
+	assert.Equal(t, 0, infoCount)
+	ctx.Logger().Info("yay")
+	assert.Equal(t, 1, infoCount)
+}
+
+func TestAsContext(t *testing.T) {
+	var gotValue any
+	normalFuncThatTakesContext := func(ctx context.Context) {
+		if logCtx, ok := ctx.(Context); ok {
+			logCtx.Logger().Info("yay")
+		}
+		gotValue = ctx.Value("key")
+	}
+	parentCtx, infoCount := infoCounterContext(t)
+	ctx := WithValue(parentCtx, "key", "value")
+
+	assert.Equal(t, 0, *infoCount)
+	normalFuncThatTakesContext(ctx)
+	assert.Equal(t, 1, *infoCount)
+	assert.Equal(t, "value", gotValue)
+}
+
+func TestWithValues(t *testing.T) {
+	var buffer bytes.Buffer
+	logger, sync := log.New("test",
+		log.WithConsoleSink(&buffer),
+	)
+	defer func(prevLogger logr.Logger) {
+		defaultLogger = prevLogger
+	}(defaultLogger)
+	SetDefaultLogger(logger)
+
+	{
+		ctx1 := Background()
+		ctx1.Logger().Info("only a", "a", 0)
+
+		ctx2 := WithValue(ctx1, "b", 1)
+		ctx2.Logger().Info("only b")
+		assert.Equal(t, 1, ctx2.Value("b"))
+
+		ctx3 := WithLogger(ctx2, ctx2.Logger().WithValues("c", 2, "d", 3))
+		ctx3.Logger().Info("bcd")
+
+		ctx2.Logger().Info("only b again")
+
+		type customKey string
+		ctx4 := WithValue(Background(), customKey("foo"), "bar")
+		// foo:bar shouldn't be added to the logger because the key isn't a string
+		ctx4.Logger().Info("foo")
+
+		ctx5 := WithValues(ctx2, "e", 4, "f", 5, 6, "six")
+		ctx5.Logger().Info("bef")
+		assert.Equal(t, "six", ctx5.Value(6))
+
+		ctx6 := WithValues(ctx2, "what does this do?")
+		ctx6.Logger().Info("silently fail I suppose")
+	}
+	assert.Nil(t, sync())
+	logs := strings.Split(strings.TrimSpace(buffer.String()), "\n")
+
+	assert.Equal(t, 7, len(logs))
+	assert.Contains(t, logs[0], `{"a": 0}`)
+	assert.Contains(t, logs[1], `{"b": 1}`)
+	assert.Contains(t, logs[2], `{"b": 1, "c": 2, "d": 3}`)
+	assert.Contains(t, logs[3], `{"b": 1}`)
+	assert.NotContains(t, logs[4], `{"foo": "bar"}`)
+	assert.Contains(t, logs[5], `{"b": 1, "e": 4, "f": 5}`)
+	assert.Contains(t, logs[6], `silently fail`)
+	assert.NotContains(t, logs[6], `what does this do?`)
+}
+
+func TestDiscardLogger(t *testing.T) {
+	var panicked bool
+	defer func() {
+		if r := recover(); r != nil {
+			panicked = true
+		}
+		assert.False(t, panicked)
+	}()
+	ctx := Background()
+	ctx.Logger().Info("this shouldn't panic")
+}
+
+func TestErrCallstack(t *testing.T) {
+	c, cancel := WithCancel(Background())
+	ctx := c.(logCtx)
+	cancel()
+	select {
+	case <-ctx.Done():
+		assert.Contains(t, ctx.Err().Error(), "TestErrCallstack")
+	case <-time.After(1 * time.Second):
+		assert.Fail(t, "context should be done")
+	}
+}
+
+func TestErrCallstackTimeout(t *testing.T) {
+	ctx, cancel := WithTimeout(Background(), 10*time.Millisecond)
+	defer cancel()
+
+	select {
+	case <-ctx.Done():
+		// Deadline exceeded errors will not have a callstack from the cancel
+		// function.
+		assert.NotContains(t, ctx.Err().Error(), "TestErrCallstackTimeout")
+	case <-time.After(1 * time.Second):
+		assert.Fail(t, "context should be done")
+	}
+}
+
+func TestErrCallstackTimeoutCancel(t *testing.T) {
+	ctx, cancel := WithTimeout(Background(), 10*time.Millisecond)
+
+	var err error
+	select {
+	case <-ctx.Done():
+		err = ctx.Err()
+	case <-time.After(1 * time.Second):
+		assert.Fail(t, "context should be done")
+	}
+
+	// Calling cancel after deadline exceeded should not overwrite the original
+	// error.
+	cancel()
+	assert.Equal(t, err, ctx.Err())
+}

pkg/custom_detectors/custom_detectors.go

@@ -0,0 +1,245 @@
+package custom_detectors
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"regexp"
+	"strings"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/custom_detectorspb"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+	"golang.org/x/sync/errgroup"
+)
+
+// The maximum number of matches from one chunk. This const is used when
+// permutating each regex match to protect the scanner from doing too much work
+// for poorly defined regexps.
+const maxTotalMatches = 100
+
+// customRegexWebhook is a CustomRegex with webhook validation that is
+// guaranteed to be valid (assuming the data is not changed after
+// initialization).
+type customRegexWebhook struct {
+	*custom_detectorspb.CustomRegex
+}
+
+// Ensure the Scanner satisfies the interface at compile time.
+var _ detectors.Detector = (*customRegexWebhook)(nil)
+
+// NewWebhookCustomRegex initializes and validates a customRegexWebhook. An
+// unexported type is intentionally returned here to ensure the values have
+// been validated.
+func NewWebhookCustomRegex(pb *custom_detectorspb.CustomRegex) (*customRegexWebhook, error) {
+	// TODO: Return all validation errors.
+	if err := ValidateKeywords(pb.Keywords); err != nil {
+		return nil, err
+	}
+	if err := ValidateRegex(pb.Regex); err != nil {
+		return nil, err
+	}
+
+	for _, verify := range pb.Verify {
+		if err := ValidateVerifyEndpoint(verify.Endpoint, verify.Unsafe); err != nil {
+			return nil, err
+		}
+		if err := ValidateVerifyHeaders(verify.Headers); err != nil {
+			return nil, err
+		}
+	}
+
+	// TODO: Copy only necessary data out of pb.
+	return &customRegexWebhook{pb}, nil
+}
+
+var httpClient = common.SaneHttpClient()
+
+func (c *customRegexWebhook) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
+	dataStr := string(data)
+	regexMatches := make(map[string][][]string, len(c.GetRegex()))
+
+	// Find all submatches for each regex.
+	for name, regex := range c.GetRegex() {
+		regex, err := regexp.Compile(regex)
+		if err != nil {
+			// This will only happen if the regex is invalid.
+			return nil, err
+		}
+		regexMatches[name] = regex.FindAllStringSubmatch(dataStr, -1)
+	}
+
+	// Permutate each individual match.
+	// {
+	//    "foo": [["match1"]]
+	//    "bar": [["match2"], ["match3"]]
+	// }
+	// becomes
+	// [
+	//    {"foo": ["match1"], "bar": ["match2"]},
+	//    {"foo": ["match1"], "bar": ["match3"]},
+	// ]
+	matches := permutateMatches(regexMatches)
+
+	g := new(errgroup.Group)
+
+	// Create result object and test for verification.
+	resultsCh := make(chan detectors.Result, maxTotalMatches)
+	for _, match := range matches {
+		match := match
+		g.Go(func() error {
+			return c.createResults(ctx, match, verify, resultsCh)
+		})
+	}
+
+	// Ignore any errors and collect as many of the results as we can.
+	_ = g.Wait()
+	close(resultsCh)
+
+	for result := range resultsCh {
+		results = append(results, result)
+	}
+
+	return results, nil
+}
+
+func (c *customRegexWebhook) createResults(ctx context.Context, match map[string][]string, verify bool, results chan<- detectors.Result) error {
+	if common.IsDone(ctx) {
+		// TODO: Log we're possibly leaving out results.
+		return ctx.Err()
+	}
+	var raw string
+	for _, values := range match {
+		// values[0] contains the entire regex match.
+		raw += values[0]
+	}
+	result := detectors.Result{
+		DetectorType: detectorspb.DetectorType_CustomRegex,
+		Raw:          []byte(raw),
+	}
+
+	if !verify {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case results <- result:
+			return nil
+		}
+	}
+	// Verify via webhook.
+	jsonBody, err := json.Marshal(map[string]map[string][]string{
+		c.GetName(): match,
+	})
+	if err != nil {
+		// This should never happen, but if it does, return nil to not
+		// disrupt other verification.
+		return nil
+	}
+	// Try each config until we successfully verify.
+	for _, verifyConfig := range c.GetVerify() {
+		if common.IsDone(ctx) {
+			// TODO: Log we're possibly leaving out results.
+			return ctx.Err()
+		}
+		req, err := http.NewRequestWithContext(ctx, "POST", verifyConfig.GetEndpoint(), bytes.NewReader(jsonBody))
+		if err != nil {
+			continue
+		}
+		for _, header := range verifyConfig.GetHeaders() {
+			key, value, found := strings.Cut(header, ":")
+			if !found {
+				// Should be unreachable due to validation.
+				continue
+			}
+			req.Header.Add(key, strings.TrimLeft(value, "\t\n\v\f\r "))
+		}
+		res, err := httpClient.Do(req)
+		if err != nil {
+			continue
+		}
+		// TODO: Read response body.
+		res.Body.Close()
+		if res.StatusCode == http.StatusOK {
+			result.Verified = true
+			break
+		}
+	}
+
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case results <- result:
+		return nil
+	}
+}
+
+func (c *customRegexWebhook) Keywords() []string {
+	return c.GetKeywords()
+}
+
+// productIndices produces a permutation of indices for each length. Example:
+// productIndices(3, 2) -> [[0 0] [1 0] [2 0] [0 1] [1 1] [2 1]]. It returns
+// a slice of length no larger than maxTotalMatches.
+func productIndices(lengths ...int) [][]int {
+	count := 1
+	for _, l := range lengths {
+		count *= l
+	}
+	if count == 0 {
+		return nil
+	}
+	if count > maxTotalMatches {
+		count = maxTotalMatches
+	}
+
+	results := make([][]int, count)
+	for i := 0; i < count; i++ {
+		j := 1
+		result := make([]int, 0, len(lengths))
+		for _, l := range lengths {
+			result = append(result, (i/j)%l)
+			j *= l
+		}
+		results[i] = result
+	}
+	return results
+}
+
+// permutateMatches converts the list of all regex matches into all possible
+// permutations selecting one from each named entry in the map. For example:
+// {"foo": [matchA, matchB], "bar": [matchC]} becomes
+//
+// [{"foo": matchA, "bar": matchC}, {"foo": matchB, "bar": matchC}]
+func permutateMatches(regexMatches map[string][][]string) []map[string][]string {
+	// Get a consistent order for names and their matching lengths.
+	// The lengths are used in calculating the permutation so order matters.
+	names := make([]string, 0, len(regexMatches))
+	lengths := make([]int, 0, len(regexMatches))
+	for key, value := range regexMatches {
+		names = append(names, key)
+		lengths = append(lengths, len(value))
+	}
+
+	// Permutate all the indices for each match. For example, if "foo" has
+	// [matchA, matchB] and "bar" has [matchC], we will get indices [0 0] [1 0].
+	permutationIndices := productIndices(lengths...)
+
+	// Build {"foo": matchA, "bar": matchC} and {"foo": matchB, "bar": matchC}
+	// from the indices.
+	var matches []map[string][]string
+	for _, permutation := range permutationIndices {
+		candidate := make(map[string][]string, len(permutationIndices))
+		for i, name := range names {
+			candidate[name] = regexMatches[name][permutation[i]]
+		}
+		matches = append(matches, candidate)
+	}
+
+	return matches
+}
+
+func (c *customRegexWebhook) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_CustomRegex
+}

pkg/custom_detectors/custom_detectors_test.go

@@ -0,0 +1,215 @@
+package custom_detectors
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/custom_detectorspb"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/protoyaml"
+)
+
+func TestCustomRegexTemplateParsing(t *testing.T) {
+	testCustomRegexTemplateYaml := `name: Internal bi tool
+keywords:
+- secret_v1_
+- pat_v2_
+regex:
+  id_pat_example: ([a-zA-Z0-9]{32})
+  secret_pat_example: ([a-zA-Z0-9]{32})
+verify:
+- endpoint: http://localhost:8000/{id_pat_example}
+  unsafe: true
+  headers:
+  - 'Authorization: Bearer {secret_pat_example.0}'
+  successRanges:
+  - 200-250
+  - '288'`
+
+	var got custom_detectorspb.CustomRegex
+	assert.NoError(t, protoyaml.UnmarshalStrict([]byte(testCustomRegexTemplateYaml), &got))
+	assert.Equal(t, "Internal bi tool", got.Name)
+	assert.Equal(t, []string{"secret_v1_", "pat_v2_"}, got.Keywords)
+	assert.Equal(t, map[string]string{
+		"id_pat_example":     "([a-zA-Z0-9]{32})",
+		"secret_pat_example": "([a-zA-Z0-9]{32})",
+	}, got.Regex)
+	assert.Equal(t, 1, len(got.Verify))
+	assert.Equal(t, "http://localhost:8000/{id_pat_example}", got.Verify[0].Endpoint)
+	assert.Equal(t, true, got.Verify[0].Unsafe)
+	assert.Equal(t, []string{"Authorization: Bearer {secret_pat_example.0}"}, got.Verify[0].Headers)
+	assert.Equal(t, []string{"200-250", "288"}, got.Verify[0].SuccessRanges)
+}
+
+func TestCustomRegexWebhookParsing(t *testing.T) {
+	testCustomRegexWebhookYaml := `name: Internal bi tool
+keywords:
+- secret_v1_
+- pat_v2_
+regex:
+  id_pat_example: ([a-zA-Z0-9]{32})
+  secret_pat_example: ([a-zA-Z0-9]{32})
+verify:
+- endpoint: http://localhost:8000/
+  unsafe: true
+  headers:
+  - 'Authorization: Bearer token'`
+
+	var got custom_detectorspb.CustomRegex
+	assert.NoError(t, protoyaml.UnmarshalStrict([]byte(testCustomRegexWebhookYaml), &got))
+	assert.Equal(t, "Internal bi tool", got.Name)
+	assert.Equal(t, []string{"secret_v1_", "pat_v2_"}, got.Keywords)
+	assert.Equal(t, map[string]string{
+		"id_pat_example":     "([a-zA-Z0-9]{32})",
+		"secret_pat_example": "([a-zA-Z0-9]{32})",
+	}, got.Regex)
+	assert.Equal(t, 1, len(got.Verify))
+	assert.Equal(t, "http://localhost:8000/", got.Verify[0].Endpoint)
+	assert.Equal(t, true, got.Verify[0].Unsafe)
+	assert.Equal(t, []string{"Authorization: Bearer token"}, got.Verify[0].Headers)
+}
+
+// TestCustomDetectorsParsing tests the full `detectors` configuration.
+func TestCustomDetectorsParsing(t *testing.T) {
+	// TODO: Support both template and webhook.
+	testYamlConfig := `detectors:
+- name: Internal bi tool
+  keywords:
+  - secret_v1_
+  - pat_v2_
+  regex:
+    id_pat_example: ([a-zA-Z0-9]{32})
+    secret_pat_example: ([a-zA-Z0-9]{32})
+  verify:
+  - endpoint: http://localhost:8000/
+    unsafe: true
+    headers:
+    - 'Authorization: Bearer token'`
+
+	var messages custom_detectorspb.CustomDetectors
+	assert.NoError(t, protoyaml.UnmarshalStrict([]byte(testYamlConfig), &messages))
+	assert.Equal(t, 1, len(messages.Detectors))
+
+	got := messages.Detectors[0]
+	assert.Equal(t, "Internal bi tool", got.Name)
+	assert.Equal(t, []string{"secret_v1_", "pat_v2_"}, got.Keywords)
+	assert.Equal(t, map[string]string{
+		"id_pat_example":     "([a-zA-Z0-9]{32})",
+		"secret_pat_example": "([a-zA-Z0-9]{32})",
+	}, got.Regex)
+	assert.Equal(t, 1, len(got.Verify))
+	assert.Equal(t, "http://localhost:8000/", got.Verify[0].Endpoint)
+	assert.Equal(t, true, got.Verify[0].Unsafe)
+	assert.Equal(t, []string{"Authorization: Bearer token"}, got.Verify[0].Headers)
+}
+
+func TestFromData_InvalidRegEx(t *testing.T) {
+	c := &customRegexWebhook{
+		&custom_detectorspb.CustomRegex{
+			Name:     "Internal bi tool",
+			Keywords: []string{"secret_v1_", "pat_v2_"},
+			Regex: map[string]string{
+				"test": "!!?(?:?)[a-zA-Z0-9]{32}", // invalid regex
+			},
+		},
+	}
+
+	_, err := c.FromData(context.Background(), false, []byte("test"))
+	assert.Error(t, err)
+}
+
+func TestProductIndices(t *testing.T) {
+	tests := []struct {
+		name  string
+		input []int
+		want  [][]int
+	}{
+		{
+			name:  "zero",
+			input: []int{3, 0},
+			want:  nil,
+		},
+		{
+			name:  "one input",
+			input: []int{3},
+			want:  [][]int{{0}, {1}, {2}},
+		},
+		{
+			name:  "two inputs",
+			input: []int{3, 2},
+			want: [][]int{
+				{0, 0}, {1, 0}, {2, 0},
+				{0, 1}, {1, 1}, {2, 1},
+			},
+		},
+		{
+			name:  "three inputs",
+			input: []int{3, 2, 3},
+			want: [][]int{
+				{0, 0, 0}, {1, 0, 0}, {2, 0, 0},
+				{0, 1, 0}, {1, 1, 0}, {2, 1, 0},
+				{0, 0, 1}, {1, 0, 1}, {2, 0, 1},
+				{0, 1, 1}, {1, 1, 1}, {2, 1, 1},
+				{0, 0, 2}, {1, 0, 2}, {2, 0, 2},
+				{0, 1, 2}, {1, 1, 2}, {2, 1, 2},
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := productIndices(tt.input...)
+			assert.Equal(t, tt.want, got)
+		})
+	}
+}
+
+func TestProductIndicesMax(t *testing.T) {
+	got := productIndices(2, 3, 4, 5, 6)
+	assert.GreaterOrEqual(t, 2*3*4*5*6, maxTotalMatches)
+	assert.Equal(t, maxTotalMatches, len(got))
+}
+
+func TestPermutateMatches(t *testing.T) {
+	tests := []struct {
+		name  string
+		input map[string][][]string
+		want  []map[string][]string
+	}{
+		{
+			name:  "two matches",
+			input: map[string][][]string{"foo": {{"matchA"}, {"matchB"}}, "bar": {{"matchC"}}},
+			want: []map[string][]string{
+				{"foo": {"matchA"}, "bar": {"matchC"}},
+				{"foo": {"matchB"}, "bar": {"matchC"}},
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := permutateMatches(tt.input)
+			assert.Equal(t, tt.want, got)
+		})
+	}
+}
+
+func TestDetector(t *testing.T) {
+	detector, err := NewWebhookCustomRegex(&custom_detectorspb.CustomRegex{
+		Name: "test",
+		// "password" is normally flagged as a false positive, but CustomRegex
+		// should allow the user to decide and report it as a result.
+		Keywords: []string{"password"},
+		Regex:    map[string]string{"regex": "password=.*"},
+	})
+	assert.NoError(t, err)
+	results, err := detector.FromData(context.Background(), false, []byte(`password="123456"`))
+	assert.NoError(t, err)
+	assert.Equal(t, 1, len(results))
+	assert.Equal(t, results[0].Raw, []byte(`password="123456"`))
+}
+
+func BenchmarkProductIndices(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		_ = productIndices(3, 2, 6)
+	}
+}

pkg/custom_detectors/regex_varstring.go

@@ -0,0 +1,44 @@
+package custom_detectors
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+)
+
+// nameGroupRegex matches `{ name . group }` ignoring any whitespace.
+var nameGroupRegex = regexp.MustCompile(`{\s*([a-zA-Z0-9-_]+)\s*(\.\s*[0-9]*)?\s*}`)
+
+// RegexVarString is a string with embedded {name.group} variables. A name may
+// only contain alphanumeric, hyphen, and underscore characters. Group is
+// optional but if provided it must be a non-negative integer. If the group is
+// omitted it defaults to 0.
+type RegexVarString struct {
+	original string
+	// map from name to group
+	variables map[string]int
+}
+
+func NewRegexVarString(original string) RegexVarString {
+	variables := make(map[string]int)
+
+	matches := nameGroupRegex.FindAllStringSubmatch(original, -1)
+	for _, match := range matches {
+		name, group := match[1], 0
+		// The second match will start with a period followed by any number
+		// of whitespace.
+		if len(match[2]) > 1 {
+			g, err := strconv.Atoi(strings.TrimSpace(match[2][1:]))
+			if err != nil {
+				continue
+			}
+			group = g
+		}
+		variables[name] = group
+	}
+
+	return RegexVarString{
+		original:  original,
+		variables: variables,
+	}
+}

pkg/custom_detectors/regex_varstring_test.go

@@ -0,0 +1,76 @@
+package custom_detectors
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestVarString(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    string
+		wantVars map[string]int
+	}{
+		{
+			name:     "empty",
+			input:    "{}",
+			wantVars: map[string]int{},
+		},
+		{
+			name:  "no subgroup",
+			input: "{hello}",
+			wantVars: map[string]int{
+				"hello": 0,
+			},
+		},
+		{
+			name:  "with subgroup",
+			input: "{hello.123}",
+			wantVars: map[string]int{
+				"hello": 123,
+			},
+		},
+		{
+			name:  "subgroup with spaces",
+			input: "{\thell0  . 123  }",
+			wantVars: map[string]int{
+				"hell0": 123,
+			},
+		},
+		{
+			name:  "multiple groups",
+			input: "foo {bar} {bazz.buzz} {buzz.2}",
+			wantVars: map[string]int{
+				"bar":  0,
+				"buzz": 2,
+			},
+		},
+		{
+			name:  "nested groups",
+			input: "{foo {bar}}",
+			wantVars: map[string]int{
+				"bar": 0,
+			},
+		},
+		{
+			name:  "decimal without number",
+			input: "{foo.}",
+			wantVars: map[string]int{
+				"foo": 0,
+			},
+		},
+		{
+			name:     "negative number",
+			input:    "{foo.-1}",
+			wantVars: map[string]int{},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := NewRegexVarString(tt.input)
+			assert.Equal(t, tt.input, got.original)
+			assert.Equal(t, tt.wantVars, got.variables)
+		})
+	}
+}

pkg/custom_detectors/validation.go

@@ -0,0 +1,109 @@
+package custom_detectors
+
+import (
+	"fmt"
+	"regexp"
+	"strconv"
+	"strings"
+)
+
+func ValidateKeywords(keywords []string) error {
+	if len(keywords) == 0 {
+		return fmt.Errorf("no keywords")
+	}
+
+	for _, keyword := range keywords {
+		if len(keyword) == 0 {
+			return fmt.Errorf("empty keyword")
+		}
+	}
+	return nil
+}
+
+func ValidateRegex(regex map[string]string) error {
+	if len(regex) == 0 {
+		return fmt.Errorf("no regex")
+	}
+	for name, reg := range regex {
+		if _, err := regexp.Compile(reg); err != nil {
+			return fmt.Errorf("regex '%s': %w", name, err)
+		}
+	}
+	return nil
+}
+
+func ValidateVerifyEndpoint(endpoint string, unsafe bool) error {
+	if len(endpoint) == 0 {
+		return fmt.Errorf("no endpoint")
+	}
+
+	if strings.HasPrefix(endpoint, "http://") && !unsafe {
+		return fmt.Errorf("http endpoint must have unsafe=true")
+	}
+	return nil
+}
+
+func ValidateVerifyHeaders(headers []string) error {
+	for _, header := range headers {
+		if !strings.Contains(header, ":") {
+			return fmt.Errorf("header %q must contain a colon", header)
+		}
+	}
+	return nil
+}
+
+func ValidateVerifyRanges(ranges []string) error {
+	const httpLowerRange = 100
+	const httpUpperRange = 599
+
+	for _, successRange := range ranges {
+		if !strings.Contains(successRange, "-") {
+			httpCode, err := strconv.Atoi(successRange)
+			if err != nil {
+				return fmt.Errorf("unable to convert http code to int %q", successRange)
+			}
+
+			if httpCode < httpLowerRange || httpCode > httpUpperRange {
+				return fmt.Errorf("invalid http status code %q", successRange)
+			}
+
+			continue
+		}
+
+		httpRange := strings.Split(successRange, "-")
+		if len(httpRange) != 2 {
+			return fmt.Errorf("invalid range format %q", successRange)
+		}
+
+		lowerBound, err := strconv.Atoi(httpRange[0])
+		if err != nil {
+			return fmt.Errorf("unable to convert lower bound to int %q", successRange)
+		}
+
+		upperBound, err := strconv.Atoi(httpRange[1])
+		if err != nil {
+			return fmt.Errorf("unable to convert upper bound to int %q", successRange)
+		}
+
+		if lowerBound > upperBound {
+			return fmt.Errorf("lower bound greater than upper bound on range %q", successRange)
+		}
+
+		if lowerBound < httpLowerRange || upperBound > httpUpperRange {
+			return fmt.Errorf("invalid http status code range %q", successRange)
+		}
+	}
+	return nil
+}
+
+func ValidateRegexVars(regex map[string]string, body ...string) error {
+	for _, b := range body {
+		matches := NewRegexVarString(b).variables
+		for match := range matches {
+			if _, ok := regex[match]; !ok {
+				return fmt.Errorf("body %q contains an unknown variable", b)
+			}
+		}
+	}
+	return nil
+}

pkg/custom_detectors/validation_test.go

@@ -0,0 +1,234 @@
+package custom_detectors
+
+import "testing"
+
+func TestCustomDetectorsKeywordValidation(t *testing.T) {
+	tests := []struct {
+		name    string
+		input   []string
+		wantErr bool
+	}{
+		{
+			name:    "Test empty list of keywords",
+			input:   []string{},
+			wantErr: true,
+		},
+		{
+			name:    "Test empty keyword",
+			input:   []string{""},
+			wantErr: true,
+		},
+		{
+			name:    "Test valid keywords",
+			input:   []string{"hello", "world"},
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateKeywords(tt.input)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateKeywords() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestCustomDetectorsRegexValidation(t *testing.T) {
+	tests := []struct {
+		name    string
+		input   map[string]string
+		wantErr bool
+	}{
+		{
+			name: "Test list of keywords",
+			input: map[string]string{
+				"id_pat_example": "([a-zA-Z0-9]{32})",
+			},
+			wantErr: false,
+		},
+		{
+			name:    "Test empty list of keywords",
+			input:   map[string]string{},
+			wantErr: true,
+		},
+		{
+			name: "Test invalid regex",
+			input: map[string]string{
+				"test": "!!?(?:?)[a-zA-Z0-9]{32}",
+			},
+			wantErr: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateRegex(tt.input)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateRegex() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestCustomDetectorsVerifyEndpointValidation(t *testing.T) {
+	tests := []struct {
+		name     string
+		endpoint string
+		unsafe   bool
+		wantErr  bool
+	}{
+		{
+			name:     "Test http endpoint with unsafe flag",
+			endpoint: "http://localhost:8000/{id_pat_example}",
+			unsafe:   true,
+			wantErr:  false,
+		},
+		{
+			name:     "Test http endpoint without unsafe flag",
+			endpoint: "http://localhost:8000/{id_pat_example}",
+			unsafe:   false,
+			wantErr:  true,
+		},
+		{
+			name:     "Test https endpoint with unsafe flag",
+			endpoint: "https://localhost:8000/{id_pat_example}",
+			unsafe:   true,
+			wantErr:  false,
+		},
+		{
+			name:     "Test https endpoint without unsafe flag",
+			endpoint: "https://localhost:8000/{id_pat_example}",
+			unsafe:   false,
+			wantErr:  false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateVerifyEndpoint(tt.endpoint, tt.unsafe)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateVerifyEndpoint() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestCustomDetectorsVerifyHeadersValidation(t *testing.T) {
+	tests := []struct {
+		name    string
+		headers []string
+		wantErr bool
+	}{
+		{
+			name:    "Test single header",
+			headers: []string{"Authorization: Bearer {secret_pat_example.0}"},
+			wantErr: false,
+		},
+		{
+			name:    "Test invalid header",
+			headers: []string{"Hello world"},
+			wantErr: true,
+		},
+		{
+			name:    "Test ugly header",
+			headers: []string{"Hello:::::::world::hi:"},
+			wantErr: false,
+		},
+		{
+			name:    "Test empty header",
+			headers: []string{},
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateVerifyHeaders(tt.headers)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateVerifyHeaders() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestCustomDetectorsVerifyRangeValidation(t *testing.T) {
+	tests := []struct {
+		name    string
+		ranges  []string
+		wantErr bool
+	}{
+		{
+			name:    "Test multiple mixed ranges",
+			ranges:  []string{"200", "300-350"},
+			wantErr: false,
+		},
+		{
+			name:    "Test invalid non-number range",
+			ranges:  []string{"hi"},
+			wantErr: true,
+		},
+		{
+			name:    "Test invalid lower to upper range",
+			ranges:  []string{"200-100"},
+			wantErr: true,
+		},
+		{
+			name:    "Test invalid http range",
+			ranges:  []string{"400-1000"},
+			wantErr: true,
+		},
+		{
+			name:    "Test multiple ranges with invalid inputs",
+			ranges:  []string{"322", "hello-world", "100-200"},
+			wantErr: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateVerifyRanges(tt.ranges)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateVerifyRanges() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestCustomDetectorsVerifyRegexVarsValidation(t *testing.T) {
+	tests := []struct {
+		name    string
+		regex   map[string]string
+		body    string
+		wantErr bool
+	}{
+		{
+			name:    "Regex defined but not used in body",
+			regex:   map[string]string{"id": "[0-9]{1,10}", "id_pat_example": "([a-zA-Z0-9]{32})"},
+			body:    "hello world",
+			wantErr: false,
+		},
+		{
+			name:    "Regex defined and is used in body",
+			regex:   map[string]string{"id": "[0-9]{1,10}", "id_pat_example": "([a-zA-Z0-9]{32})"},
+			body:    "hello world {id}",
+			wantErr: false,
+		},
+		{
+			name:    "Regex var in body but not defined",
+			regex:   map[string]string{"id": "[0-9]{1,10}", "id_pat_example": "([a-zA-Z0-9]{32})"},
+			body:    "hello world {hello}",
+			wantErr: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ValidateRegexVars(tt.regex, tt.body)
+
+			if (got != nil && !tt.wantErr) || (got == nil && tt.wantErr) {
+				t.Errorf("ValidateRegexVars() error = %v, wantErr %v", got, tt.wantErr)
+			}
+		})
+	}
+}

pkg/decoders/base64.go

@@ -0,0 +1,78 @@
+package decoders
+
+import (
+	"bytes"
+	"encoding/base64"
+	"strings"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
+)
+
+type Base64 struct{}
+
+var (
+	b64Charset  = []byte("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=")
+	b64EndChars = "+/="
+)
+
+func getSubstringsOfCharacterSet(data []byte, charset []byte, threshold int) []string {
+	count := 0
+	substrings := []string{}
+	letters := strings.Builder{}
+	if len(data) == 0 {
+		return nil
+	}
+	for _, char := range string(data) {
+		if bytes.ContainsRune(charset, char) {
+			letters.WriteRune(char)
+			count++
+		} else {
+			if count > threshold {
+				substrings = appendB64Substring(letters, substrings)
+			}
+			letters.Reset()
+			count = 0
+		}
+	}
+
+	if count > threshold && len(letters.String()) > 0 {
+		substrings = appendB64Substring(letters, substrings)
+	}
+
+	return substrings
+}
+
+func appendB64Substring(letters strings.Builder, substrings []string) []string {
+
+	substring := strings.TrimLeft(letters.String(), b64EndChars)
+	// handle key=value
+	if strings.Contains(strings.TrimRight(substring, b64EndChars), "=") {
+		split := strings.SplitN(substring, "=", 2)
+		substrings = append(substrings, split[len(split)-1])
+	} else {
+		substrings = append(substrings, substring)
+	}
+	return substrings
+}
+
+func (d *Base64) FromChunk(chunk *sources.Chunk) *sources.Chunk {
+
+	encodedSubstrings := getSubstringsOfCharacterSet(chunk.Data, b64Charset, 20)
+	decodedSubstrings := map[string][]byte{}
+
+	for _, str := range encodedSubstrings {
+		dec, err := base64.StdEncoding.DecodeString(str)
+		if err == nil && len(dec) > 0 {
+			decodedSubstrings[str] = dec
+		}
+	}
+
+	if len(decodedSubstrings) > 0 {
+		for substring, dec := range decodedSubstrings {
+			chunk.Data = bytes.Replace(chunk.Data, []byte(substring), dec, 1)
+		}
+		return chunk
+	}
+
+	return nil
+}

pkg/decoders/base64_test.go

@@ -0,0 +1,102 @@
+package decoders
+
+import (
+	"testing"
+
+	"github.com/kylelemons/godebug/pretty"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
+)
+
+func TestBase64_FromChunk(t *testing.T) {
+	tests := []struct {
+		name  string
+		chunk *sources.Chunk
+		want  *sources.Chunk
+	}{
+		{
+			name: "only b64 chunk",
+			chunk: &sources.Chunk{
+				Data: []byte(`bG9uZ2VyLWVuY29kZWQtc2VjcmV0LXRlc3Q=`),
+			},
+			want: &sources.Chunk{
+				Data: []byte(`longer-encoded-secret-test`),
+			},
+		},
+		{
+			name: "mixed content",
+			chunk: &sources.Chunk{
+				Data: []byte(`token: bG9uZ2VyLWVuY29kZWQtc2VjcmV0LXRlc3Q=`),
+			},
+			want: &sources.Chunk{
+				Data: []byte(`token: longer-encoded-secret-test`),
+			},
+		},
+		{
+			name: "no chunk",
+			chunk: &sources.Chunk{
+				Data: []byte(``),
+			},
+			want: nil,
+		},
+		{
+			name: "env var (looks like all b64 decodable but has `=` in the middle)",
+			chunk: &sources.Chunk{
+				Data: []byte(`some-encoded-secret=dGVzdHNlY3JldA==`),
+			},
+			want: &sources.Chunk{
+				Data: []byte(`some-encoded-secret=testsecret`),
+			},
+		},
+		{
+			name: "has longer b64 inside",
+			chunk: &sources.Chunk{
+				Data: []byte(`some-encoded-secret="bG9uZ2VyLWVuY29kZWQtc2VjcmV0LXRlc3Q="`),
+			},
+			want: &sources.Chunk{
+				Data: []byte(`some-encoded-secret="longer-encoded-secret-test"`),
+			},
+		},
+		{
+			name: "many possible substrings",
+			chunk: &sources.Chunk{
+				Data: []byte(`Many substrings in this slack message could be base64 decoded
+				but only dGhpcyBlbmNhcHN1bGF0ZWQgc2VjcmV0 should be decoded.`),
+			},
+			want: &sources.Chunk{
+				Data: []byte(`Many substrings in this slack message could be base64 decoded
+				but only this encapsulated secret should be decoded.`),
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			d := &Base64{}
+			got := d.FromChunk(tt.chunk)
+			if tt.want != nil {
+				if got == nil {
+					t.Fatal("got nil, did not want nil")
+				}
+				if diff := pretty.Compare(string(got.Data), string(tt.want.Data)); diff != "" {
+					t.Errorf("Base64FromChunk() %s diff: (-got +want)\n%s", tt.name, diff)
+				}
+			} else {
+				if got != nil {
+					t.Error("Expected nil chunk")
+				}
+			}
+		})
+	}
+}
+
+func BenchmarkFromChunk(benchmark *testing.B) {
+	d := Base64{}
+	for name, data := range detectors.MustGetBenchmarkData() {
+		benchmark.Run(name, func(b *testing.B) {
+			for n := 0; n < b.N; n++ {
+				d.FromChunk(&sources.Chunk{Data: data})
+			}
+		})
+	}
+}

pkg/decoders/decoders.go

@@ -6,7 +6,8 @@ import (
 
 func DefaultDecoders() []Decoder {
 	return []Decoder{
-		&Plain{},
+		&UTF8{},
+		&Base64{},
 	}
 }
 

pkg/decoders/plain.go

@@ -1,14 +0,0 @@
-package decoders
-
-import (
-	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
-)
-
-// Ensure the Decoder satisfies the interface at compile time
-var _ Decoder = (*Plain)(nil)
-
-type Plain struct{}
-
-func (d *Plain) FromChunk(chunk *sources.Chunk) *sources.Chunk {
-	return chunk
-}

pkg/decoders/utf8.go

@@ -0,0 +1,44 @@
+package decoders
+
+import (
+	"bytes"
+	"unicode/utf8"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
+)
+
+type UTF8 struct{}
+
+func (d *UTF8) FromChunk(chunk *sources.Chunk) *sources.Chunk {
+	if chunk == nil || len(chunk.Data) == 0 {
+		return nil
+	}
+
+	if !utf8.Valid(chunk.Data) {
+		chunk.Data = extractSubstrings(chunk.Data)
+		return chunk
+	}
+
+	return chunk
+}
+
+// extractSubstrings performs similarly to the strings binutil,
+// extacting contigous portions of printable characters that we care
+// about from some bytes
+func extractSubstrings(b []byte) []byte {
+	fields := bytes.FieldsFunc(b, func(r rune) bool {
+		// https://www.rapidtables.com/code/text/ascii-table.html
+		// split on anything that is not ascii space through tilde
+		return !(r > 31 && r < 127)
+	})
+
+	keep := [][]byte{}
+	for _, field := range fields {
+		// Remove fields shorter than 6 characters.
+		if bts := bytes.TrimSpace(field); len(bts) > 5 {
+			keep = append(keep, bts)
+		}
+	}
+
+	return bytes.Join(keep, []byte("\n"))
+}

pkg/decoders/utf8_test.go

@@ -0,0 +1,87 @@
+package decoders
+
+import (
+	"testing"
+
+	"github.com/kylelemons/godebug/pretty"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
+)
+
+func TestUTF8_FromChunk(t *testing.T) {
+	type args struct {
+		chunk *sources.Chunk
+	}
+	tests := []struct {
+		name    string
+		d       *UTF8
+		args    args
+		want    *sources.Chunk
+		wantErr bool
+	}{
+		{
+			name: "successful UTF8 decode",
+			d:    &UTF8{},
+			args: args{
+				chunk: &sources.Chunk{Data: []byte("plain 'ol chunk that should decode successfully")},
+			},
+			want:    &sources.Chunk{Data: []byte("plain 'ol chunk that should decode successfully")},
+			wantErr: false,
+		},
+		{
+			name: "successful binary decode",
+			d:    &UTF8{},
+			args: args{
+				chunk: &sources.Chunk{Data: []byte("\xf0\x28\x8c\x28 not-entirely utf8 chunk that should decode successfully")},
+			},
+			want:    &sources.Chunk{Data: []byte("( not-entirely utf8 chunk that should decode successfully")},
+			wantErr: false,
+		},
+		{
+			name: "unsuccessful decode",
+			d:    &UTF8{},
+			args: args{
+				chunk: nil,
+			},
+			want:    nil,
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			d := &UTF8{}
+			got := d.FromChunk(tt.args.chunk)
+			if got != nil && tt.want != nil {
+				if diff := pretty.Compare(string(got.Data), string(tt.want.Data)); diff != "" {
+					t.Errorf("%s: Plain.FromChunk() diff: (-got +want)\n%s", tt.name, diff)
+				}
+			} else {
+				if diff := pretty.Compare(got, tt.want); diff != "" {
+					t.Errorf("%s: Plain.FromChunk() diff: (-got +want)\n%s", tt.name, diff)
+				}
+			}
+		})
+	}
+}
+
+var testBytes = []byte(`some words   with random spaces and
+	
+newlines with           
+arbitrary length           
+of
+
+	hey
+
+the lines themselves.
+
+and
+short
+words
+that
+go
+away.`)
+
+func Benchmark_extractSubstrings(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		extractSubstrings(testBytes)
+	}
+}

pkg/detectors/abbysale/abbysale.go

@@ -2,7 +2,6 @@ package abbysale
 
 import (
 	"context"
-	"fmt"
 	"net/http"
 	"regexp"
 	"strings"
@@ -14,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"abbysale"}) + `\b([a-z0-9A-Z]{40})\b`)
 )
 
@@ -52,14 +51,14 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			if err != nil {
 				continue
 			}
-			req.Header.Add("x-api-key", fmt.Sprintf("%s", resMatch))
+			req.Header.Add("x-api-key", resMatch)
 			res, err := client.Do(req)
 			if err == nil {
 				defer res.Body.Close()
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -70,5 +69,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Abbysale
 }

pkg/detectors/abbysale/abbysale_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package abbysale
 
 import (

pkg/detectors/abstract/abstract.go

@@ -14,13 +14,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"abstract"}) + `\b([0-9a-z]{32})\b`)
 )
 
@@ -59,7 +59,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -70,5 +70,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Abstract
 }

pkg/detectors/abstract/abstract_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package abstract
 
 import (

pkg/detectors/abuseipdb/abuseipdb.go

@@ -2,14 +2,12 @@ package abuseipdb
 
 import (
 	"context"
-	"io/ioutil"
-
+	"io"
+	"net/http"
 	// "log"
 	"regexp"
 	"strings"
 
-	"net/http"
-
 	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
@@ -17,13 +15,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"abuseipdb"}) + `\b([a-z0-9]{80})\b`)
 )
 
@@ -58,11 +56,11 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			req.Header.Add("Key", resMatch)
 			res, err := client.Do(req)
 			if err == nil {
-				bodyBytes, err := ioutil.ReadAll(res.Body)
+				bodyBytes, err := io.ReadAll(res.Body)
 				if err == nil {
 					bodyString := string(bodyBytes)
 					validResponse := strings.Contains(bodyString, `ipAddress`)
-					//errCode := strings.Contains(bodyString, `AbuseIPDB APIv2 Server.`)
+					// errCode := strings.Contains(bodyString, `AbuseIPDB APIv2 Server.`)
 
 					defer res.Body.Close()
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
@@ -72,7 +70,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 							s1.Verified = false
 						}
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -83,5 +81,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AbuseIPDB
 }

pkg/detectors/abuseipdb/abuseipdb_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package abuseipdb
 
 import (

pkg/detectors/accuweather/accuweather.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"accuweather"}) + `([a-z0-9A-Z\%]{35})\b`)
 )
 
@@ -57,7 +57,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -68,5 +68,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Accuweather
 }

pkg/detectors/accuweather/accuweather_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package accuweather
 
 import (

pkg/detectors/adafruitio/adafruitio.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(`\b(aio\_[a-zA-Z0-9]{28})\b`)
 )
 
@@ -57,7 +57,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -68,5 +68,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AdafruitIO
 }

pkg/detectors/adafruitio/adafruitio_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package adafruitio
 
 import (

pkg/detectors/adobeio/adobeio.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"adobe"}) + `\b([a-z0-9]{32})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"adobe"}) + `\b([a-zA-Z0-9.]{12})\b`)
 )
@@ -51,6 +51,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_AdobeIO,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
@@ -66,7 +67,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -79,5 +80,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AdobeIO
 }

pkg/detectors/adobeio/adobeio_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package adobeio
 
 import (

pkg/detectors/adzuna/adzuna.go

@@ -14,13 +14,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"adzuna"}) + `\b([a-z0-9]{32})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"adzuna"}) + `\b([a-z0-9]{8})\b`)
 )
@@ -52,6 +52,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_Adzuna,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
@@ -65,7 +66,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -77,5 +78,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		}
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Adzuna
 }

pkg/detectors/adzuna/adzuna_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package adzuna
 
 import (

pkg/detectors/aeroworkflow/aeroworkflow.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"aeroworkflow"}) + `\b([a-zA-Z0-9^!]{20})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"aeroworkflow"}) + `\b([0-9]{1,})\b`)
 )
@@ -52,6 +52,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_Aeroworkflow,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
@@ -67,7 +68,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -80,5 +81,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Aeroworkflow
 }

pkg/detectors/aeroworkflow/aeroworkflow_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package aeroworkflow
 
 import (

pkg/detectors/agora/agora.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat    = regexp.MustCompile(detectors.PrefixRegex([]string{"agora"}) + `\b([a-z0-9]{32})\b`)
 	secretPat = regexp.MustCompile(detectors.PrefixRegex([]string{"agora"}) + `\b([a-z0-9]{32})\b`)
 )
@@ -53,6 +53,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_Agora,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resSecret),
 			}
 
 			if verify {
@@ -68,7 +69,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -80,5 +81,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		}
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Agora
 }

pkg/detectors/agora/agora_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package agora
 
 import (

pkg/detectors/aha/aha.go

@@ -1,4 +1,4 @@
-package viewneo
+package aha
 
 import (
 	"context"
@@ -14,27 +14,28 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
-	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"viewneo"}) + `\b([a-z0-9A-Z]{120,300}.[a-z0-9A-Z]{150,300}.[a-z0-9A-Z-_]{600,800})`)
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
+	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"aha"}) + `\b([0-9a-f]{64})\b`)
 )
 
 // Keywords are used for efficiently pre-filtering chunks.
 // Use identifiers in the secret preferably, or the provider name.
 func (s Scanner) Keywords() []string {
-	return []string{"viewneo"}
+	return []string{"aha"}
 }
 
-// FromData will find and optionally verify Viewneo secrets in a given set of bytes.
+// FromData will find and optionally verify Aha secrets in a given set of bytes.
 func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
 	dataStr := string(data)
 
 	matches := keyPat.FindAllStringSubmatch(dataStr, -1)
+
 	for _, match := range matches {
 		if len(match) != 2 {
 			continue
@@ -42,16 +43,16 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		resMatch := strings.TrimSpace(match[1])
 
 		s1 := detectors.Result{
-			DetectorType: detectorspb.DetectorType_Viewneo,
+			DetectorType: detectorspb.DetectorType_Aha,
 			Raw:          []byte(resMatch),
 		}
 
 		if verify {
-			req, err := http.NewRequestWithContext(ctx, "GET", "https://cloud.viewneo.com/api/v1.0/playlist", nil)
+			req, err := http.NewRequestWithContext(ctx, "GET", "https://company.aha.io/api/v1/me", nil)
 			if err != nil {
 				continue
 			}
-			req.Header.Add("Accept", "application/json")
+			req.Header.Add("Accept", "application/vnd.aha+json; version=3")
 			req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", resMatch))
 			res, err := client.Do(req)
 			if err == nil {
@@ -59,7 +60,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -70,5 +71,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Aha
 }

pkg/detectors/aha/aha_test.go

@@ -0,0 +1,119 @@
+//go:build detectors
+// +build detectors
+
+package aha
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/kylelemons/godebug/pretty"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+)
+
+func TestAha_FromChunk(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
+	defer cancel()
+	testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors3")
+	if err != nil {
+		t.Fatalf("could not get test secrets from GCP: %s", err)
+	}
+	secret := testSecrets.MustGetField("AHA")
+	inactiveSecret := testSecrets.MustGetField("AHA_INACTIVE")
+
+	type args struct {
+		ctx    context.Context
+		data   []byte
+		verify bool
+	}
+	tests := []struct {
+		name    string
+		s       Scanner
+		args    args
+		want    []detectors.Result
+		wantErr bool
+	}{
+		{
+			name: "found, verified",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte(fmt.Sprintf("You can find a aha secret %s within", secret)),
+				verify: true,
+			},
+			want: []detectors.Result{
+				{
+					DetectorType: detectorspb.DetectorType_Aha,
+					Verified:     true,
+				},
+			},
+			wantErr: false,
+		},
+		{
+			name: "found, unverified",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte(fmt.Sprintf("You can find a aha secret %s within but not valid", inactiveSecret)), // the secret would satisfy the regex but not pass validation
+				verify: true,
+			},
+			want: []detectors.Result{
+				{
+					DetectorType: detectorspb.DetectorType_Aha,
+					Verified:     false,
+				},
+			},
+			wantErr: false,
+		},
+		{
+			name: "not found",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte("You cannot find the secret within"),
+				verify: true,
+			},
+			want:    nil,
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := Scanner{}
+			got, err := s.FromData(tt.args.ctx, tt.args.verify, tt.args.data)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("Aha.FromData() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			for i := range got {
+				if len(got[i].Raw) == 0 {
+					t.Fatalf("no raw secret present: \n %+v", got[i])
+				}
+				got[i].Raw = nil
+			}
+			if diff := pretty.Compare(got, tt.want); diff != "" {
+				t.Errorf("Aha.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
+			}
+		})
+	}
+}
+
+func BenchmarkFromData(benchmark *testing.B) {
+	ctx := context.Background()
+	s := Scanner{}
+	for name, data := range detectors.MustGetBenchmarkData() {
+		benchmark.Run(name, func(b *testing.B) {
+			for n := 0; n < b.N; n++ {
+				_, err := s.FromData(ctx, false, data)
+				if err != nil {
+					b.Fatal(err)
+				}
+			}
+		})
+	}
+}

pkg/detectors/airbrakeprojectkey/airbrakeprojectkey.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"airbrake"}) + `\b([a-zA-Z-0-9]{32})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"airbrake"}) + `\b([0-9]{6})\b`)
 )
@@ -53,6 +53,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_AirbrakeProjectKey,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
@@ -84,5 +85,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AirbrakeProjectKey
 }

pkg/detectors/airbrakeprojectkey/airbrakeprojectkey_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package airbrakeprojectkey
 
 import (

pkg/detectors/airbrakeuserkey/airbrakeuserkey.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"airbrake"}) + `\b([a-zA-Z-0-9]{40})\b`)
 )
 
@@ -57,7 +57,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -68,5 +68,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AirbrakeUserKey
 }

pkg/detectors/airbrakeuserkey/airbrakeuserkey_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package airbrakeuserkey
 
 import (

pkg/detectors/airship/airship.go

@@ -14,13 +14,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"airship"}) + `\b([0-9Aa-zA-Z]{91})\b`)
 )
 
@@ -60,7 +60,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -71,5 +71,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Airship
 }

pkg/detectors/airship/airship_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package airship
 
 import (

pkg/detectors/airtableapikey/airtableapikey.go

@@ -14,7 +14,7 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
@@ -53,6 +53,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				DetectorType: detectorspb.DetectorType_AirtableApiKey,
 				Redacted:     appRes,
 				Raw:          []byte(keyRes),
+				RawV2:        []byte(keyRes + appRes),
 			}
 
 			if verify {
@@ -78,5 +79,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		}
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AirtableApiKey
 }

pkg/detectors/airtableapikey/airtableapikey_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package airtableapikey
 
 import (

pkg/detectors/airvisual/airvisual.go

@@ -14,13 +14,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"airvisual"}) + `\b([a-z0-9-]{36})\b`)
 )
 
@@ -60,7 +60,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -71,5 +71,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AirVisual
 }

pkg/detectors/airvisual/airvisual_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package airvisual
 
 import (

pkg/detectors/alchemy/alchemy.go

@@ -0,0 +1,76 @@
+package alchemy
+
+import (
+	"context"
+	"net/http"
+	"regexp"
+	"strings"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+)
+
+type Scanner struct{}
+
+// Ensure the Scanner satisfies the interface at compile time.
+var _ detectors.Detector = (*Scanner)(nil)
+
+var (
+	client = common.SaneHttpClient()
+
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
+	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"alchemy"}) + `\b([0-9a-zA-Z]{23}_[0-9a-zA-Z]{8})\b`)
+)
+
+// Keywords are used for efficiently pre-filtering chunks.
+// Use identifiers in the secret preferably, or the provider name.
+func (s Scanner) Keywords() []string {
+	return []string{"alchemy"}
+}
+
+// FromData will find and optionally verify Alchemy secrets in a given set of bytes.
+func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
+	dataStr := string(data)
+
+	matches := keyPat.FindAllStringSubmatch(dataStr, -1)
+
+	for _, match := range matches {
+		if len(match) != 2 {
+			continue
+		}
+		resMatch := strings.TrimSpace(match[1])
+
+		s1 := detectors.Result{
+			DetectorType: detectorspb.DetectorType_Alchemy,
+			Raw:          []byte(resMatch),
+		}
+
+		if verify {
+			req, err := http.NewRequestWithContext(ctx, "GET", "https://eth-mainnet.g.alchemy.com/v2/"+resMatch+"/getNFTs/?owner=vitalik.eth", nil)
+			if err != nil {
+				continue
+			}
+			res, err := client.Do(req)
+			if err == nil {
+				defer res.Body.Close()
+				if res.StatusCode >= 200 && res.StatusCode < 300 {
+					s1.Verified = true
+				} else {
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
+					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
+						continue
+					}
+				}
+			}
+		}
+
+		results = append(results, s1)
+	}
+
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Alchemy
+}

pkg/detectors/alchemy/alchemy_test.go

@@ -0,0 +1,119 @@
+//go:build detectors
+// +build detectors
+
+package alchemy
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/kylelemons/godebug/pretty"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
+)
+
+func TestAlchemy_FromChunk(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
+	defer cancel()
+	testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors4")
+	if err != nil {
+		t.Fatalf("could not get test secrets from GCP: %s", err)
+	}
+	secret := testSecrets.MustGetField("ALCHEMY")
+	inactiveSecret := testSecrets.MustGetField("ALCHEMY_INACTIVE")
+
+	type args struct {
+		ctx    context.Context
+		data   []byte
+		verify bool
+	}
+	tests := []struct {
+		name    string
+		s       Scanner
+		args    args
+		want    []detectors.Result
+		wantErr bool
+	}{
+		{
+			name: "found, verified",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte(fmt.Sprintf("You can find a alchemy secret %s within", secret)),
+				verify: true,
+			},
+			want: []detectors.Result{
+				{
+					DetectorType: detectorspb.DetectorType_Alchemy,
+					Verified:     true,
+				},
+			},
+			wantErr: false,
+		},
+		{
+			name: "found, unverified",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte(fmt.Sprintf("You can find a alchemy secret %s within but not valid", inactiveSecret)), // the secret would satisfy the regex but not pass validation
+				verify: true,
+			},
+			want: []detectors.Result{
+				{
+					DetectorType: detectorspb.DetectorType_Alchemy,
+					Verified:     false,
+				},
+			},
+			wantErr: false,
+		},
+		{
+			name: "not found",
+			s:    Scanner{},
+			args: args{
+				ctx:    context.Background(),
+				data:   []byte("You cannot find the secret within"),
+				verify: true,
+			},
+			want:    nil,
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := Scanner{}
+			got, err := s.FromData(tt.args.ctx, tt.args.verify, tt.args.data)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("Alchemy.FromData() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			for i := range got {
+				if len(got[i].Raw) == 0 {
+					t.Fatalf("no raw secret present: \n %+v", got[i])
+				}
+				got[i].Raw = nil
+			}
+			if diff := pretty.Compare(got, tt.want); diff != "" {
+				t.Errorf("Alchemy.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
+			}
+		})
+	}
+}
+
+func BenchmarkFromData(benchmark *testing.B) {
+	ctx := context.Background()
+	s := Scanner{}
+	for name, data := range detectors.MustGetBenchmarkData() {
+		benchmark.Run(name, func(b *testing.B) {
+			for n := 0; n < b.N; n++ {
+				_, err := s.FromData(ctx, false, data)
+				if err != nil {
+					b.Fatal(err)
+				}
+			}
+		})
+	}
+}

pkg/detectors/alconost/alconost.go

@@ -15,13 +15,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"alconost"}) + `\b([0-9Aa-z]{32})\b`)
 )
 
@@ -63,7 +63,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -74,5 +74,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Alconost
 }

pkg/detectors/alconost/alconost_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package alconost
 
 import (

pkg/detectors/alegra/alegra.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"alegra"}) + `\b([a-z0-9-]{20})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"alegra"}) + `\b([a-zA-Z0-9.-@]{25,30})\b`)
 )
@@ -53,6 +53,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_Alegra,
 				Raw:          []byte(tokenPatMatch),
+				RawV2:        []byte(tokenPatMatch + userPatMatch),
 			}
 
 			if verify {
@@ -67,7 +68,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(tokenPatMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -79,5 +80,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		}
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Alegra
 }

pkg/detectors/alegra/alegra_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package alegra
 
 import (

pkg/detectors/aletheiaapi/aletheiaapi.go

@@ -14,13 +14,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"aletheiaapi"}) + `\b([A-Z0-9]{32})\b`)
 )
 
@@ -61,7 +61,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 				if res.StatusCode >= 200 && res.StatusCode < 300 {
 					s1.Verified = true
 				} else {
-					//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+					// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 					if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 						continue
 					}
@@ -72,5 +72,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 		results = append(results, s1)
 	}
 
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AletheiaApi
 }

pkg/detectors/aletheiaapi/aletheiaapi_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package aletheiaapi
 
 import (

pkg/detectors/algoliaadminkey/algoliaadminkey.go

@@ -13,13 +13,13 @@ import (
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
 	client = common.SaneHttpClient()
 
-	//Make sure that your group is surrounded in boundry characters such as below to reduce false positives
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
 	keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"algolia"}) + `\b([a-zA-Z0-9]{32})\b`)
 	idPat  = regexp.MustCompile(detectors.PrefixRegex([]string{"algolia"}) + `\b([A-Z0-9]{10})\b`)
 )
@@ -51,6 +51,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_AlgoliaAdminKey,
 				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
@@ -66,7 +67,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 					if res.StatusCode >= 200 && res.StatusCode < 300 {
 						s1.Verified = true
 					} else {
-						//This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
 						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
 							continue
 						}
@@ -77,5 +78,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
 			results = append(results, s1)
 		}
 	}
-	return detectors.CleanResults(results), nil
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_AlgoliaAdminKey
 }

pkg/detectors/algoliaadminkey/algoliaadminkey_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package algoliaadminkey
 
 import (

pkg/detectors/alibaba/alibaba.go

@@ -2,23 +2,33 @@ package alibaba
 
 import (
 	"context"
+	"crypto/hmac"
+	"crypto/sha1"
+	"encoding/base64"
+	"math/rand"
+	"net/http"
+	"net/url"
 	"regexp"
+	"strconv"
+	"strings"
 	"time"
 
-	"github.com/aliyun/alibaba-cloud-sdk-go/services/ecs"
-	log "github.com/sirupsen/logrus"
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
 )
 
 type Scanner struct{}
 
-// Ensure the Scanner satisfies the interface at compile time
+// Ensure the Scanner satisfies the interface at compile time.
 var _ detectors.Detector = (*Scanner)(nil)
 
 var (
-	keyPat    = regexp.MustCompile(`\b(LTAI[a-zA-Z0-9]{17,21})[\"' ;\s]*`)
-	secretPat = regexp.MustCompile(`\b([a-zA-Z0-9]{30})\b`)
+	client = common.SaneHttpClient()
+
+	// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
+	keyPat = regexp.MustCompile(`\b([a-zA-Z0-9]{30})\b`)
+	idPat  = regexp.MustCompile(`\b(LTAI[a-zA-Z0-9]{17,21})[\"';\s]*`)
 )
 
 // Keywords are used for efficiently pre-filtering chunks.
@@ -27,58 +37,100 @@ func (s Scanner) Keywords() []string {
 	return []string{"LTAI"}
 }
 
+func randString(n int) string {
+	rand.Seed(time.Now().UnixNano())
+	const alphanum = "0123456789abcdefghijklmnopqrstuvwxyz"
+	var bytes = make([]byte, n)
+	rand.Read(bytes)
+	for i, b := range bytes {
+		bytes[i] = alphanum[b%byte(len(alphanum))]
+	}
+	return string(bytes)
+}
+
+func GetSignature(input, key string) string {
+	key_for_sign := []byte(key)
+	h := hmac.New(sha1.New, key_for_sign)
+	h.Write([]byte(input))
+	return base64.StdEncoding.EncodeToString(h.Sum(nil))
+}
+func buildStringToSign(method, input string) string {
+	filter := strings.Replace(input, "+", "%20", -1)
+	filter = strings.Replace(filter, "%7E", "~", -1)
+	filter = strings.Replace(filter, "*", "%2A", -1)
+	filter = method + "&%2F&" + url.QueryEscape(filter)
+	return filter
+}
+
 // FromData will find and optionally verify Alibaba secrets in a given set of bytes.
 func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
 	dataStr := string(data)
 	matches := keyPat.FindAllStringSubmatch(dataStr, -1)
+	idMatches := idPat.FindAllStringSubmatch(dataStr, -1)
+
 	for _, match := range matches {
-		//Plausible key pat found, look for secrets match
-		secMatches := secretPat.FindAllStringSubmatch(dataStr, -1)
+		if len(match) != 2 {
+			continue
+		}
+		resMatch := strings.TrimSpace(match[1])
 
-		for _, secMatch := range secMatches {
-
-			if len(match) != 2 {
+		for _, idMatch := range idMatches {
+			if len(idMatch) != 2 {
 				continue
 			}
 
-			s := detectors.Result{
+			resIdMatch := strings.TrimSpace(idMatch[1])
+
+			s1 := detectors.Result{
 				DetectorType: detectorspb.DetectorType_Alibaba,
-				Raw:          []byte(match[1]),
-				Redacted:     match[1],
+				Raw:          []byte(resMatch),
+				RawV2:        []byte(resMatch + resIdMatch),
 			}
 
 			if verify {
-				ecsClient, err := ecs.NewClientWithAccessKey(
-					"us-east-1", // your region ID
-					match[1],    // your AccessKey ID
-					secMatch[1]) // your AccessKey Secret
+				req, err := http.NewRequestWithContext(ctx, "GET", "http://ecs.aliyuncs.com/?", nil)
 				if err != nil {
-					log.WithError(err).Debug("error creating alibaba client, skipping")
 					continue
 				}
-				// Create an API request and set parameters
-				request := ecs.CreateDescribeInstancesRequest()
-				request.ConnectTimeout = time.Duration(5) * time.Second
-				request.Scheme = "https"
-				request.Domain = "ecs.aliyuncs.com"
-				// Initiate the request and handle exceptions
-				_, err = ecsClient.DescribeInstances(request)
-				if err != nil {
-					s.Verified = false
+				dateISO := time.Now().UTC().Format("2006-01-02T15:04:05Z07:00")
+				params := req.URL.Query()
+				params.Add("AccessKeyId", resIdMatch)
+				params.Add("Action", "DescribeRegions")
+				params.Add("Format", "JSON")
+				params.Add("SignatureMethod", "HMAC-SHA1")
+				params.Add("SignatureNonce", randString(16))
+				params.Add("SignatureVersion", "1.0")
+				params.Add("Timestamp", dateISO)
+				params.Add("Version", "2014-05-26")
 
-				} else {
-					s.Verified = true
+				stringToSign := buildStringToSign(req.Method, params.Encode())
+				signature := GetSignature(stringToSign, resMatch+"&") // Get Signature HMAC SHA1
+				params.Add("Signature", signature)
+				req.URL.RawQuery = params.Encode()
+
+				req.Header.Add("Content-Type", "text/xml;charset=utf-8")
+				req.Header.Add("Content-Length", strconv.Itoa(len(params.Encode())))
+				res, err := client.Do(req)
+				if err == nil {
+					defer res.Body.Close()
+					if res.StatusCode >= 200 && res.StatusCode < 300 {
+						s1.Verified = true
+					} else {
+						// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
+						if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
+							continue
+						}
+					}
 				}
 			}
 
-			if !s.Verified {
-				if detectors.IsKnownFalsePositive(string(s.Raw), detectors.DefaultFalsePositives, true) {
-					continue
-				}
-			}
-
-			results = append(results, s)
+			results = append(results, s1)
 		}
 	}
-	return
+
+	return results, nil
+}
+
+func (s Scanner) Type() detectorspb.DetectorType {
+	return detectorspb.DetectorType_Alibaba
 }

pkg/detectors/alibaba/alibaba_test.go

@@ -1,3 +1,6 @@
+//go:build detectors
+// +build detectors
+
 package alibaba
 
 import (
@@ -7,20 +10,21 @@ import (
 	"time"
 
 	"github.com/kylelemons/godebug/pretty"
-	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
+
+	"github.com/trufflesecurity/trufflehog/v3/pkg/common"
 	"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
 )
 
 func TestAlibaba_FromChunk(t *testing.T) {
 	ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
 	defer cancel()
-	testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors2")
+	testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors4")
 	if err != nil {
 		t.Fatalf("could not get test secrets from GCP: %s", err)
 	}
 	secret := testSecrets.MustGetField("ALIBABA_SECRET")
-	secretInactive := testSecrets.MustGetField("ALIBABA_SECRET_INACTIVE")
+	inactiveSecret := testSecrets.MustGetField("ALIBABA_SECRET_INACTIVE")
 	id := testSecrets.MustGetField("ALIBABA_ID")
 
 	type args struct {
@@ -40,14 +44,13 @@ func TestAlibaba_FromChunk(t *testing.T) {
 			s:    Scanner{},
 			args: args{
 				ctx:    context.Background(),
-				data:   []byte(fmt.Sprintf("Alibaba keys are here: %s\n %s within", id, secret)),
+				data:   []byte(fmt.Sprintf("You can find a alibaba secret %s within alibaba %s", secret, id)),
 				verify: true,
 			},
 			want: []detectors.Result{
 				{
 					DetectorType: detectorspb.DetectorType_Alibaba,
 					Verified:     true,
-					Redacted:     id,
 				},
 			},
 			wantErr: false,
@@ -57,14 +60,13 @@ func TestAlibaba_FromChunk(t *testing.T) {
 			s:    Scanner{},
 			args: args{
 				ctx:    context.Background(),
-				data:   []byte(fmt.Sprintf("Alibaba keys are here: %s\n %s within", id, secretInactive)),
+				data:   []byte(fmt.Sprintf("You can find a alibaba secret %s within alibaba %s but not valid", inactiveSecret, id)), // the secret would satisfy the regex but not pass validation
 				verify: true,
 			},
 			want: []detectors.Result{
 				{
 					DetectorType: detectorspb.DetectorType_Alibaba,
 					Verified:     false,
-					Redacted:     id,
 				},
 			},
 			wantErr: false,
@@ -91,7 +93,7 @@ func TestAlibaba_FromChunk(t *testing.T) {
 			}
 			for i := range got {
 				if len(got[i].Raw) == 0 {
-					t.Fatal("no raw secret present")
+					t.Fatalf("no raw secret present: \n %+v", got[i])
 				}
 				got[i].Raw = nil
 			}
@@ -101,3 +103,18 @@ func TestAlibaba_FromChunk(t *testing.T) {
 		})
 	}
 }
+
+func BenchmarkFromData(benchmark *testing.B) {
+	ctx := context.Background()
+	s := Scanner{}
+	for name, data := range detectors.MustGetBenchmarkData() {
+		benchmark.Run(name, func(b *testing.B) {
+			for n := 0; n < b.N; n++ {
+				_, err := s.FromData(ctx, false, data)
+				if err != nil {
+					b.Fatal(err)
+				}
+			}
+		})
+	}
+}