Compare commits

..

37 Commits

Author SHA1 Message Date
Zubair Khan 50b05655af Merge branch 'main' into thog-scanner-detector 2023-07-06 19:15:44 -04:00
Zubair Khan d8814d2036 update test 2023-07-06 19:15:15 -04:00
Zubair Khan ecebfde7e7 remove debug logs 2023-07-06 19:13:51 -04:00
Zubair Khan 4988bc7fea remove unused client 2023-07-06 19:13:11 -04:00
Zubair Khan a1a948c2c2 proto stuff 2023-07-06 19:09:04 -04:00
Zubair Khan b38857edb4 fix missing api key, tighten up regex pattern, use response body check (#1438) 2023-07-06 16:35:52 -04:00
Richard Gomez 23757dbe0a remove image4 detector (#1461) 2023-07-06 12:56:09 -07:00
Peter Dave Hello 7a55a146a3 Remove additional apk clean up in Dockerfile (#1440) 2023-07-06 12:55:08 -07:00
Zubair Khan 9840de71aa patterns and test 2023-07-06 15:36:21 -04:00
Zubair Khan 9a46b9b8ac save progress 2023-07-06 14:03:32 -04:00
Zubair Khan 39d8d0319a init 2023-07-06 13:23:47 -04:00
Zachary Rice a99d89d711 fix typo (#1452) 2023-07-05 14:14:18 -05:00
dependabot[bot] d017181251 Bump cloud.google.com/go/storage from 1.30.1 to 1.31.0 (#1442)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.30.1 to 1.31.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.30.1...pubsub/v1.31.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-07-05 11:12:58 -05:00
dependabot[bot] 7f56f97522 Bump google.golang.org/api from 0.128.0 to 0.129.0 (#1441)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.128.0 to 0.129.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.128.0...v0.129.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-05 11:12:48 -05:00
Zachary Rice 8a508e6bcd Add missing keywords for sqlserver (#1449) 2023-07-05 11:12:19 -05:00
dependabot[bot] 987610d310 Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 (#1444)
Bumps google.golang.org/protobuf from 1.30.0 to 1.31.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-05 09:53:56 -05:00
dependabot[bot] 598158720c Bump cloud.google.com/go/secretmanager from 1.11.0 to 1.11.1 (#1443)
Bumps [cloud.google.com/go/secretmanager](https://github.com/googleapis/google-cloud-go) from 1.11.0 to 1.11.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.11.0...asset/v1.11.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/secretmanager
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 10:11:49 -07:00
dependabot[bot] e7db276ace Bump github.com/envoyproxy/protoc-gen-validate from 1.0.1 to 1.0.2 (#1445)
Bumps [github.com/envoyproxy/protoc-gen-validate](https://github.com/envoyproxy/protoc-gen-validate) from 1.0.1 to 1.0.2.
- [Release notes](https://github.com/envoyproxy/protoc-gen-validate/releases)
- [Changelog](https://github.com/bufbuild/protoc-gen-validate/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/envoyproxy/protoc-gen-validate/compare/v1.0.1...v1.0.2)

---
updated-dependencies:
- dependency-name: github.com/envoyproxy/protoc-gen-validate
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-03 15:33:52 -07:00
Zachary Rice 452734adc8 remove head from git diff command, rename unstaged to staged (#1439) 2023-06-29 15:33:30 -05:00
Zachary Rice 18a70b64bb Introduce trufflehog:ignore tag feature (#1433)
* init ignore

* cleanup and add test

* update readme
2023-06-29 08:45:56 -05:00
roxanne-tampus 00920984e3 added opsgenie detector (#650)
* added opsgenie detector

* update interface and import

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-06-27 16:43:25 -07:00
Zubair Khan d6375ba921 verify response body with expected keywords (#1419)
* verify response body with expected keywords

* remove debug log

* add extra test case

* migrate from ioutil to io

* close body and only check for one keyword

* cleanup
2023-06-27 11:46:15 -04:00
Miccah 8ea49de490 Don't return on okta credential failed verification (#1432) 2023-06-27 09:21:39 -05:00
Zachary Rice 4a77688097 use stringer again for now (#1430) 2023-06-26 14:33:54 -05:00
trufflesteeeve 11bff81def Use url redaction in git (#1399)
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-06-26 13:56:08 -05:00
dependabot[bot] 06f2d3a162 Bump github.com/xanzy/go-gitlab from 0.85.0 to 0.86.0 (#1425)
Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.85.0 to 0.86.0.
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](https://github.com/xanzy/go-gitlab/compare/v0.85.0...v0.86.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 11:39:20 -07:00
Zubair Khan f52946b996 Add Couchbase Detector (#1385)
* init

* add detector type

* rotate leaked credentials

* tighten up username pattern

* isolated prefixregex as overrriding new line stuff

* passwordPat working now

* add username test

* fix edge case

* cleanup

* make linter happy

* make linter happy rd 2

* skip error logging

* fix test

* add password regex helper func

* make test more robust

* cleanup PR

* remove comments

* clarify prepend rationale
2023-06-26 14:37:10 -04:00
Miccah 945c27cb82 Fix docker source to return any chunk errors (#1429) 2023-06-26 12:12:46 -05:00
Brendan Shaklovitz da5301ea1e Exit with non-zero exit code on chunk source error (#1286)
* Exit with non-zero exit code on chunk source error

* Exit with a non-zero exit code whenever we hit an error getting
  chunks. Previously the error would be logged but trufflehog would exit
  with a 0 (success) status code.

* fix gcs test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
2023-06-26 11:39:57 -05:00
dependabot[bot] 7cefea6562 Bump cloud.google.com/go/storage from 1.29.0 to 1.30.1 (#1424)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.29.0 to 1.30.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.29.0...pubsub/v1.30.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 08:44:45 -07:00
Chris Atkin 6e6895b48e Update Slack webhook error text for verification (#1427)
This updates the matched error text to determine the verified status of a Slack webhook, as this has been updated on Slack's API.
2023-06-26 08:44:17 -07:00
dependabot[bot] 146ddb351b Bump golang.org/x/sync from 0.2.0 to 0.3.0 (#1426)
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.2.0 to 0.3.0.
- [Commits](https://github.com/golang/sync/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 08:27:48 -07:00
dependabot[bot] ab84f1fbc6 Bump go.mongodb.org/mongo-driver from 1.11.6 to 1.12.0 (#1423)
Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.11.6 to 1.12.0.
- [Release notes](https://github.com/mongodb/mongo-go-driver/releases)
- [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.11.6...v1.12.0)

---
updated-dependencies:
- dependency-name: go.mongodb.org/mongo-driver
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 08:27:09 -07:00
dependabot[bot] 3ea766e8c2 Bump github.com/go-ldap/ldap/v3 from 3.4.4 to 3.4.5 (#1422)
Bumps [github.com/go-ldap/ldap/v3](https://github.com/go-ldap/ldap) from 3.4.4 to 3.4.5.
- [Release notes](https://github.com/go-ldap/ldap/releases)
- [Commits](https://github.com/go-ldap/ldap/compare/v3.4.4...v3.4.5)

---
updated-dependencies:
- dependency-name: github.com/go-ldap/ldap/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 08:25:53 -07:00
Zubair Khan cd67f6bf16 prevent www from being a key to prevent fp (#1418) 2023-06-25 11:55:11 -04:00
Dustin Decker eeefde1ec9 Ensure results are collected correctly when verification is off, and dedupe twilio (#1420) 2023-06-23 14:14:08 -07:00
Miccah f3152b6885 Implement SourceUnitUnmarshaller for all sources (#1416)
* Implement CommonSourceUnitUnmarshaller

* Add SourceUnitUnmarshaller to all sources using

All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.

* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
56 changed files with 1747 additions and 358 deletions
+1 -2
View File
@@ -10,8 +10,7 @@ RUN --mount=type=cache,target=/go/pkg/mod \
FROM alpine:3.15
RUN apk add --no-cache bash git openssh-client ca-certificates \
&& rm -rf /var/cache/apk/* && \
update-ca-certificates
&& update-ca-certificates
COPY --from=builder /build/trufflehog /usr/bin/trufflehog
COPY entrypoint.sh /etc/entrypoint.sh
RUN chmod +x /etc/entrypoint.sh
+2
View File
@@ -151,6 +151,8 @@ trufflehog docker --image trufflesecurity/secrets --only-verified
+ Unauthenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
+ It says a private key was verified, what does that mean?
+ Check out our Driftwood blog post to learn how to do this, in short we've confirmed the key can be used live for SSH or SSL [Blog post](https://trufflesecurity.com/blog/driftwood-know-if-private-keys-are-sensitive/)
+ Is there an easy way to ignore specific secrets?
+ If the scanned source [supports line numbers](https://github.com/trufflesecurity/trufflehog/blob/d6375ba92172fd830abb4247cca15e3176448c5d/pkg/engine/engine.go#L358-L365), then you can add a `trufflehog:ignore` comment on the line containing the secret to ignore that secrets.
# :newspaper: What's new in v3?
+21 -19
View File
@@ -5,8 +5,8 @@ go 1.20
replace github.com/jpillora/overseer => github.com/trufflesecurity/overseer v1.1.7-custom5
require (
cloud.google.com/go/secretmanager v1.11.0
cloud.google.com/go/storage v1.29.0
cloud.google.com/go/secretmanager v1.11.1
cloud.google.com/go/storage v1.31.0
github.com/Azure/go-autorest/autorest/azure/auth v0.5.11
github.com/TheZeroSlave/zapsentry v1.15.0
github.com/aws/aws-sdk-go v1.44.83
@@ -14,15 +14,16 @@ require (
github.com/bill-rich/go-syslog v0.0.0-20220413021637-49edb52a574c
github.com/bitfinexcom/bitfinex-api-go v0.0.0-20210608095005-9e0b26f200fb
github.com/bradleyfalzon/ghinstallation/v2 v2.4.0
github.com/couchbase/gocb/v2 v2.6.3
github.com/crewjam/rfc5424 v0.1.0
github.com/denisenkom/go-mssqldb v0.12.3
github.com/envoyproxy/protoc-gen-validate v1.0.1
github.com/envoyproxy/protoc-gen-validate v1.0.2
github.com/fatih/color v1.15.0
github.com/felixge/fgprof v0.9.3
github.com/getsentry/sentry-go v0.22.0
github.com/go-errors/errors v1.4.2
github.com/go-git/go-git/v5 v5.7.0
github.com/go-ldap/ldap/v3 v3.4.4
github.com/go-ldap/ldap/v3 v3.4.5
github.com/go-logr/logr v1.2.4
github.com/go-logr/zapr v1.2.4
github.com/go-redis/redis v6.15.9+incompatible
@@ -30,7 +31,7 @@ require (
github.com/gobwas/glob v0.2.3
github.com/golang-jwt/jwt v3.2.2+incompatible
github.com/google/go-cmp v0.5.9
github.com/google/go-containerregistry v0.15.2
github.com/google/go-containerregistry v0.14.0
github.com/google/go-github/v42 v42.0.0
github.com/googleapis/gax-go/v2 v2.11.0
github.com/hashicorp/go-retryablehttp v0.7.4
@@ -50,26 +51,26 @@ require (
github.com/sergi/go-diff v1.3.1
github.com/stretchr/testify v1.8.4
github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502
github.com/xanzy/go-gitlab v0.85.0
go.mongodb.org/mongo-driver v1.11.6
github.com/xanzy/go-gitlab v0.86.0
go.mongodb.org/mongo-driver v1.12.0
go.uber.org/zap v1.24.0
golang.org/x/crypto v0.10.0
golang.org/x/exp v0.0.0-20221018205818-5c77f4b2bbd7
golang.org/x/oauth2 v0.9.0
golang.org/x/sync v0.2.0
golang.org/x/sync v0.3.0
golang.org/x/text v0.10.0
google.golang.org/api v0.128.0
google.golang.org/protobuf v1.30.0
google.golang.org/api v0.129.0
google.golang.org/protobuf v1.31.0
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/h2non/gock.v1 v1.1.2
sigs.k8s.io/yaml v1.3.0
)
require (
cloud.google.com/go v0.110.0 // indirect
cloud.google.com/go v0.110.2 // indirect
cloud.google.com/go/compute v1.19.3 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.0.1 // indirect
cloud.google.com/go/iam v1.1.0 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest v0.11.24 // indirect
github.com/Azure/go-autorest/autorest/adal v0.9.18 // indirect
@@ -77,7 +78,7 @@ require (
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20220621081337-cb9428e4ac1e // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230518184743-7afd39499903 // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
@@ -93,6 +94,7 @@ require (
github.com/cloudflare/circl v1.3.3 // indirect
github.com/connesc/cipherio v0.2.1 // indirect
github.com/containerd/stargz-snapshotter/estargz v0.14.3 // indirect
github.com/couchbase/gocbcore/v10 v10.2.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/docker/cli v23.0.5+incompatible // indirect
@@ -116,7 +118,7 @@ require (
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd // indirect
github.com/google/s2a-go v0.1.4 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.4 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.5 // indirect
github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 // indirect
github.com/hashicorp/errwrap v1.0.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
@@ -153,25 +155,25 @@ require (
github.com/vbatts/tar-split v0.11.3 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.1 // indirect
github.com/xdg-go/stringprep v1.0.3 // indirect
github.com/xdg-go/scram v1.1.2 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
github.com/yusufpapurcu/wmi v1.2.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
go4.org v0.0.0-20200411211856-f5505b9728dd // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/mod v0.11.0 // indirect
golang.org/x/net v0.11.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.8.0 // indirect
golang.org/x/tools v0.10.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/grpc v1.55.0 // indirect
google.golang.org/grpc v1.56.1 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
+49 -41
View File
@@ -7,8 +7,8 @@ cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTj
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys=
cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY=
cloud.google.com/go v0.110.2 h1:sdFPBr6xG9/wkBbfhmUz/JmZC7X6LavQgcrVINrKiVA=
cloud.google.com/go v0.110.2/go.mod h1:k04UEeEtb6ZBRTv3dZz4CeJC3jKGxyhl0sAiVVquxiw=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/compute v1.19.3 h1:DcTwsFgGev/wV5+q8o2fzgcHOaac+DKGC91ZlvpsQds=
@@ -17,16 +17,16 @@ cloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1h
cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY=
cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/iam v1.0.1 h1:lyeCAU6jpnVNrE9zGQkTl3WgNgK/X+uWwaw0kynZJMU=
cloud.google.com/go/iam v1.0.1/go.mod h1:yR3tmSL8BcZB4bxByRv2jkSIahVmCtfKZwLYGBalRE8=
cloud.google.com/go/iam v1.1.0 h1:67gSqaPukx7O8WLLHMa0PNs3EBGd2eE4d+psbO/CO94=
cloud.google.com/go/iam v1.1.0/go.mod h1:nxdHjaKfCr7fNYx/HJMM8LgiMugmveWlkatear5gVyk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/secretmanager v1.11.0 h1:tNzxvs3fP57iTrhaS1XdEPwg4LSPKiDNgp+ZLbj0hks=
cloud.google.com/go/secretmanager v1.11.0/go.mod h1:qeQq0/jyJqrGeULu0GkRsVSPKTvf98AEqJnuEIQiJwA=
cloud.google.com/go/secretmanager v1.11.1 h1:cLTCwAjFh9fKvU6F13Y4L9vPcx9yiWPyWXE4+zkuEQs=
cloud.google.com/go/secretmanager v1.11.1/go.mod h1:znq9JlXgTNdBeQk9TBW/FnR/W4uChEKGeqQWAJ8SXFw=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.29.0 h1:6weCgzRvMg7lzuUurI4697AqIRPU1SvzHhynwpW31jI=
cloud.google.com/go/storage v1.29.0/go.mod h1:4puEjyTKnku6gfKoTfNOU/W+a9JyuVNxjpS5GBrB8h4=
cloud.google.com/go/storage v1.31.0 h1:+S3LjjEN2zZ+L5hOwj4+1OkGCsLVe0NzpXKQ1pSdTCI=
cloud.google.com/go/storage v1.31.0/go.mod h1:81ams1PrhW16L4kF7qg+4mTq7SRs5HsbDTM0bWvrwJ0=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v0.19.0/go.mod h1:h6H6c8enJmmocHUbLiiGY6sx7f9i+X3m1CHdd5c6Rdw=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.11.0/go.mod h1:HcM1YX14R7CJcghJGOYCgdezslRSVzqwLf/q+4Y2r/0=
@@ -49,8 +49,8 @@ github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+Z
github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8=
github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/Azure/go-ntlmssp v0.0.0-20220621081337-cb9428e4ac1e h1:NeAW1fUYUEWhft7pkxDf6WoUvEZJ/uOKsvtpjLnn8MU=
github.com/Azure/go-ntlmssp v0.0.0-20220621081337-cb9428e4ac1e/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 h1:mFRzDkZVAjdal+s7s0MwaRv9igoPqLRdzOLzw/8Xvq8=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
@@ -68,6 +68,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafo
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
github.com/alexbrainman/sspi v0.0.0-20210105120005-909beea2cc74 h1:Kk6a4nehpJ3UuJRqlA3JxYxBZEqCeOmATOvrbT4p9RA=
github.com/alexbrainman/sspi v0.0.0-20210105120005-909beea2cc74/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
@@ -115,6 +117,13 @@ github.com/connesc/cipherio v0.2.1 h1:FGtpTPMbKNNWByNrr9aEBtaJtXjqOzkIXNYJp6OEyc
github.com/connesc/cipherio v0.2.1/go.mod h1:ukY0MWJDFnJEbXMQtOcn2VmTpRfzcTz4OoVrWGGJZcA=
github.com/containerd/stargz-snapshotter/estargz v0.14.3 h1:OqlDCK3ZVUO6C3B/5FSkDwbkEETK84kQgEeFwDC+62k=
github.com/containerd/stargz-snapshotter/estargz v0.14.3/go.mod h1:KY//uOCIkSuNAHhJogcZtrNHdKrA99/FCCRjE3HD36o=
github.com/couchbase/gocb/v2 v2.6.3 h1:5RsMo+RRfK0mVxHLAfpBz3/tHlgXZb1WBNItLk9Ab+c=
github.com/couchbase/gocb/v2 v2.6.3/go.mod h1:yF5F6BHTZ/ZowhEuZbySbXrlI4rHd1TIhm5azOaMbJU=
github.com/couchbase/gocbcore/v10 v10.2.3 h1:PEkRSNSkKjUBXx82Ucr094+anoiCG5GleOOQZOHo6D4=
github.com/couchbase/gocbcore/v10 v10.2.3/go.mod h1:lYQIIk+tzoMcwtwU5GzPbDdqEkwkH3isI2rkSpfL0oM=
github.com/couchbaselabs/gocaves/client v0.0.0-20230307083111-cc3960c624b1/go.mod h1:AVekAZwIY2stsJOMWLAS/0uA/+qdp7pjO8EHnl61QkY=
github.com/couchbaselabs/gocaves/client v0.0.0-20230404095311-05e3ba4f0259 h1:2TXy68EGEzIMHOx9UvczR5ApVecwCfQZ0LjkmwMI6g4=
github.com/couchbaselabs/gocaves/client v0.0.0-20230404095311-05e3ba4f0259/go.mod h1:AVekAZwIY2stsJOMWLAS/0uA/+qdp7pjO8EHnl61QkY=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/crewjam/rfc5424 v0.1.0 h1:MSeXJm22oKovLzWj44AHwaItjIMUMugYGkEzfa831H8=
@@ -147,8 +156,8 @@ github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1m
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/envoyproxy/protoc-gen-validate v1.0.1 h1:kt9FtLiooDc0vbwTLhdg3dyNX1K9Qwa1EK9LcD4jVUQ=
github.com/envoyproxy/protoc-gen-validate v1.0.1/go.mod h1:0vj8bNkYbSTNS2PIyH87KZaeN4x9zpL9Qt8fQC7d+vs=
github.com/envoyproxy/protoc-gen-validate v1.0.2 h1:QkIBuU5k+x7/QXPvPPnWXWlCdaBFApVqftFV6k087DA=
github.com/envoyproxy/protoc-gen-validate v1.0.2/go.mod h1:GpiZQP3dDbg4JouG/NNS7QWXpgx6x8QiMKdmN72jogE=
github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs=
github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw=
github.com/felixge/fgprof v0.9.3 h1:VvyZxILNuCiUCSXtPtYmmtGvb65nqXh2QFWc0Wpf2/g=
@@ -173,8 +182,8 @@ github.com/go-git/go-git/v5 v5.7.0 h1:t9AudWVLmqzlo+4bqdf7GY+46SUuRsx59SboFxkq2a
github.com/go-git/go-git/v5 v5.7.0/go.mod h1:coJHKEOk5kUClpsNlXrUvPrDxY3w3gjHvhcZd8Fodw8=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-ldap/ldap/v3 v3.4.4 h1:qPjipEpt+qDa6SI/h1fzuGWoRUY+qqQ9sOZq67/PYUs=
github.com/go-ldap/ldap/v3 v3.4.4/go.mod h1:fe1MsuN5eJJ1FeLT/LEBVdWfNWKh459R7aXgXtJC+aI=
github.com/go-ldap/ldap/v3 v3.4.5 h1:ekEKmaDrpvR2yf5Nc/DClsGG9lAmdDixe44mLzlW5r8=
github.com/go-ldap/ldap/v3 v3.4.5/go.mod h1:bMGIq3AGbytbaMwf8wdv5Phdxz0FWHTIYMSzyrYgnQs=
github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ=
github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/zapr v1.2.4 h1:QHVo+6stLbfJmYGkQ7uGHUCu5hnAFAj6mDe6Ea0SeOo=
@@ -246,8 +255,8 @@ github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-containerregistry v0.15.2 h1:MMkSh+tjSdnmJZO7ljvEqV1DjfekB6VUEAZgy3a+TQE=
github.com/google/go-containerregistry v0.15.2/go.mod h1:wWK+LnOv4jXMM23IT/F1wdYftGWGr47Is8CG+pmHK1Q=
github.com/google/go-containerregistry v0.14.0 h1:z58vMqHxuwvAsVwvKEkmVBz2TlgBgH5k6koEXBtlYkw=
github.com/google/go-containerregistry v0.14.0/go.mod h1:aiJ2fp/SXvkWgmYHioXnbMdlgB8eXiiYOY55gfN91Wk=
github.com/google/go-github/v42 v42.0.0 h1:YNT0FwjPrEysRkLIiKuEfSvBPCGKphW5aS5PxwaoLec=
github.com/google/go-github/v42 v42.0.0/go.mod h1:jgg/jvyI0YlDOM1/ps6XYh04HNQ3vKf0CVko62/EhRg=
github.com/google/go-github/v52 v52.0.0 h1:uyGWOY+jMQ8GVGSX8dkSwCzlehU3WfdxQ7GweO/JP7M=
@@ -268,8 +277,8 @@ github.com/google/s2a-go v0.1.4/go.mod h1:Ej+mSEMGRnqRzjc7VtF+jdBwYG5fuJfiZ8ELkj
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.2.4 h1:uGy6JWR/uMIILU8wbf+OkstIrNiMjGpEIyhx8f6W7s4=
github.com/googleapis/enterprise-certificate-proxy v0.2.4/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k=
github.com/googleapis/enterprise-certificate-proxy v0.2.5 h1:UR4rDjcgpgEnqpIEvkiqTYKBCKLNmlge2eVjoZfySzM=
github.com/googleapis/enterprise-certificate-proxy v0.2.5/go.mod h1:RxW0N9901Cko1VOCW3SXCpWP+mlIEkk2tP7jnHy9a3w=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gax-go/v2 v2.11.0 h1:9V9PWXEsWnPpQhu/PeQIkS4eGzMlTLGgt80cUUI8Ki4=
@@ -419,6 +428,7 @@ github.com/smartystreets/gunit v1.1.3 h1:32x+htJCu3aMswhPw3teoJ+PnWPONqdNgaGs6Qt
github.com/smartystreets/gunit v1.1.3/go.mod h1:EH5qMBab2UclzXUcpR8b93eHsIlp9u+pDQIRp5DZNzQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
@@ -427,17 +437,15 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502 h1:34icjjmqJ2HPjrSuJYEkdZ+0ItmGQAQ75cRHIiftIyE=
github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502/go.mod h1:p9lPsd+cx33L3H9nNoecRRxPssFKUwwI50I3pZ0yT+8=
github.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw=
github.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY=
github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/trufflesecurity/overseer v1.1.7-custom5 h1:xu+Fg6fkSRifUPzUCl7N8HmobJ6WGOkIApGnM7mJS6w=
github.com/trufflesecurity/overseer v1.1.7-custom5/go.mod h1:nT9w37AiO1Nop2VhVhNfzAFaPjthvxgpDV3XKsxYkcI=
github.com/ulikunitz/xz v0.5.6/go.mod h1:2bypXElzHzzJZwzH67Y6wb67pO62Rzfn7BSiF4ABRW8=
@@ -446,16 +454,16 @@ github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0o
github.com/urfave/cli v1.22.12/go.mod h1:sSBEIC79qR6OvcmsD4U3KABeOTxDqQtdDnaFuUN30b8=
github.com/vbatts/tar-split v0.11.3 h1:hLFqsOLQ1SsppQNTMpkpPXClLDfC2A3Zgy9OUU+RVck=
github.com/vbatts/tar-split v0.11.3/go.mod h1:9QlHN18E+fEH7RdG+QAJJcuya3rqT7eXSTY7wGrAokY=
github.com/xanzy/go-gitlab v0.85.0 h1:E/wjnsd/mM5kV6O9y5+i6zxjx+wfAwa97sgcT1ETNwk=
github.com/xanzy/go-gitlab v0.85.0/go.mod h1:5ryv+MnpZStBH8I/77HuQBsMbBGANtVpLWC15qOjWAw=
github.com/xanzy/go-gitlab v0.86.0 h1:jR8V9cK9jXRQDb46KOB20NCF3ksY09luaG0IfXE6p7w=
github.com/xanzy/go-gitlab v0.86.0/go.mod h1:5ryv+MnpZStBH8I/77HuQBsMbBGANtVpLWC15qOjWAw=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.1.1 h1:VOMT+81stJgXW3CpHyqHN3AXDYIMsx56mEFrB37Mb/E=
github.com/xdg-go/scram v1.1.1/go.mod h1:RaEWvsqvNKKvBPvcKeFjrG2cJqOkHTiyTpzz23ni57g=
github.com/xdg-go/stringprep v1.0.3 h1:kdwGpVNwPFtjs98xCGkHjQtGKh86rDcRZN17QEMCOIs=
github.com/xdg-go/stringprep v1.0.3/go.mod h1:W3f5j4i+9rC0kuIEJL0ky1VpHXQU3ocBgklLGvcBnW8=
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d h1:splanxYIlg+5LfHAM6xpdFEAYOk8iySO56hMFq6uLyA=
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -463,8 +471,8 @@ github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.mongodb.org/mongo-driver v1.11.6 h1:XM7G6PjiGAO5betLF13BIa5TlLUUE3uJ/2Ox3Lz1K+o=
go.mongodb.org/mongo-driver v1.11.6/go.mod h1:G9TgswdsWjX4tmDA5zfs2+6AEPpYJwqblyjsfuh8oXY=
go.mongodb.org/mongo-driver v1.12.0 h1:aPx33jmn/rQuJXPQLZQ8NtfPQG8CaqgLThFtqRb0PiE=
go.mongodb.org/mongo-driver v1.12.0/go.mod h1:AZkxhPnFJUoH7kZlFkVKucV20K387miPfm7oimrSmK0=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
@@ -528,8 +536,8 @@ golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk=
golang.org/x/mod v0.10.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU=
golang.org/x/mod v0.11.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -578,8 +586,8 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI=
golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -674,8 +682,8 @@ golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4f
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.8.0 h1:vSDcovVPld282ceKgDimkRSC8kpaH1dgyc9UMzlt84Y=
golang.org/x/tools v0.8.0/go.mod h1:JxBZ99ISMI5ViVkT1tr6tdNmXeTrcpVSD3vZ1RsRdN4=
golang.org/x/tools v0.10.0 h1:tvDr/iQoUqNdohiYm0LmmKcBk+q86lb9EprIUFhHHGg=
golang.org/x/tools v0.10.0/go.mod h1:UJwyiVBsOA2uwvK/e5OY3GTpDUJriEd+/YlqAwLPmyM=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -690,8 +698,8 @@ google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.128.0 h1:RjPESny5CnQRn9V6siglged+DZCgfu9l6mO9dkX9VOg=
google.golang.org/api v0.128.0/go.mod h1:Y611qgqaE92On/7g65MQgxYul3c0rEB894kniWLY750=
google.golang.org/api v0.129.0 h1:2XbdjjNfFPXQyufzQVwPf1RRnHH8Den2pfNE2jw7L8w=
google.golang.org/api v0.129.0/go.mod h1:dFjiXlanKwWE3612X97llhsoI36FAoIiRj3aTl5b/zE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -732,8 +740,8 @@ google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTp
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
google.golang.org/grpc v1.55.0 h1:3Oj82/tFSCeUrRTg/5E/7d/W5A1tj6Ky1ABAuZuv5ag=
google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8=
google.golang.org/grpc v1.56.1 h1:z0dNfjIl0VpaZ9iSVjA6daGatAYwPGstTjt5vkRMFkQ=
google.golang.org/grpc v1.56.1/go.mod h1:I9bI3vqKfayGqPUAwGdOSu7kt6oIJLixfffKrpXqQ9s=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -746,8 +754,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+1 -1
View File
@@ -476,7 +476,7 @@ func run(state overseer.State) {
}
}
// asynchronously wait for scanning to finish and cleanup
go e.Finish(ctx)
go e.Finish(ctx, logFatal)
if !*jsonLegacy && !*jsonOut {
fmt.Fprintf(os.Stderr, "🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷\n\n")
+33 -1
View File
@@ -2,6 +2,7 @@ package common
import (
"fmt"
"regexp"
"strconv"
"strings"
)
@@ -15,7 +16,11 @@ const RegexPattern = "0-9a-z"
const AlphaNumPattern = "0-9a-zA-Z"
const HexPattern = "0-9a-f"
//Custom Regex functions
type RegexState struct {
compiledRegex *regexp.Regexp
}
// Custom Regex functions
func BuildRegex(pattern string, specialChar string, length int) string {
return fmt.Sprintf(`\b([%s%s]{%s})\b`, pattern, specialChar, strconv.Itoa(length))
}
@@ -37,3 +42,30 @@ func RangeValidation(rangeInput string) bool {
func ToUpperCase(input string) string {
return strings.ToUpper(input)
}
func (r RegexState) Matches(data []byte) []string {
matches := r.compiledRegex.FindAllStringSubmatch(string(data), -1)
res := make([]string, 0, len(matches))
// trim off spaces and different quote types ('").
for i := range matches {
res = append(res, strings.Trim(matches[i][1], `"' )`))
}
return res
}
// UsernameRegexCheck constructs an username usernameRegex pattern from a given pattern of excluded characters.
func UsernameRegexCheck(pattern string) RegexState {
raw := fmt.Sprintf(`(?im)(?:user|usr)\S{0,40}?[:=\s]{1,3}[ '"=]{0,1}([^:%+v]{4,40})\b`, pattern)
return RegexState{regexp.MustCompile(raw)}
}
// PasswordRegexCheck constructs an username usernameRegex pattern from a given pattern of excluded characters.
func PasswordRegexCheck(pattern string) RegexState {
raw := fmt.Sprintf(`(?im)(?:pass|password)\S{0,40}?[:=\s]{1,3}[ '"=]{0,1}([^:%+v]{4,40})`, pattern)
return RegexState{regexp.MustCompile(raw)}
}
+59
View File
@@ -0,0 +1,59 @@
package common
import (
"github.com/stretchr/testify/assert"
"regexp"
"testing"
)
const (
usernamePattern = `?()/\+=\s\n`
passwordPattern = `^<>;.*&|£\n\s`
usernameRegex = `(?im)(?:user|usr)\S{0,40}?[:=\s]{1,3}[ '"=]{0,1}([^:?()/\+=\s\n]{4,40})\b`
passwordRegex = `(?im)(?:pass|password)\S{0,40}?[:=\s]{1,3}[ '"=]{0,1}([^:^<>;.*&|£\n\s]{4,40})`
)
func TestUsernameRegexCheck(t *testing.T) {
usernameRegexPat := UsernameRegexCheck(usernamePattern)
expectedRegexPattern := regexp.MustCompile(usernameRegex)
if usernameRegexPat.compiledRegex.String() != expectedRegexPattern.String() {
t.Errorf("\n got %v \n want %v", usernameRegexPat.compiledRegex, expectedRegexPattern)
}
testString := `username = "johnsmith123"
username='johnsmith123'
username:="johnsmith123"
username = johnsmith123
username=johnsmith123`
expectedStr := []string{"johnsmith123", "johnsmith123", "johnsmith123", "johnsmith123", "johnsmith123"}
usernameRegexMatches := usernameRegexPat.Matches([]byte(testString))
assert.Exactly(t, usernameRegexMatches, expectedStr)
}
func TestPasswordRegexCheck(t *testing.T) {
passwordRegexPat := PasswordRegexCheck(passwordPattern)
expectedRegexPattern := regexp.MustCompile(passwordRegex)
assert.Equal(t, passwordRegexPat.compiledRegex, expectedRegexPattern)
testString := `password = "johnsmith123$!"
password='johnsmith123$!'
password:="johnsmith123$!"
password = johnsmith123$!
password=johnsmith123$!
PasswordAuthenticator(username, "johnsmith123$!")`
expectedStr := []string{"johnsmith123$!", "johnsmith123$!", "johnsmith123$!", "johnsmith123$!", "johnsmith123$!",
"johnsmith123$!"}
passwordRegexMatches := passwordRegexPat.Matches([]byte(testString))
assert.Exactly(t, passwordRegexMatches, expectedStr)
}
+19
View File
@@ -1,5 +1,11 @@
package common
import (
"bufio"
"io"
"strings"
)
func AddStringSliceItem(item string, slice *[]string) {
for _, i := range *slice {
if i == item {
@@ -17,3 +23,16 @@ func RemoveStringSliceItem(item string, slice *[]string) {
}
}
}
func ResponseContainsSubstring(reader io.ReadCloser, target string) (bool, error) {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
if strings.Contains(scanner.Text(), target) {
return true, nil
}
}
if err := scanner.Err(); err != nil {
return false, err
}
return false, nil
}
+59
View File
@@ -1,7 +1,9 @@
package common
import (
"io"
"reflect"
"strings"
"testing"
)
@@ -74,3 +76,60 @@ func TestRemoveItem(t *testing.T) {
}
}
}
// Test ParseResponseForKeywords with a reader that contains the keyword and a reader that doesn't.
func TestParseResponseForKeywords(t *testing.T) {
testCases := []struct {
name string
input string
keyword string
expected bool
}{
{
name: "Should find keyword",
input: "ey: abc",
keyword: "ey",
expected: true,
},
{
name: "Should not find keyword",
input: "fake response",
keyword: "ey",
expected: false,
},
{
name: "Empty string",
input: "",
keyword: "ey",
expected: false,
},
{
name: "Keyword at end",
input: "abc ey",
keyword: "ey",
expected: true,
},
{
name: "Keyword at start",
input: "ey abc",
keyword: "ey",
expected: true,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
testReader := strings.NewReader(tc.input)
testReadCloser := io.NopCloser(testReader)
found, err := ResponseContainsSubstring(testReadCloser, tc.keyword)
if err != nil {
t.Errorf("Error: %v", err)
}
if found != tc.expected {
t.Errorf("Expected %v, got %v", tc.expected, found)
}
})
}
}
+152
View File
@@ -0,0 +1,152 @@
package couchbase
import (
"context"
"fmt"
"log"
"regexp"
"strings"
"time"
"unicode"
"github.com/couchbase/gocb/v2"
"github.com/trufflesecurity/trufflehog/v3/pkg/common"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
)
type Scanner struct{}
// Ensure the Scanner satisfies the interface at compile time.
var _ detectors.Detector = (*Scanner)(nil)
var (
// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
connectionStringPat = regexp.MustCompile(`\bcb\.[a-z0-9]+\.cloud\.couchbase\.com\b`)
usernamePat = `?()/\+=\s\n`
passwordPat = `^<>;.*&|£\n\s`
//passwordPat = regexp.MustCompile(`(?i)(?:pass|pwd)(?:.|[\n\r]){0,15}(\b[^<>;.*&|£\n\s]{8,100}$)`)
// passwordPat = regexp.MustCompile(`(?im)(?:pass|pwd)\S{0,40}?[:=\s]{1,3}[ '"=]{0,1}([^:?()/\+=\s\n]{4,40})\b`)
)
func meetsCouchbasePasswordRequirements(password string) (string, bool) {
var hasLower, hasUpper, hasNumber, hasSpecialChar bool
for _, char := range password {
switch {
case unicode.IsLower(char):
hasLower = true
case unicode.IsUpper(char):
hasUpper = true
case unicode.IsNumber(char):
hasNumber = true
case unicode.IsPunct(char) || unicode.IsSymbol(char):
hasSpecialChar = true
}
if hasLower && hasUpper && hasNumber && hasSpecialChar {
return password, true
}
}
return "", false
}
// Keywords are used for efficiently pre-filtering chunks.
// Use identifiers in the secret preferably, or the provider name.
func (s Scanner) Keywords() []string {
return []string{"couchbase://", "couchbases://"}
}
// FromData will find and optionally verify Couchbase secrets in a given set of bytes.
func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
dataStr := string(data)
connectionStringMatches := connectionStringPat.FindAllStringSubmatch(dataStr, -1)
// prepend 'couchbases://' to the connection string as the connection
// string format is couchbases://cb.stuff.cloud.couchbase.com but the
// cb.stuff.cloud.couchbase.com may be separated from the couchbases:// in codebases.
for i, connectionStringMatch := range connectionStringMatches {
connectionStringMatches[i][0] = "couchbases://" + connectionStringMatch[0]
}
usernameRegexState := common.UsernameRegexCheck(usernamePat)
usernameMatches := usernameRegexState.Matches(data)
passwordRegexState := common.PasswordRegexCheck(passwordPat)
passwordMatches := passwordRegexState.Matches(data)
for _, connectionStringMatch := range connectionStringMatches {
resConnectionStringMatch := strings.TrimSpace(connectionStringMatch[0])
for _, resUsernameMatch := range usernameMatches {
for _, resPasswordMatch := range passwordMatches {
_, metPasswordRequirements := meetsCouchbasePasswordRequirements(resPasswordMatch)
if !metPasswordRequirements {
continue
}
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_Couchbase,
Raw: []byte(fmt.Sprintf("%s:%s@%s", resUsernameMatch, resPasswordMatch, resConnectionStringMatch)),
}
if verify {
options := gocb.ClusterOptions{
Authenticator: gocb.PasswordAuthenticator{
Username: resUsernameMatch,
Password: resPasswordMatch,
},
}
// Sets a pre-configured profile called "wan-development" to help avoid latency issues
// when accessing Capella from a different Wide Area Network
// or Availability Zone (e.g. your laptop).
if err := options.ApplyProfile(gocb.ClusterConfigProfileWanDevelopment); err != nil {
log.Fatal("apply profile err", err)
}
// Initialize the Connection
cluster, err := gocb.Connect(resConnectionStringMatch, options)
if err != nil {
continue
}
// We'll ping the KV nodes in our cluster.
pings, err := cluster.Ping(&gocb.PingOptions{
Timeout: time.Second * 5,
})
if err != nil {
continue
}
for _, ping := range pings.Services {
for _, pingEndpoint := range ping {
if pingEndpoint.State == gocb.PingStateOk {
s1.Verified = true
break
} else {
// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
if detectors.IsKnownFalsePositive(resPasswordMatch, detectors.DefaultFalsePositives, true) {
continue
}
}
}
}
}
results = append(results, s1)
}
}
}
return results, nil
}
func (s Scanner) Type() detectorspb.DetectorType {
return detectorspb.DetectorType_Couchbase
}
+122
View File
@@ -0,0 +1,122 @@
//go:build detectors
// +build detectors
package couchbase
import (
"context"
"fmt"
"testing"
"time"
"github.com/kylelemons/godebug/pretty"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
"github.com/trufflesecurity/trufflehog/v3/pkg/common"
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
)
func TestCouchbase_FromChunk(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors5")
if err != nil {
t.Fatalf("could not get test secrets from GCP: %s", err)
}
endpoint := testSecrets.MustGetField("COUCHBASE_ENDPOINT")
username := testSecrets.MustGetField("COUCHBASE_USERNAME")
password := testSecrets.MustGetField("COUCHBASE_PASSWORD")
inactiveSecret := testSecrets.MustGetField("COUCHBASE_INACTIVE_PASSWORD")
type args struct {
ctx context.Context
data []byte
verify bool
}
tests := []struct {
name string
s Scanner
args args
want []detectors.Result
wantErr bool
}{
{
name: "found, verified",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("db uri: %s \n username = %s \n password = %s", endpoint, username, password)),
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_Couchbase,
Verified: true,
},
},
wantErr: false,
},
{
name: "found, unverified",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("db uri: %s \n username = %s \n password = %s", endpoint, username, inactiveSecret)), // the secret would satisfy the regex but not pass validation
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_Couchbase,
Verified: false,
},
},
wantErr: false,
},
{
name: "not found",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte("You cannot find the secret within"),
verify: true,
},
want: nil,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := Scanner{}
got, err := s.FromData(tt.args.ctx, tt.args.verify, tt.args.data)
if (err != nil) != tt.wantErr {
t.Errorf("Couchbase.FromData() error = %v, wantErr %v", err, tt.wantErr)
return
}
for i := range got {
if len(got[i].Raw) == 0 {
t.Fatalf("no raw secret present: \n %+v", got[i])
}
got[i].Raw = nil
}
if diff := pretty.Compare(got, tt.want); diff != "" {
t.Errorf("Couchbase.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
}
})
}
}
func BenchmarkFromData(benchmark *testing.B) {
ctx := context.Background()
s := Scanner{}
for name, data := range detectors.MustGetBenchmarkData() {
benchmark.Run(name, func(b *testing.B) {
for n := 0; n < b.N; n++ {
_, err := s.FromData(ctx, false, data)
if err != nil {
b.Fatal(err)
}
}
})
}
}
+3 -15
View File
@@ -3,10 +3,8 @@ package docusign
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"github.com/go-errors/errors"
"io"
"net/http"
"regexp"
"strings"
@@ -83,25 +81,15 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
return nil, errors.WrapPrefix(err, "Error making request", 0)
}
// Read the response body
body, err := io.ReadAll(res.Body)
if err != nil {
return nil, errors.WrapPrefix(err, "Error reading response body", 0)
}
// Close the response body
verifiedBodyResponse, err := common.ResponseContainsSubstring(res.Body, "ey")
res.Body.Close()
// Parse the response body into a Response struct
var parsedResponse Response
err = json.Unmarshal(body, &parsedResponse)
if err != nil {
return nil, errors.WrapPrefix(err, "Error parsing response", 0)
return nil, err
}
if err == nil {
if res.StatusCode >= 200 && res.StatusCode < 300 && strings.HasPrefix(parsedResponse.AccessToken, "ey") {
if res.StatusCode >= 200 && res.StatusCode < 300 && verifiedBodyResponse {
s1.Verified = true
} else {
// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
+4 -4
View File
@@ -6,7 +6,7 @@ import (
"unicode"
)
var DefaultFalsePositives = []FalsePositive{"example", "xxxxxx", "aaaaaa", "abcde", "00000", "sample"}
var DefaultFalsePositives = []FalsePositive{"example", "xxxxxx", "aaaaaa", "abcde", "00000", "sample", "www"}
type FalsePositive string
@@ -31,9 +31,9 @@ var FalsePositiveWordlists = Wordlists{
programmingBookWords: bytesToCleanWordList(programmingBookWords),
}
//IsKnownFalsePositives will not return a valid secret finding if any of the disqualifying conditions are met
//Currently that includes: No number, english word in key, or matches common example pattens.
//Only the secret key material should be passed into this function
// IsKnownFalsePositives will not return a valid secret finding if any of the disqualifying conditions are met
// Currently that includes: No number, english word in key, or matches common example pattens.
// Only the secret key material should be passed into this function
func IsKnownFalsePositive(match string, falsePositives []FalsePositive, wordCheck bool) bool {
for _, fp := range falsePositives {
+1 -1
View File
@@ -63,7 +63,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
resp, err := common.SaneHttpClient().Do(req)
if err != nil {
return results, err
continue
}
defer resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
@@ -1,7 +1,8 @@
package image4
package opsgenie
import (
"context"
"fmt"
"net/http"
"regexp"
"strings"
@@ -20,16 +21,20 @@ var (
client = common.SaneHttpClient()
// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"image4"}) + `\b([0-9a-zA-Z]{22}[0-9a-zA-Z\=]{2})`)
keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"opsgenie"}) + `\b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b`)
)
func (s Scanner) Type() detectorspb.DetectorType {
return detectorspb.DetectorType_Opsgenie
}
// Keywords are used for efficiently pre-filtering chunks.
// Use identifiers in the secret preferably, or the provider name.
func (s Scanner) Keywords() []string {
return []string{"image4"}
return []string{"opsgenie"}
}
// FromData will find and optionally verify Image4 secrets in a given set of bytes.
// FromData will find and optionally verify Opsgenie secrets in a given set of bytes.
func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
dataStr := string(data)
@@ -42,16 +47,16 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
resMatch := strings.TrimSpace(match[1])
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_Image4,
DetectorType: detectorspb.DetectorType_Opsgenie,
Raw: []byte(resMatch),
}
if verify {
req, err := http.NewRequestWithContext(ctx, "GET", "https://api.image4.io/v1.0/subscription", nil)
req, err := http.NewRequestWithContext(ctx, "GET", "https://api.opsgenie.com/v2/users", nil)
if err != nil {
continue
}
req.SetBasicAuth(resMatch, "")
req.Header.Add("Authorization", fmt.Sprintf("GenieKey %s", resMatch))
res, err := client.Do(req)
if err == nil {
defer res.Body.Close()
@@ -69,9 +74,5 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
results = append(results, s1)
}
return results, nil
}
func (s Scanner) Type() detectorspb.DetectorType {
return detectorspb.DetectorType_Image4
return detectors.CleanResults(results), nil
}
@@ -1,7 +1,4 @@
//go:build detectors
// +build detectors
package image4
package opsgenie
import (
"context"
@@ -16,15 +13,15 @@ import (
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
)
func TestImage4_FromChunk(t *testing.T) {
func TestOpsgenie_FromChunk(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors3")
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors4")
if err != nil {
t.Fatalf("could not get test secrets from GCP: %s", err)
}
secret := testSecrets.MustGetField("IMAGE4")
inactiveSecret := testSecrets.MustGetField("IMAGE4_INACTIVE")
secret := testSecrets.MustGetField("OPSGENIE")
inactiveSecret := testSecrets.MustGetField("OPSGENIE_INACTIVE")
type args struct {
ctx context.Context
@@ -43,12 +40,12 @@ func TestImage4_FromChunk(t *testing.T) {
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("You can find a image4 secret %s within", secret)),
data: []byte(fmt.Sprintf("You can find a opsgenie secret %s within", secret)),
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_Image4,
DetectorType: detectorspb.DetectorType_Opsgenie,
Verified: true,
},
},
@@ -59,12 +56,12 @@ func TestImage4_FromChunk(t *testing.T) {
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("You can find a image4 secret %s within but not valid", inactiveSecret)), // the secret would satisfy the regex but not pass validation
data: []byte(fmt.Sprintf("You can find a opsgenie secret %s within but not valid", inactiveSecret)), // the secret would satisfy the regex but not pass validation
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_Image4,
DetectorType: detectorspb.DetectorType_Opsgenie,
Verified: false,
},
},
@@ -87,7 +84,7 @@ func TestImage4_FromChunk(t *testing.T) {
s := Scanner{}
got, err := s.FromData(tt.args.ctx, tt.args.verify, tt.args.data)
if (err != nil) != tt.wantErr {
t.Errorf("Image4.FromData() error = %v, wantErr %v", err, tt.wantErr)
t.Errorf("Opsgenie.FromData() error = %v, wantErr %v", err, tt.wantErr)
return
}
for i := range got {
@@ -97,7 +94,7 @@ func TestImage4_FromChunk(t *testing.T) {
got[i].Raw = nil
}
if diff := pretty.Compare(got, tt.want); diff != "" {
t.Errorf("Image4.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
t.Errorf("Opsgenie.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
}
})
}
+16 -21
View File
@@ -38,19 +38,18 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
keyMatches := keyPat.FindAllString(dataStr, -1)
for _, key := range keyMatches {
secMatches := secretPat.FindAllString(dataStr, -1)
if verify {
secMatches := secretPat.FindAllString(dataStr, -1)
for _, secret := range secMatches {
for _, secret := range secMatches {
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_RazorPay,
Raw: []byte(key),
RawV2: []byte(key + secret),
Redacted: key,
}
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_RazorPay,
Raw: []byte(key),
RawV2: []byte(key + secret),
Redacted: key,
}
if verify {
req, err := http.NewRequest("GET", "https://api.razorpay.com/v1/items?count=1", nil)
if err != nil {
continue
@@ -66,25 +65,21 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
if res.StatusCode >= 200 && res.StatusCode < 300 {
if json.Valid(bodyBytes) {
s1.Verified = true
} else {
s1.Verified = false
}
} else {
// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
if detectors.IsKnownFalsePositive(key, detectors.DefaultFalsePositives, true) {
continue
}
}
}
results = append(results, s1)
}
if !s1.Verified && detectors.IsKnownFalsePositive(key, detectors.DefaultFalsePositives, true) {
continue
}
results = append(results, s1)
}
}
results = detectors.CleanResults(results)
return
return detectors.CleanResults(results), nil
}
func (s Scanner) Type() detectorspb.DetectorType {
+21 -11
View File
@@ -2,7 +2,7 @@ package shortcut
import (
"context"
"io"
"github.com/go-errors/errors"
"net/http"
"regexp"
"strings"
@@ -21,7 +21,7 @@ var (
client = common.SaneHttpClient()
// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"shortcut"}) + `\b([0-9a-f-]{36})\b`)
keyPat = regexp.MustCompile(detectors.PrefixRegex([]string{"shortcut"}) + `\b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b`)
)
// Keywords are used for efficiently pre-filtering chunks.
@@ -48,23 +48,33 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
}
if verify {
req, err := http.NewRequestWithContext(ctx, "GET", "https://api.app.shortcut.com/api/v3/projects", nil)
req, err := http.NewRequestWithContext(ctx, "GET", "https://api.app.shortcut.com/api/v3/member", nil)
if err != nil {
continue
}
req.Header.Add("Content-Type", "application/json")
req.Header.Add("Shortcut-Token", resMatch)
res, err := client.Do(req)
if err == nil {
defer res.Body.Close()
bodyBytes, err := io.ReadAll(res.Body)
if err != nil {
continue
}
body := string(bodyBytes)
if strings.Contains(body, "app_url") {
if err != nil {
return nil, errors.WrapPrefix(err, "Error making request", 0)
}
verifiedBodyResponse, err := common.ResponseContainsSubstring(res.Body, "name")
res.Body.Close()
if err != nil {
return nil, err
}
if err == nil {
if res.StatusCode >= 200 && res.StatusCode < 300 && verifiedBodyResponse {
s1.Verified = true
} else {
// This function will check false positives for common test words, but also it will make sure the key appears 'random' enough to be a real key.
if detectors.IsKnownFalsePositive(resMatch, detectors.DefaultFalsePositives, true) {
continue
}
}
}
}
+1 -1
View File
@@ -19,7 +19,7 @@ import (
func TestShortcut_FromChunk(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors2")
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors5")
if err != nil {
t.Fatalf("could not get test secrets from GCP: %s", err)
}
+1 -1
View File
@@ -62,7 +62,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
continue
}
body := string(bodyBytes)
if (res.StatusCode >= 200 && res.StatusCode < 300) || (res.StatusCode == 400 && strings.Contains(body, "no_text")) {
if (res.StatusCode >= 200 && res.StatusCode < 300) || (res.StatusCode == 400 && (strings.Contains(body, "no_text") || strings.Contains(body, "missing_text"))) {
s1.Verified = true
}
}
+1 -1
View File
@@ -24,7 +24,7 @@ var (
// Keywords are used for efficiently pre-filtering chunks.
// Use identifiers in the secret preferably, or the provider name.
func (s Scanner) Keywords() []string {
return []string{"sql", "database", "Data Source"}
return []string{"sql", "database", "Data Source", "Server=", "Network address="}
}
// FromData will find and optionally verify SpotifyKey secrets in a given set of bytes.
@@ -0,0 +1,309 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.20.0
// source: api.proto
package scannerpb
import (
context "context"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
status "google.golang.org/grpc/status"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
type HeartbeatRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Status string `protobuf:"bytes,1,opt,name=status,proto3" json:"status,omitempty"`
Reason string `protobuf:"bytes,2,opt,name=reason,proto3" json:"reason,omitempty"`
Version string `protobuf:"bytes,3,opt,name=version,proto3" json:"version,omitempty"`
}
func (x *HeartbeatRequest) Reset() {
*x = HeartbeatRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_api_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *HeartbeatRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*HeartbeatRequest) ProtoMessage() {}
func (x *HeartbeatRequest) ProtoReflect() protoreflect.Message {
mi := &file_api_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use HeartbeatRequest.ProtoReflect.Descriptor instead.
func (*HeartbeatRequest) Descriptor() ([]byte, []int) {
return file_api_proto_rawDescGZIP(), []int{0}
}
func (x *HeartbeatRequest) GetStatus() string {
if x != nil {
return x.Status
}
return ""
}
func (x *HeartbeatRequest) GetReason() string {
if x != nil {
return x.Reason
}
return ""
}
func (x *HeartbeatRequest) GetVersion() string {
if x != nil {
return x.Version
}
return ""
}
type HeartbeatResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
}
func (x *HeartbeatResponse) Reset() {
*x = HeartbeatResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_api_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *HeartbeatResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*HeartbeatResponse) ProtoMessage() {}
func (x *HeartbeatResponse) ProtoReflect() protoreflect.Message {
mi := &file_api_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use HeartbeatResponse.ProtoReflect.Descriptor instead.
func (*HeartbeatResponse) Descriptor() ([]byte, []int) {
return file_api_proto_rawDescGZIP(), []int{1}
}
var File_api_proto protoreflect.FileDescriptor
var file_api_proto_rawDesc = []byte{
0x0a, 0x09, 0x61, 0x70, 0x69, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x03, 0x61, 0x70, 0x69,
0x22, 0x5c, 0x0a, 0x10, 0x48, 0x65, 0x61, 0x72, 0x74, 0x62, 0x65, 0x61, 0x74, 0x52, 0x65, 0x71,
0x75, 0x65, 0x73, 0x74, 0x12, 0x16, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x01,
0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x16, 0x0a, 0x06,
0x72, 0x65, 0x61, 0x73, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x72, 0x65,
0x61, 0x73, 0x6f, 0x6e, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18,
0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x13,
0x0a, 0x11, 0x48, 0x65, 0x61, 0x72, 0x74, 0x62, 0x65, 0x61, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x32, 0x4a, 0x0a, 0x0c, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76,
0x69, 0x63, 0x65, 0x12, 0x3a, 0x0a, 0x09, 0x48, 0x65, 0x61, 0x72, 0x74, 0x62, 0x65, 0x61, 0x74,
0x12, 0x15, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x48, 0x65, 0x61, 0x72, 0x74, 0x62, 0x65, 0x61, 0x74,
0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x16, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x48, 0x65,
0x61, 0x72, 0x74, 0x62, 0x65, 0x61, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42,
0x5e, 0x5a, 0x5c, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x74, 0x72,
0x75, 0x66, 0x66, 0x6c, 0x65, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x2f, 0x74, 0x72,
0x75, 0x66, 0x66, 0x6c, 0x65, 0x68, 0x6f, 0x67, 0x2f, 0x76, 0x33, 0x2f, 0x70, 0x6b, 0x67, 0x2f,
0x64, 0x65, 0x74, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73, 0x2f, 0x74, 0x72, 0x75, 0x66, 0x66, 0x6c,
0x65, 0x68, 0x6f, 0x67, 0x65, 0x6e, 0x74, 0x65, 0x72, 0x70, 0x72, 0x69, 0x73, 0x65, 0x73, 0x63,
0x61, 0x6e, 0x6e, 0x65, 0x72, 0x2f, 0x73, 0x63, 0x61, 0x6e, 0x6e, 0x65, 0x72, 0x70, 0x62, 0x62,
0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_api_proto_rawDescOnce sync.Once
file_api_proto_rawDescData = file_api_proto_rawDesc
)
func file_api_proto_rawDescGZIP() []byte {
file_api_proto_rawDescOnce.Do(func() {
file_api_proto_rawDescData = protoimpl.X.CompressGZIP(file_api_proto_rawDescData)
})
return file_api_proto_rawDescData
}
var file_api_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_api_proto_goTypes = []interface{}{
(*HeartbeatRequest)(nil), // 0: api.HeartbeatRequest
(*HeartbeatResponse)(nil), // 1: api.HeartbeatResponse
}
var file_api_proto_depIdxs = []int32{
0, // 0: api.AgentService.Heartbeat:input_type -> api.HeartbeatRequest
1, // 1: api.AgentService.Heartbeat:output_type -> api.HeartbeatResponse
1, // [1:2] is the sub-list for method output_type
0, // [0:1] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_api_proto_init() }
func file_api_proto_init() {
if File_api_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_api_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*HeartbeatRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_api_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*HeartbeatResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_api_proto_rawDesc,
NumEnums: 0,
NumMessages: 2,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_api_proto_goTypes,
DependencyIndexes: file_api_proto_depIdxs,
MessageInfos: file_api_proto_msgTypes,
}.Build()
File_api_proto = out.File
file_api_proto_rawDesc = nil
file_api_proto_goTypes = nil
file_api_proto_depIdxs = nil
}
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConnInterface
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
const _ = grpc.SupportPackageIsVersion6
// AgentServiceClient is the client API for AgentService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type AgentServiceClient interface {
Heartbeat(ctx context.Context, in *HeartbeatRequest, opts ...grpc.CallOption) (*HeartbeatResponse, error)
}
type agentServiceClient struct {
cc grpc.ClientConnInterface
}
func NewAgentServiceClient(cc grpc.ClientConnInterface) AgentServiceClient {
return &agentServiceClient{cc}
}
func (c *agentServiceClient) Heartbeat(ctx context.Context, in *HeartbeatRequest, opts ...grpc.CallOption) (*HeartbeatResponse, error) {
out := new(HeartbeatResponse)
err := c.cc.Invoke(ctx, "/api.AgentService/Heartbeat", in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// AgentServiceServer is the server API for AgentService service.
type AgentServiceServer interface {
Heartbeat(context.Context, *HeartbeatRequest) (*HeartbeatResponse, error)
}
// UnimplementedAgentServiceServer can be embedded to have forward compatible implementations.
type UnimplementedAgentServiceServer struct {
}
func (*UnimplementedAgentServiceServer) Heartbeat(context.Context, *HeartbeatRequest) (*HeartbeatResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method Heartbeat not implemented")
}
func RegisterAgentServiceServer(s *grpc.Server, srv AgentServiceServer) {
s.RegisterService(&_AgentService_serviceDesc, srv)
}
func _AgentService_Heartbeat_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(HeartbeatRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(AgentServiceServer).Heartbeat(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: "/api.AgentService/Heartbeat",
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(AgentServiceServer).Heartbeat(ctx, req.(*HeartbeatRequest))
}
return interceptor(ctx, in, info, handler)
}
var _AgentService_serviceDesc = grpc.ServiceDesc{
ServiceName: "api.AgentService",
HandlerType: (*AgentServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "Heartbeat",
Handler: _AgentService_Heartbeat_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "api.proto",
}
@@ -0,0 +1,244 @@
// Code generated by protoc-gen-validate. DO NOT EDIT.
// source: api.proto
package scannerpb
import (
"bytes"
"errors"
"fmt"
"net"
"net/mail"
"net/url"
"regexp"
"sort"
"strings"
"time"
"unicode/utf8"
"google.golang.org/protobuf/types/known/anypb"
)
// ensure the imports are used
var (
_ = bytes.MinRead
_ = errors.New("")
_ = fmt.Print
_ = utf8.UTFMax
_ = (*regexp.Regexp)(nil)
_ = (*strings.Reader)(nil)
_ = net.IPv4len
_ = time.Duration(0)
_ = (*url.URL)(nil)
_ = (*mail.Address)(nil)
_ = anypb.Any{}
_ = sort.Sort
)
// Validate checks the field values on HeartbeatRequest with the rules defined
// in the proto definition for this message. If any rules are violated, the
// first error encountered is returned, or nil if there are no violations.
func (m *HeartbeatRequest) Validate() error {
return m.validate(false)
}
// ValidateAll checks the field values on HeartbeatRequest with the rules
// defined in the proto definition for this message. If any rules are
// violated, the result is a list of violation errors wrapped in
// HeartbeatRequestMultiError, or nil if none found.
func (m *HeartbeatRequest) ValidateAll() error {
return m.validate(true)
}
func (m *HeartbeatRequest) validate(all bool) error {
if m == nil {
return nil
}
var errors []error
// no validation rules for Status
// no validation rules for Reason
// no validation rules for Version
if len(errors) > 0 {
return HeartbeatRequestMultiError(errors)
}
return nil
}
// HeartbeatRequestMultiError is an error wrapping multiple validation errors
// returned by HeartbeatRequest.ValidateAll() if the designated constraints
// aren't met.
type HeartbeatRequestMultiError []error
// Error returns a concatenation of all the error messages it wraps.
func (m HeartbeatRequestMultiError) Error() string {
var msgs []string
for _, err := range m {
msgs = append(msgs, err.Error())
}
return strings.Join(msgs, "; ")
}
// AllErrors returns a list of validation violation errors.
func (m HeartbeatRequestMultiError) AllErrors() []error { return m }
// HeartbeatRequestValidationError is the validation error returned by
// HeartbeatRequest.Validate if the designated constraints aren't met.
type HeartbeatRequestValidationError struct {
field string
reason string
cause error
key bool
}
// Field function returns field value.
func (e HeartbeatRequestValidationError) Field() string { return e.field }
// Reason function returns reason value.
func (e HeartbeatRequestValidationError) Reason() string { return e.reason }
// Cause function returns cause value.
func (e HeartbeatRequestValidationError) Cause() error { return e.cause }
// Key function returns key value.
func (e HeartbeatRequestValidationError) Key() bool { return e.key }
// ErrorName returns error name.
func (e HeartbeatRequestValidationError) ErrorName() string { return "HeartbeatRequestValidationError" }
// Error satisfies the builtin error interface
func (e HeartbeatRequestValidationError) Error() string {
cause := ""
if e.cause != nil {
cause = fmt.Sprintf(" | caused by: %v", e.cause)
}
key := ""
if e.key {
key = "key for "
}
return fmt.Sprintf(
"invalid %sHeartbeatRequest.%s: %s%s",
key,
e.field,
e.reason,
cause)
}
var _ error = HeartbeatRequestValidationError{}
var _ interface {
Field() string
Reason() string
Key() bool
Cause() error
ErrorName() string
} = HeartbeatRequestValidationError{}
// Validate checks the field values on HeartbeatResponse with the rules defined
// in the proto definition for this message. If any rules are violated, the
// first error encountered is returned, or nil if there are no violations.
func (m *HeartbeatResponse) Validate() error {
return m.validate(false)
}
// ValidateAll checks the field values on HeartbeatResponse with the rules
// defined in the proto definition for this message. If any rules are
// violated, the result is a list of violation errors wrapped in
// HeartbeatResponseMultiError, or nil if none found.
func (m *HeartbeatResponse) ValidateAll() error {
return m.validate(true)
}
func (m *HeartbeatResponse) validate(all bool) error {
if m == nil {
return nil
}
var errors []error
if len(errors) > 0 {
return HeartbeatResponseMultiError(errors)
}
return nil
}
// HeartbeatResponseMultiError is an error wrapping multiple validation errors
// returned by HeartbeatResponse.ValidateAll() if the designated constraints
// aren't met.
type HeartbeatResponseMultiError []error
// Error returns a concatenation of all the error messages it wraps.
func (m HeartbeatResponseMultiError) Error() string {
var msgs []string
for _, err := range m {
msgs = append(msgs, err.Error())
}
return strings.Join(msgs, "; ")
}
// AllErrors returns a list of validation violation errors.
func (m HeartbeatResponseMultiError) AllErrors() []error { return m }
// HeartbeatResponseValidationError is the validation error returned by
// HeartbeatResponse.Validate if the designated constraints aren't met.
type HeartbeatResponseValidationError struct {
field string
reason string
cause error
key bool
}
// Field function returns field value.
func (e HeartbeatResponseValidationError) Field() string { return e.field }
// Reason function returns reason value.
func (e HeartbeatResponseValidationError) Reason() string { return e.reason }
// Cause function returns cause value.
func (e HeartbeatResponseValidationError) Cause() error { return e.cause }
// Key function returns key value.
func (e HeartbeatResponseValidationError) Key() bool { return e.key }
// ErrorName returns error name.
func (e HeartbeatResponseValidationError) ErrorName() string {
return "HeartbeatResponseValidationError"
}
// Error satisfies the builtin error interface
func (e HeartbeatResponseValidationError) Error() string {
cause := ""
if e.cause != nil {
cause = fmt.Sprintf(" | caused by: %v", e.cause)
}
key := ""
if e.key {
key = "key for "
}
return fmt.Sprintf(
"invalid %sHeartbeatResponse.%s: %s%s",
key,
e.field,
e.reason,
cause)
}
var _ error = HeartbeatResponseValidationError{}
var _ interface {
Field() string
Reason() string
Key() bool
Cause() error
ErrorName() string
} = HeartbeatResponseValidationError{}
@@ -0,0 +1,117 @@
package trufflehogenterprisescanner
import (
"context"
"crypto/tls"
"crypto/x509"
"regexp"
"strings"
"github.com/pkg/errors"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/metadata"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/trufflehogenterprisescanner/scannerpb"
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
)
type Scanner struct{}
// Ensure the Scanner satisfies the interface at compile time.
var _ detectors.Detector = (*Scanner)(nil)
var (
// Make sure that your group is surrounded in boundary characters such as below to reduce false positives.
keyPat = regexp.MustCompile(`\bthog-agent-[0-9a-f]{32}\b`)
groupPat = regexp.MustCompile(`\bthog-scanner-[a-zA-Z]+\b`)
addressPat = regexp.MustCompile(`\b[a-z]+-[a-z]+-[a-z]+.[a-z]+\.[a-z][0-9]\.[a-z]+\.trufflehog\.org:[0-9]{4}\b`)
)
// Keywords are used for efficiently pre-filtering chunks.
// Use identifiers in the secret preferably, or the provider name.
func (s Scanner) Keywords() []string {
return []string{"thog"}
}
// FromData will find and optionally verify TrufflehogEnterpriseScanner secrets in a given set of bytes.
func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (results []detectors.Result, err error) {
dataStr := string(data)
keyMatches := keyPat.FindAllStringSubmatch(dataStr, -1)
groupMatches := groupPat.FindAllStringSubmatch(dataStr, -1)
addressMatches := addressPat.FindAllStringSubmatch(dataStr, -1)
for _, keyMatch := range keyMatches {
if len(keyMatch) != 1 {
continue
}
resKeyMatch := strings.TrimSpace(keyMatch[0])
for _, groupMatch := range groupMatches {
if len(groupMatch) != 1 {
continue
}
resGroupMatch := strings.TrimSpace(groupMatch[0])
for _, addressMatch := range addressMatches {
if len(addressMatch) != 1 {
continue
}
resAddressMatch := strings.TrimSpace(addressMatch[0])
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_TrufflehogEnterpriseScanner,
Raw: []byte(resKeyMatch),
}
if verify {
err := grpcHeartbeat(resAddressMatch, resGroupMatch, resKeyMatch)
if err == nil {
s1.Verified = true
}
}
results = append(results, s1)
}
}
}
return results, nil
}
func (s Scanner) Type() detectorspb.DetectorType {
return detectorspb.DetectorType_TrufflehogEnterpriseScanner
}
func grpcHeartbeat(address, group, token string) error {
ctx := context.Background()
ctx = metadata.NewOutgoingContext(ctx,
metadata.New(map[string]string{
"name": strings.TrimSpace(group),
"authorization": strings.TrimSpace(token),
}))
systemRoots, err := x509.SystemCertPool()
if err != nil {
panic(errors.Wrap(err, "cannot load root CA certs"))
}
creds := credentials.NewTLS(&tls.Config{
RootCAs: systemRoots,
})
conn, err := grpc.Dial(address, grpc.WithTransportCredentials(creds))
if err != nil {
return errors.Wrap(err, "cannot dial")
}
defer conn.Close()
c := scannerpb.NewAgentServiceClient(conn)
_, err = c.Heartbeat(ctx, &scannerpb.HeartbeatRequest{Status: "GOOD", Reason: "", Version: "v1.0.0"})
if err != nil {
return errors.Wrap(err, "cannot heartbeat")
}
return nil
}
@@ -0,0 +1,121 @@
//go:build detectors
// +build detectors
package trufflehogenterprisescanner
import (
"context"
"fmt"
"testing"
"time"
"github.com/kylelemons/godebug/pretty"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
"github.com/trufflesecurity/trufflehog/v3/pkg/common"
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/detectorspb"
)
func TestTrufflehogEnterpriseScanner_FromChunk(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
testSecrets, err := common.GetSecret(ctx, "trufflehog-testing", "detectors5")
if err != nil {
t.Fatalf("could not get test secrets from GCP: %s", err)
}
address := testSecrets.MustGetField("THOG_SCANNER_ADDRESS")
group := testSecrets.MustGetField("THOG_SCANNER_GROUP")
token := testSecrets.MustGetField("THOG_SCANNER_TOKEN")
inactiveToken := testSecrets.MustGetField("THOG_SCANNER_TOKEN_INACTIVE")
type args struct {
ctx context.Context
data []byte
verify bool
}
tests := []struct {
name string
s Scanner
args args
want []detectors.Result
wantErr bool
}{
{
name: "found, verified",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("You can find a thog scanner secret %s for %s at %s within", token, group, address)),
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_TrufflehogEnterpriseScanner,
Verified: true,
},
},
wantErr: false,
},
{
name: "found, unverified",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte(fmt.Sprintf("You can find a thog scanner secret %s for %s at %s within but not valid", inactiveToken, group, address)), // the secret would satisfy the regex but not pass validation
verify: true,
},
want: []detectors.Result{
{
DetectorType: detectorspb.DetectorType_TrufflehogEnterpriseScanner,
Verified: false,
},
},
wantErr: false,
},
{
name: "not found",
s: Scanner{},
args: args{
ctx: context.Background(),
data: []byte("You cannot find the secret within"),
verify: true,
},
want: nil,
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := Scanner{}
got, err := s.FromData(tt.args.ctx, tt.args.verify, tt.args.data)
if (err != nil) != tt.wantErr {
t.Errorf("Trufflehogenterprisescanner.FromData() error = %v, wantErr %v", err, tt.wantErr)
return
}
for i := range got {
if len(got[i].Raw) == 0 {
t.Fatalf("no raw secret present: \n %+v", got[i])
}
got[i].Raw = nil
}
if diff := pretty.Compare(got, tt.want); diff != "" {
t.Errorf("Trufflehogenterprisescanner.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
}
})
}
}
func BenchmarkFromData(benchmark *testing.B) {
ctx := context.Background()
s := Scanner{}
for name, data := range detectors.MustGetBenchmarkData() {
benchmark.Run(name, func(b *testing.B) {
for n := 0; n < b.N; n++ {
_, err := s.FromData(ctx, false, data)
if err != nil {
b.Fatal(err)
}
}
})
}
}
+17 -16
View File
@@ -21,6 +21,7 @@ var (
identifierPat = regexp.MustCompile(`(?i)sid.{0,20}AC[0-9a-f]{32}`) // Should we have this? Seems restrictive.
sidPat = regexp.MustCompile(`\bAC[0-9a-f]{32}\b`)
keyPat = regexp.MustCompile(`\b[0-9a-f]{32}\b`)
client = common.SaneHttpClient()
)
// Keywords are used for efficiently pre-filtering chunks.
@@ -43,15 +44,15 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
sidMatches := sidPat.FindAllString(dataStr, -1)
for _, sid := range sidMatches {
s := detectors.Result{
DetectorType: detectorspb.DetectorType_Twilio,
Raw: []byte(sid),
Redacted: sid,
}
for _, key := range keyMatches {
s1 := detectors.Result{
DetectorType: detectorspb.DetectorType_Twilio,
Raw: []byte(sid),
RawV2: []byte(sid + key),
Redacted: sid,
}
if verify {
client := common.SaneHttpClient()
for _, key := range keyMatches {
if verify {
form := url.Values{}
form.Add("FriendlyName", "MyServiceName")
@@ -71,22 +72,22 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
res.Body.Close() // The request body is unused.
if res.StatusCode >= 200 && res.StatusCode < 300 {
s.Verified = true
s1.Verified = true
}
}
}
}
if !s.Verified && detectors.IsKnownFalsePositive(string(s.Raw), detectors.DefaultFalsePositives, true) {
continue
}
if !s1.Verified && detectors.IsKnownFalsePositive(string(s1.Raw), detectors.DefaultFalsePositives, true) {
continue
}
if len(keyMatches) > 0 {
results = append(results, s)
if len(keyMatches) > 0 {
results = append(results, s1)
}
}
}
return
return detectors.CleanResults(results), nil
}
func (s Scanner) Type() detectorspb.DetectorType {
+1
View File
@@ -98,6 +98,7 @@ func TestTwilio_FromChunk(t *testing.T) {
t.Fatal("no raw secret present")
}
got[i].Raw = nil
got[i].RawV2 = nil
}
if diff := pretty.Compare(got, tt.want); diff != "" {
t.Errorf("Twilio.FromData() %s diff: (-got +want)\n%s", tt.name, diff)
+5 -5
View File
@@ -1,6 +1,7 @@
package engine
import (
"fmt"
"runtime"
"github.com/go-errors/errors"
@@ -38,14 +39,13 @@ func (e *Engine) ScanCircleCI(ctx context.Context, token string) error {
return errors.WrapPrefix(err, "failed to init Circle CI source", 0)
}
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := circleSource.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "error scanning Circle CI")
return fmt.Errorf("error scanning CircleCI: %w", err)
}
}()
return nil
})
return nil
}
+2 -2
View File
@@ -323,7 +323,6 @@ import (
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/iconfinder"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/iexapis"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/iexcloud"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/image4"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/imagekit"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/imagga"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/impala"
@@ -459,6 +458,7 @@ import (
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/opengraphr"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/openuv"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/openweather"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/opsgenie"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/optimizely"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/owlbot"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors/packagecloud"
@@ -1417,7 +1417,6 @@ func DefaultDetectors() []detectors.Detector {
exportsdk.Scanner{},
rechargepayments.Scanner{},
browserstack.Scanner{},
image4.Scanner{},
crossbrowsertesting.Scanner{},
flowdash.Scanner{},
lunchmoney.Scanner{},
@@ -1525,6 +1524,7 @@ func DefaultDetectors() []detectors.Detector {
aiven.Scanner{},
prefect.Scanner{},
buildkitev2.Scanner{},
opsgenie.Scanner{},
}
}
+5 -5
View File
@@ -1,6 +1,7 @@
package engine
import (
"fmt"
"runtime"
"github.com/go-errors/errors"
@@ -25,14 +26,13 @@ func (e *Engine) ScanDocker(ctx context.Context, conn *anypb.Any) error {
return errors.WrapPrefix(err, "could not init docker source", 0)
}
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := dockerSource.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "error scanning docker image")
return fmt.Errorf("error scanning docker image: %w", err)
}
}()
return nil
})
return nil
}
+30 -9
View File
@@ -10,6 +10,7 @@ import (
"time"
ahocorasick "github.com/petar-dambovaliev/aho-corasick"
"golang.org/x/sync/errgroup"
"google.golang.org/protobuf/proto"
"github.com/trufflesecurity/trufflehog/v3/pkg/common"
@@ -32,7 +33,7 @@ type Engine struct {
chunksScanned uint64
bytesScanned uint64
detectorAvgTime sync.Map
sourcesWg sync.WaitGroup
sourcesWg *errgroup.Group
workersWg sync.WaitGroup
// filterUnverified is used to reduce the number of unverified results.
// If there are multiple unverified results for the same chunk for the same detector,
@@ -52,6 +53,8 @@ func WithConcurrency(concurrency int) EngineOption {
}
}
const ignoreTag = "trufflehog:ignore"
func WithDetectors(verify bool, d ...detectors.Detector) EngineOption {
return func(e *Engine) {
if e.detectors == nil {
@@ -123,6 +126,11 @@ func Start(ctx context.Context, options ...EngineOption) *Engine {
}
ctx.Logger().V(2).Info("engine started", "workers", e.concurrency)
sourcesWg, egCtx := errgroup.WithContext(ctx)
sourcesWg.SetLimit(e.concurrency)
e.sourcesWg = sourcesWg
ctx.SetParent(egCtx)
if len(e.decoders) == 0 {
e.decoders = decoders.DefaultDecoders()
}
@@ -188,10 +196,14 @@ func Start(ctx context.Context, options ...EngineOption) *Engine {
// Finish waits for running sources to complete and workers to finish scanning
// chunks before closing their respective channels. Once Finish is called, no
// more sources may be scanned by the engine.
func (e *Engine) Finish(ctx context.Context) {
func (e *Engine) Finish(ctx context.Context, logFunc func(error, string, ...any)) {
defer common.RecoverWithExit(ctx)
// wait for the sources to finish putting chunks onto the chunks channel
e.sourcesWg.Wait()
sourceErr := e.sourcesWg.Wait()
if sourceErr != nil {
logFunc(sourceErr, "error occurred while collecting chunks")
}
close(e.chunks)
// wait for the workers to finish processing all of the chunks and putting
// results onto the results channel
@@ -305,6 +317,7 @@ func (e *Engine) detectorWorker(ctx context.Context) {
}
for _, result := range results {
resultChunk := chunk
ignoreLinePresent := false
if SupportsLineNumbers(chunk.SourceType) {
copyChunk := *chunk
copyMetaDataClone := proto.Clone(chunk.SourceMetadata)
@@ -312,9 +325,12 @@ func (e *Engine) detectorWorker(ctx context.Context) {
copyChunk.SourceMetadata = copyMetaData
}
fragStart, mdLine := FragmentFirstLine(&copyChunk)
SetResultLineNumber(&copyChunk, &result, fragStart, mdLine)
ignoreLinePresent = SetResultLineNumber(&copyChunk, &result, fragStart, mdLine)
resultChunk = &copyChunk
}
if ignoreLinePresent {
continue
}
result.DecoderType = decoderType
e.results <- detectors.CopyMetadata(resultChunk, result)
@@ -367,14 +383,18 @@ func SupportsLineNumbers(sourceType sourcespb.SourceType) bool {
}
// FragmentLineOffset sets the line number for a provided source chunk with a given detector result.
func FragmentLineOffset(chunk *sources.Chunk, result *detectors.Result) int64 {
func FragmentLineOffset(chunk *sources.Chunk, result *detectors.Result) (int64, bool) {
lines := bytes.Split(chunk.Data, []byte("\n"))
for i, line := range lines {
if bytes.Contains(line, result.Raw) {
return int64(i)
// if the line contains the ignore tag, we should ignore the result
if bytes.Contains(line, []byte(ignoreTag)) {
return int64(i), true
}
return int64(i), false
}
}
return 0
return 0, false
}
// FragmentFirstLine returns the first line number of a fragment along with a pointer to the value to update in the
@@ -401,7 +421,8 @@ func FragmentFirstLine(chunk *sources.Chunk) (int64, *int64) {
}
// SetResultLineNumber sets the line number in the provided result.
func SetResultLineNumber(chunk *sources.Chunk, result *detectors.Result, fragStart int64, mdLine *int64) {
offset := FragmentLineOffset(chunk, result)
func SetResultLineNumber(chunk *sources.Chunk, result *detectors.Result, fragStart int64, mdLine *int64) bool {
offset, skip := FragmentLineOffset(chunk, result)
*mdLine = fragStart + offset
return skip
}
+64
View File
@@ -0,0 +1,64 @@
package engine
import (
"testing"
"github.com/trufflesecurity/trufflehog/v3/pkg/detectors"
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
)
func TestFragmentLineOffset(t *testing.T) {
tests := []struct {
name string
chunk *sources.Chunk
result *detectors.Result
expectedLine int64
ignore bool
}{
{
name: "ignore found on same line",
chunk: &sources.Chunk{
Data: []byte("line1\nline2\nsecret here trufflehog:ignore\nline4"),
},
result: &detectors.Result{
Raw: []byte("secret here"),
},
expectedLine: 2,
ignore: true,
},
{
name: "no ignore",
chunk: &sources.Chunk{
Data: []byte("line1\nline2\nsecret here\nline4"),
},
result: &detectors.Result{
Raw: []byte("secret here"),
},
expectedLine: 2,
ignore: false,
},
{
name: "ignore on different line",
chunk: &sources.Chunk{
Data: []byte("line1\nline2\ntrufflehog:ignore\nline4\nsecret here\nline6"),
},
result: &detectors.Result{
Raw: []byte("secret here"),
},
expectedLine: 4,
ignore: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
lineOffset, isIgnored := FragmentLineOffset(tt.chunk, tt.result)
if lineOffset != tt.expectedLine {
t.Errorf("Expected line offset to be %d, got %d", tt.expectedLine, lineOffset)
}
if isIgnored != tt.ignore {
t.Errorf("Expected isIgnored to be %v, got %v", tt.ignore, isIgnored)
}
})
}
}
+5 -5
View File
@@ -1,6 +1,7 @@
package engine
import (
"fmt"
"runtime"
"github.com/go-errors/errors"
@@ -37,14 +38,13 @@ func (e *Engine) ScanFileSystem(ctx context.Context, c sources.FilesystemConfig)
return errors.WrapPrefix(err, "could not init filesystem source", 0)
}
fileSystemSource.WithFilter(c.Filter)
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := fileSystemSource.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "error scanning filesystem")
return fmt.Errorf("error scanning filesystem: %w", err)
}
}()
return nil
})
return nil
}
+4 -5
View File
@@ -54,14 +54,13 @@ func (e *Engine) ScanGCS(ctx context.Context, c sources.GCSConfig) error {
return fmt.Errorf("failed to initialize GCS source: %w", err)
}
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
if err := source.Chunks(ctx, e.ChunksChan()); err != nil {
ctx.Logger().Error(err, "could not scan GCS")
return fmt.Errorf("could not scan GCS: %w", err)
}
}()
return nil
})
return nil
}
+21 -2
View File
@@ -5,6 +5,7 @@ import (
"testing"
"github.com/trufflesecurity/trufflehog/v3/pkg/context"
"github.com/trufflesecurity/trufflehog/v3/pkg/decoders"
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
)
@@ -55,12 +56,30 @@ func TestScanGCS(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
e := &Engine{}
err := e.ScanGCS(context.Background(), test.gcsConfig)
ctx, cancel := context.WithCancel(context.TODO())
defer cancel()
e := Start(ctx,
WithConcurrency(1),
WithDecoders(decoders.DefaultDecoders()...),
WithDetectors(false, DefaultDetectors()...),
)
go func() {
resultCount := 0
for range e.ResultsChan() {
resultCount++
}
}()
err := e.ScanGCS(ctx, test.gcsConfig)
if err != nil && !test.wantErr && !strings.Contains(err.Error(), "googleapi: Error 400: Bad Request") {
t.Errorf("ScanGCS() got: %v, want: %v", err, nil)
return
}
logFatalFunc := func(_ error, _ string, _ ...any) {
t.Fatalf("error logging function should not have been called")
}
e.Finish(ctx, logFatalFunc)
if err == nil && test.wantErr {
t.Errorf("ScanGCS() got: %v, want: %v", err, "error")
+4 -5
View File
@@ -66,14 +66,13 @@ func (e *Engine) ScanGit(ctx context.Context, c sources.GitConfig) error {
"source_type", sourcespb.SourceType_SOURCE_TYPE_GIT.String(),
"source_name", "git",
)
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := gitSource.ScanRepo(ctx, repo, c.RepoPath, scanOptions, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "could not scan repo")
return fmt.Errorf("could not scan repo: %w", err)
}
}()
return nil
})
return nil
}
+41 -32
View File
@@ -53,39 +53,45 @@ func TestGitEngine(t *testing.T) {
base: "2f251b8c1e72135a375b659951097ec7749d4af9",
},
} {
e := Start(ctx,
WithConcurrency(1),
WithDecoders(decoders.DefaultDecoders()...),
WithDetectors(false, DefaultDetectors()...),
)
cfg := sources.GitConfig{
RepoPath: path,
HeadRef: tTest.branch,
BaseRef: tTest.base,
MaxDepth: tTest.maxDepth,
Filter: tTest.filter,
}
if err := e.ScanGit(ctx, cfg); err != nil {
return
}
go e.Finish(ctx)
resultCount := 0
for result := range e.ResultsChan() {
switch meta := result.SourceMetadata.GetData().(type) {
case *source_metadatapb.MetaData_Git:
if tTest.expected[meta.Git.Commit].B != string(result.Raw) {
t.Errorf("%s: unexpected result. Got: %s, Expected: %s", tName, string(result.Raw), tTest.expected[meta.Git.Commit].B)
}
if tTest.expected[meta.Git.Commit].LineNumber != result.SourceMetadata.GetGit().Line {
t.Errorf("%s: unexpected line number. Got: %d, Expected: %d", tName, result.SourceMetadata.GetGit().Line, tTest.expected[meta.Git.Commit].LineNumber)
}
t.Run(tName, func(t *testing.T) {
e := Start(ctx,
WithConcurrency(1),
WithDecoders(decoders.DefaultDecoders()...),
WithDetectors(false, DefaultDetectors()...),
)
cfg := sources.GitConfig{
RepoPath: path,
HeadRef: tTest.branch,
BaseRef: tTest.base,
MaxDepth: tTest.maxDepth,
Filter: tTest.filter,
}
if err := e.ScanGit(ctx, cfg); err != nil {
return
}
resultCount++
}
if resultCount != len(tTest.expected) {
t.Errorf("%s: unexpected number of results. Got: %d, Expected: %d", tName, resultCount, len(tTest.expected))
}
logFatalFunc := func(_ error, _ string, _ ...any) {
t.Fatalf("error logging function should not have been called")
}
go e.Finish(ctx, logFatalFunc)
resultCount := 0
for result := range e.ResultsChan() {
switch meta := result.SourceMetadata.GetData().(type) {
case *source_metadatapb.MetaData_Git:
if tTest.expected[meta.Git.Commit].B != string(result.Raw) {
t.Errorf("%s: unexpected result. Got: %s, Expected: %s", tName, string(result.Raw), tTest.expected[meta.Git.Commit].B)
}
if tTest.expected[meta.Git.Commit].LineNumber != result.SourceMetadata.GetGit().Line {
t.Errorf("%s: unexpected line number. Got: %d, Expected: %d", tName, result.SourceMetadata.GetGit().Line, tTest.expected[meta.Git.Commit].LineNumber)
}
}
resultCount++
}
if resultCount != len(tTest.expected) {
t.Errorf("%s: unexpected number of results. Got: %d, Expected: %d", tName, resultCount, len(tTest.expected))
}
})
}
}
@@ -124,5 +130,8 @@ func BenchmarkGitEngine(b *testing.B) {
return
}
}
e.Finish(ctx)
logFatalFunc := func(_ error, _ string, _ ...any) {
b.Fatalf("error logging function should not have been called")
}
e.Finish(ctx, logFatalFunc)
}
+6 -5
View File
@@ -1,6 +1,8 @@
package engine
import (
"fmt"
gogit "github.com/go-git/go-git/v5"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/anypb"
@@ -58,14 +60,13 @@ func (e *Engine) ScanGitHub(ctx context.Context, c sources.GithubConfig) error {
scanOptions := git.NewScanOptions(opts...)
source.WithScanOptions(scanOptions)
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := source.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "could not scan github")
return fmt.Errorf("could not scan github: %w", err)
}
}()
return nil
})
return nil
}
+4 -5
View File
@@ -63,14 +63,13 @@ func (e *Engine) ScanGitLab(ctx context.Context, c sources.GitlabConfig) error {
}
gitlabSource.WithScanOptions(scanOptions)
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := gitlabSource.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "error scanning GitLab")
return fmt.Errorf("error scanning GitLab: %w", err)
}
}()
return nil
})
return nil
}
+4 -5
View File
@@ -65,14 +65,13 @@ func (e *Engine) ScanS3(ctx context.Context, c sources.S3Config) error {
return errors.WrapPrefix(err, "failed to init S3 source", 0)
}
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := s3Source.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "error scanning S3")
return fmt.Errorf("error scanning S3: %w", err)
}
}()
return nil
})
return nil
}
+5 -5
View File
@@ -1,6 +1,7 @@
package engine
import (
"fmt"
"os"
"github.com/go-errors/errors"
@@ -53,14 +54,13 @@ func (e *Engine) ScanSyslog(ctx context.Context, c sources.SyslogConfig) error {
return err
}
e.sourcesWg.Add(1)
go func() {
e.sourcesWg.Go(func() error {
defer common.RecoverWithExit(ctx)
defer e.sourcesWg.Done()
err := source.Chunks(ctx, e.ChunksChan())
if err != nil {
ctx.Logger().Error(err, "could not scan syslog")
return fmt.Errorf("could not scan syslog: %w", err)
}
}()
return nil
})
return nil
}
+3 -3
View File
@@ -140,10 +140,10 @@ func (c *Parser) RepoPath(ctx context.Context, source string, head string, abbre
return c.executeCommand(ctx, cmd)
}
// Unstaged parses the output of the `git diff` command for the `source` path.
func (c *Parser) Unstaged(ctx context.Context, source string) (chan Commit, error) {
// Staged parses the output of the `git diff` command for the `source` path.
func (c *Parser) Staged(ctx context.Context, source string) (chan Commit, error) {
// Provide the --cached flag to diff to get the diff of the staged changes.
args := []string{"-C", source, "diff", "-p", "--cached", "--full-history", "--diff-filter=AM", "--date=format:%a %b %d %H:%M:%S %Y %z", "HEAD"}
args := []string{"-C", source, "diff", "-p", "--cached", "--full-history", "--diff-filter=AM", "--date=format:%a %b %d %H:%M:%S %Y %z"}
cmd := exec.Command("git", args...)
+16 -6
View File
@@ -991,6 +991,8 @@ const (
DetectorType_Aiven DetectorType = 917
DetectorType_Prefect DetectorType = 918
DetectorType_Docusign DetectorType = 919
DetectorType_Couchbase DetectorType = 920
DetectorType_TrufflehogEnterpriseScanner DetectorType = 922
)
// Enum value maps for DetectorType.
@@ -1912,6 +1914,8 @@ var (
917: "Aiven",
918: "Prefect",
919: "Docusign",
920: "Couchbase",
922: "TrufflehogEnterpriseScanner",
}
DetectorType_value = map[string]int32{
"Alibaba": 0,
@@ -2830,6 +2834,8 @@ var (
"Aiven": 917,
"Prefect": 918,
"Docusign": 919,
"Couchbase": 920,
"TrufflehogEnterpriseScanner": 922,
}
)
@@ -3194,7 +3200,7 @@ var file_detectors_proto_rawDesc = []byte{
0x0a, 0x0b, 0x44, 0x65, 0x63, 0x6f, 0x64, 0x65, 0x72, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a,
0x07, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x10, 0x00, 0x12, 0x09, 0x0a, 0x05, 0x50, 0x4c,
0x41, 0x49, 0x4e, 0x10, 0x01, 0x12, 0x0a, 0x0a, 0x06, 0x42, 0x41, 0x53, 0x45, 0x36, 0x34, 0x10,
0x02, 0x12, 0x09, 0x0a, 0x05, 0x55, 0x54, 0x46, 0x31, 0x36, 0x10, 0x03, 0x2a, 0xf9, 0x72, 0x0a,
0x02, 0x12, 0x09, 0x0a, 0x05, 0x55, 0x54, 0x46, 0x31, 0x36, 0x10, 0x03, 0x2a, 0xab, 0x73, 0x0a,
0x0c, 0x44, 0x65, 0x74, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a,
0x07, 0x41, 0x6c, 0x69, 0x62, 0x61, 0x62, 0x61, 0x10, 0x00, 0x12, 0x08, 0x0a, 0x04, 0x41, 0x4d,
0x51, 0x50, 0x10, 0x01, 0x12, 0x07, 0x0a, 0x03, 0x41, 0x57, 0x53, 0x10, 0x02, 0x12, 0x09, 0x0a,
@@ -4114,11 +4120,15 @@ var file_detectors_proto_rawDesc = []byte{
0x0b, 0x4e, 0x75, 0x47, 0x65, 0x74, 0x41, 0x70, 0x69, 0x4b, 0x65, 0x79, 0x10, 0x94, 0x07, 0x12,
0x0a, 0x0a, 0x05, 0x41, 0x69, 0x76, 0x65, 0x6e, 0x10, 0x95, 0x07, 0x12, 0x0c, 0x0a, 0x07, 0x50,
0x72, 0x65, 0x66, 0x65, 0x63, 0x74, 0x10, 0x96, 0x07, 0x12, 0x0d, 0x0a, 0x08, 0x44, 0x6f, 0x63,
0x75, 0x73, 0x69, 0x67, 0x6e, 0x10, 0x97, 0x07, 0x42, 0x3d, 0x5a, 0x3b, 0x67, 0x69, 0x74, 0x68,
0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x74, 0x72, 0x75, 0x66, 0x66, 0x6c, 0x65, 0x73, 0x65,
0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x2f, 0x74, 0x72, 0x75, 0x66, 0x66, 0x6c, 0x65, 0x68, 0x6f,
0x67, 0x2f, 0x76, 0x33, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x70, 0x62, 0x2f, 0x64, 0x65, 0x74, 0x65,
0x63, 0x74, 0x6f, 0x72, 0x73, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x75, 0x73, 0x69, 0x67, 0x6e, 0x10, 0x97, 0x07, 0x12, 0x0e, 0x0a, 0x09, 0x43, 0x6f, 0x75, 0x63,
0x68, 0x62, 0x61, 0x73, 0x65, 0x10, 0x98, 0x07, 0x12, 0x20, 0x0a, 0x1b, 0x54, 0x72, 0x75, 0x66,
0x66, 0x6c, 0x65, 0x68, 0x6f, 0x67, 0x45, 0x6e, 0x74, 0x65, 0x72, 0x70, 0x72, 0x69, 0x73, 0x65,
0x53, 0x63, 0x61, 0x6e, 0x6e, 0x65, 0x72, 0x10, 0x9a, 0x07, 0x42, 0x3d, 0x5a, 0x3b, 0x67, 0x69,
0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x74, 0x72, 0x75, 0x66, 0x66, 0x6c, 0x65,
0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x2f, 0x74, 0x72, 0x75, 0x66, 0x66, 0x6c, 0x65,
0x68, 0x6f, 0x67, 0x2f, 0x76, 0x33, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x70, 0x62, 0x2f, 0x64, 0x65,
0x74, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f,
0x33,
}
var (
+3 -1
View File
@@ -31,10 +31,12 @@ type Source struct {
jobPool *errgroup.Group
sources.Progress
client *http.Client
sources.CommonSourceUnitUnmarshaller
}
// Ensure the Source satisfies the interface at compile time.
// Ensure the Source satisfies the interfaces at compile time.
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
+3 -1
View File
@@ -30,12 +30,14 @@ type Source struct {
verify bool
conn sourcespb.Docker
sources.Progress
sources.CommonSourceUnitUnmarshaller
}
var FilesizeLimitBytes int64 = 50 * 1024 * 1024 // 50MB
// Ensure the Source satisfies the interface at compile time.
// Ensure the Source satisfies the interfaces at compile time.
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
+3 -1
View File
@@ -39,10 +39,12 @@ type Source struct {
log logr.Logger
filter *common.Filter
sources.Progress
sources.CommonSourceUnitUnmarshaller
}
// Ensure the Source satisfies the interface at compile time
// Ensure the Source satisfies the interfaces at compile time
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
+3 -1
View File
@@ -30,8 +30,9 @@ import (
const defaultCachePersistIncrement = 2500
// Ensure the Source satisfies the interface at compile time.
// Ensure the Source satisfies the interfaces at compile time.
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
@@ -69,6 +70,7 @@ type Source struct {
mu sync.Mutex
sources.Progress // progress is not thread safe
sources.CommonSourceUnitUnmarshaller
}
// persistableCache is a wrapper around cache.Cache that allows
+9 -4
View File
@@ -67,8 +67,9 @@ func NewGit(sourceType sourcespb.SourceType, jobID, sourceID int64, sourceName s
}
}
// Ensure the Source satisfies the interface at compile time.
// Ensure the Source satisfies the interfaces at compile time.
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
@@ -481,7 +482,7 @@ func (s *Git) ScanStaged(ctx context.Context, repo *git.Repository, path string,
// Get the URL metadata for reporting (may be empty).
urlMetadata := getSafeRemoteURL(repo, "origin")
commitChan, err := gitparse.NewParser().Unstaged(ctx, path)
commitChan, err := gitparse.NewParser().Staged(ctx, path)
if err != nil {
return err
}
@@ -738,7 +739,7 @@ func PrepareRepoSinceCommit(ctx context.Context, uriString, commitHash string) (
var path string
switch {
case uri.User != nil:
ctx.Logger().V(1).Info("cloning repo with authentication", "uri", uri)
ctx.Logger().V(1).Info("cloning repo with authentication", "uri", uri.Redacted())
password, ok := uri.User.Password()
if !ok {
return "", true, fmt.Errorf("password must be included in Git repo URL when username is provided")
@@ -776,7 +777,7 @@ func PrepareRepo(ctx context.Context, uriString string) (string, bool, error) {
remote = true
switch {
case uri.User != nil:
ctx.Logger().V(1).Info("cloning repo with authentication", "uri", uri)
ctx.Logger().V(1).Info("cloning repo with authentication", "uri", uri.Redacted())
password, ok := uri.User.Password()
if !ok {
return "", remote, fmt.Errorf("password must be included in Git repo URL when username is provided")
@@ -875,3 +876,7 @@ func handleBinary(ctx context.Context, repo *git.Repository, chunksChan chan *so
return nil
}
func (s *Source) UnmarshalSourceUnit(data []byte) (sources.SourceUnit, error) {
return UnmarshalUnit(data)
}
+40
View File
@@ -0,0 +1,40 @@
package git
import (
"encoding/json"
"fmt"
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
)
const (
UnitRepo string = "repo"
UnitDir string = "dir"
)
// Ensure SourceUnit implements the interface at compile time.
var _ sources.SourceUnit = SourceUnit{}
// A git source unit can be two kinds of units: either a local directory path
// or a remote repository.
type SourceUnit struct {
Kind string `json:"kind"`
ID string `json:"id"`
}
// Implement sources.SourceUnit interface.
func (u SourceUnit) SourceUnitID() string {
return u.ID
}
// Helper function to unmarshal raw bytes into our SourceUnit struct.
func UnmarshalUnit(data []byte) (sources.SourceUnit, error) {
var unit SourceUnit
if err := json.Unmarshal(data, &unit); err != nil {
return nil, err
}
if unit.ID == "" || (unit.Kind != UnitRepo && unit.Kind != UnitDir) {
return nil, fmt.Errorf("not a git.SourceUnit")
}
return unit, nil
}
+4 -86
View File
@@ -69,14 +69,17 @@ type Source struct {
mu sync.Mutex
publicMap map[string]source_metadatapb.Visibility
sources.Progress
sources.CommonSourceUnitUnmarshaller
}
func (s *Source) WithScanOptions(scanOptions *git.ScanOptions) {
s.scanOptions = scanOptions
}
// Ensure the Source satisfies the interface at compile time
// Ensure the Source satisfies the interfaces at compile time
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
var endsWithGithub = regexp.MustCompile(`github\.com/?$`)
// Type returns the type of source.
@@ -663,17 +666,11 @@ func (s *Source) scan(ctx context.Context, installationClient *github.Client, ch
defer func(start time.Time) {
logger.V(2).Info(fmt.Sprintf("scanned %d/%d repos", scanned, len(s.repos)), "repo_size", repoSize, "duration_seconds", time.Since(start).Seconds())
}(now)
fmt.Printf("Scanning repoURL %s, path %+v, repo %+v \n ", repoURL, path, repo)
if err = s.git.ScanRepo(ctx, repo, path, s.scanOptions, chunksChan); err != nil {
scanErrs.Add(fmt.Errorf("error scanning repo %s: %w", repoURL, err))
return nil
}
atomic.AddUint64(&scanned, 1)
fmt.Println("SCANNNNN COMMENTS")
if err = s.scanComments(ctx, repoURL, chunksChan); err != nil {
fmt.Println("test")
}
return nil
})
@@ -922,82 +919,3 @@ func (s *Source) setProgressCompleteWithRepo(index int, offset int, repoURL stri
s.SetProgressComplete(index+offset, len(s.repos)+offset, fmt.Sprintf("Repo: %s", repoURL), encodedResumeInfo)
}
func (s *Source) scanComments(ctx context.Context, repoPath string, chunksChan chan *sources.Chunk) error {
s.log.Info("scanning comments")
fmt.Print("repoPath: ", repoPath, "\n")
trimmedURL := removeURLAndSplit(repoPath)
owner := trimmedURL[1]
repo := trimmedURL[2]
//for _, part := range trimmedURL {
// fmt.Println(part)
//}
sortType := "created"
directionType := "desc"
issueOpts := &github.IssueListCommentsOptions{
Sort: &sortType,
Direction: &directionType,
ListOptions: github.ListOptions{
PerPage: defaultPagination,
Page: 1,
},
}
issueComments, resp, err := s.apiClient.Issues.ListComments(ctx, owner, repo, 0, issueOpts)
prOpts := &github.PullRequestListCommentsOptions{
Sort: sortType,
Direction: directionType,
ListOptions: github.ListOptions{
PerPage: defaultPagination,
Page: 1,
},
}
prComments, resp, err := s.apiClient.PullRequests.ListComments(ctx, owner, repo, 0, prOpts)
fmt.Println("issue comments: ", issueComments)
fmt.Println("pr comments: ", prComments)
fmt.Println("resp: ", resp)
fmt.Println("err: ", err)
//for _, comment := range comments {
// // Create chunk and send it to the channel.
// chunk := &sources.Chunk{
// SourceName: s.name,
// SourceID: s.SourceID(),
// SourceType: s.Type(),
// SourceMetadata: &source_metadatapb.MetaData{
// Data: &source_metadatapb.MetaData_Github{
// Github: &source_metadatapb.Github{
// Link: sanitizer.UTF8(comment.GetHTMLURL()),
// Username: sanitizer.UTF8(comment.GetUser().GetLogin()),
// Email: sanitizer.UTF8(comment.GetUser().GetEmail()),
// Repository: sanitizer.UTF8(repo),
// Timestamp: sanitizer.UTF8(comment.GetCreatedAt().String()),
// },
// },
// },
// Data: []byte(sanitizer.UTF8(comment.GetBody())),
// Verify: s.verify,
// }
//
// select {
// case <-ctx.Done():
// return ctx.Err()
// case chunksChan <- chunk:
// }
//}
return nil
}
func removeURLAndSplit(url string) []string {
trimmedURL := strings.TrimPrefix(url, "https://")
trimmedURL = strings.TrimSuffix(trimmedURL, ".git")
splitURL := strings.Split(trimmedURL, "/")
return splitURL
}
+3 -1
View File
@@ -47,10 +47,12 @@ type Source struct {
resumeInfoMutex sync.Mutex
sources.Progress
jobPool *errgroup.Group
sources.CommonSourceUnitUnmarshaller
}
// Ensure the Source satisfies the interface at compile time.
// Ensure the Source satisfies the interfaces at compile time.
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source.
// It is used for matching source types in configuration and job input.
+7 -2
View File
@@ -46,10 +46,12 @@ type Source struct {
conn *sourcespb.S3
jobPool *errgroup.Group
maxObjectSize int64
sources.CommonSourceUnitUnmarshaller
}
// Ensure the Source satisfies the interface at compile time
// Ensure the Source satisfies the interfaces at compile time
var _ sources.Source = (*Source)(nil)
var _ sources.SourceUnitUnmarshaller = (*Source)(nil)
// Type returns the type of source
func (s *Source) Type() sourcespb.SourceType {
@@ -195,7 +197,10 @@ func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk) err
})
if err != nil {
s.log.Error(err, "could not list objects in s3 bucket", "bucket", bucket)
return fmt.Errorf(
"could not list objects in s3 bucket: bucket %s: %w",
bucket,
err)
}
}
s.SetProgressComplete(len(bucketsToScan), len(bucketsToScan), fmt.Sprintf("Completed scanning source %s. %d objects scanned.", s.name, objectCount), "")
+26 -1
View File
@@ -1,12 +1,37 @@
package sources
import (
"encoding/json"
"fmt"
)
// Ensure CommonSourceUnit implements SourceUnit at compile time.
var _ SourceUnit = CommonSourceUnit{}
// CommonSourceUnit is a common implementation of SourceUnit that Sources can
// use instead of implementing their own types.
type CommonSourceUnit struct {
ID string `json:"source_unit_id"`
}
// Implement the SourceUnit interface.
// SourceUnitID implements the SourceUnit interface.
func (c CommonSourceUnit) SourceUnitID() string {
return c.ID
}
// CommonSourceUnitUnmarshaller is an implementation of SourceUnitUnmarshaller
// for the CommonSourceUnit. A source can embed this struct to gain the
// functionality of converting []byte to a CommonSourceUnit.
type CommonSourceUnitUnmarshaller struct{}
// UnmarshalSourceUnit implements the SourceUnitUnmarshaller interface.
func (c CommonSourceUnitUnmarshaller) UnmarshalSourceUnit(data []byte) (SourceUnit, error) {
var unit CommonSourceUnit
if err := json.Unmarshal(data, &unit); err != nil {
return nil, err
}
if unit.ID == "" {
return nil, fmt.Errorf("not a CommonSourceUnit")
}
return unit, nil
}
+19
View File
@@ -0,0 +1,19 @@
syntax = "proto3";
package api;
option go_package = "github.com/trufflesecurity/trufflehog/v3/pkg/detectors/trufflehogenterprisescanner/scannerpb";
message HeartbeatRequest {
string status = 1;
string reason = 2;
string version = 3;
}
message HeartbeatResponse {}
service AgentService {
rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);
}
+2
View File
@@ -928,6 +928,8 @@ enum DetectorType {
Aiven = 917;
Prefect = 918;
Docusign = 919;
Couchbase = 920;
TrufflehogEnterpriseScanner = 922;
}
message Result {
+7
View File
@@ -37,3 +37,10 @@ protoc -I proto/ \
--go_out=plugins=grpc:./pkg/pb/custom_detectorspb --go_opt=paths=source_relative \
--validate_out="lang=go,paths=source_relative:./pkg/pb/custom_detectorspb" \
proto/custom_detectors.proto
protoc -I proto/ \
-I ${GOPATH}/src \
-I /usr/local/include \
-I ${GOPATH}/src/github.com/envoyproxy/protoc-gen-validate \
--go_out=plugins=grpc:./pkg/detectors/trufflehogenterprisescanner/scannerpb --go_opt=paths=source_relative \
--validate_out="lang=go,paths=source_relative:./pkg/detectors/trufflehogenterprisescanner/scannerpb" \
proto/api.proto