Commit Graph

180 Commits

Author SHA1 Message Date
Richard Gomez b57b1c1aa7 feat(voiceflow): basic detector (#1900) 2023-10-18 16:17:11 -05:00
Shreyas Sriram 7f534d0bb7 Fix off by one (#1891) 2023-10-17 07:02:27 -07:00
s.shivasurya 040167178c added cody gateway token detection code (#1883)
* added cody gateway token detection code

* resolved conflict
2023-10-13 09:09:04 -06:00
Dustin Decker 52ed87edb7 Add an option to filter unverified results using shannon entropy (#1875)
* Add an option to filter unverified results using shannon entropy

* lint

* add test, update test, and optimize
2023-10-08 19:52:28 -07:00
Corben Leo 179a7e4cbc Detector-Competition-New: add anthropic api key detector (#1861)
* feat(anthropic): add anthropic api key detector

* Detector-Competition-Fix: fix remove debug println
2023-10-05 11:34:40 -05:00
Corben Leo bf1cce43e5 Detector-Competition-New: add ramp.com client id & secret detector (#1862) 2023-10-05 09:40:30 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d 5df6afdbf4 Separate gitlab detectors (#1819)
* update gitlabv2 to tri-state

* updating secret to s1 to match convention

* consolidating both versions of the gitlab detector

* remove gitlabV2 references

* Delete temp.txt

delete test file (note: these are not real secrets)

* updating gitlabV1 detector to only work w/ v1 secrets, and v2 detector only w/ v2 secrets

* update package name and add to defaults

* cleanup nesting

* lowercase package names

* update v1 detector to explicitly ignore results with glpat

* nit

* update package name
2023-09-28 12:36:46 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d e645827fcb [chore] add figmav2 to defaults (#1820)
* add figma to defaults

* update figma detector package to use versioning
2023-09-28 13:35:51 -04:00
joeleonjr 699547b7d3 consolidated pr and issue descr/comment flags (#1827) 2023-09-27 15:54:02 -04:00
Ankush Goel faf46175e4 added Web3 Storage detector (#1789)
* added Web3 Storage detector

* fixed the regex

* removed test and disabled token
2023-09-27 12:09:39 -05:00
joeleonjr 1e42dae734 added PR and Issue body scanning (#1816)
* added PR and Issue body scanning; adjusted CLI args to fit

* removed print statement from debugging

* removed exclude-commits; adjusted CLI flags

* minor changes to match main branch

* fixing logic

* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
ahrav 6affc903e1 add line to link for azure repos. (#1801) 2023-09-21 16:07:11 -07:00
ahrav a8c89c59b9 [bug] - fix link line (#1793)
* fix link line.

* rename.
2023-09-20 14:46:00 -07:00
ahrav 47d5ddebf2 Ability to update line number in link (#1788)
* Add functionality to update a source's link in the metadata with the updated line number.

* update comment.

* add logic to engine.

* only update link for non empty links.

* add tests for bb.
2023-09-19 15:39:13 -07:00
ahrav 22876f8381 replace interface{} with any. (#1771) 2023-09-15 04:35:15 -07:00
Miccah dbcb888063 Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Miccah 72b6a9ec6b Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
Miccah be4d0bcb41 Refactor SourceManager to remove Enrollment (#1740)
* Refactor SourceManager to remove Enrollment

Initializing the Source will be the responsibility of the caller. The
SourceManager exposes a GetIDs method for getting a source and job ID.

* Update tests

* Update engine usage

* Update apiClient interface to have one GetIDs method

* Update SourceManager usage in engine
2023-09-12 16:58:38 -07:00
ahrav fdeccf06a0 cache dupes w/ different decoders (#1754)
* only cache dupes that have different decoders.

* add test.

* remove file.

* update comment.
2023-09-11 08:18:48 -07:00
Marwan Sulaiman 3aa5369608 Add Tailscale detector (#1719)
* Add tailscale detector

* PR feedback: match on first element
2023-09-07 19:11:17 -07:00
s.shivasurya 6695cf1dce added sourcegraph token verification detection (#1730) 2023-08-31 08:47:13 -07:00
Zubair Khan 66271aed7b add detectors that were missed (#1716) 2023-08-28 18:02:11 -07:00
Mike Vanbuskirk 64dd49f9ce add role assumption for s3 source (#1477)
* add role assumption for s3 source

* refactor role assumption to repeatable string

user can pass array of roles to assume

* refactor s3 chunks to handle passed roleARNs

* add role-session name

use timestamp to make dynamic

* add docstring for rolearn strings()

* make sure role ars are passed into source

* refactor role assumption functionality

break s3 bucket scanning into sep. function

* add log check on assume role

* fix role iteration

- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role

* add comment

* protobuf revert for merge

* re-run make proto

* lint cleanup

* cleanup TODOs

* drop redundant switch case in assumerole client

* use less verbose 'ctx' designator

* breakout functionality from Chunks

- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets

* remake protobuf defs

* allow scan to continue on single bucket err

* add readme docs

* minor fixups
2023-08-17 20:30:20 -04:00
Miccah fae54c7ffa Add ScanChunk to allow injecting Chunks into the SourceManager's channel (#1634)
With the introduction of the SourceManager, the chunks channel became
private and read-only. This provides a method to write chunks into the
channel as we transition away from needing to do that.
2023-08-16 16:09:23 -07:00
Zubair Khan db89e345d7 correct logging output for github comments and add oss flags (#1632)
* correct logging output

* add flags

* respect oss cli flags for github comment scanning

* improve copy
2023-08-16 18:23:59 -04:00
Miccah eae66ccf7e Refactor FragmentLineOffset to match multiline secrets (#1612)
* Refactor FragmentLineOffset to match multiline secrets

* Add tests and benchmarks

* Use bytes.Count and fix an ignore tag edge case
2023-08-14 10:51:41 -07:00
Miccah 1cd600f70f Use SourceManager in engine (#1586)
* Add SourceManager to Engine struct

* Update Engine methods to use the SourceManager

* Fix GCS test

The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.

* JobProgress race fixes

* Add contextual values

* Remove unused code

* Add debug logs

* Rename WithConcurrency to WithConcurrentSources

* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
Savely Krasovsky d062834997 initial support for bare repositories (#1499)
* feat: initial support for bare repositories

* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: go-git update with pre-receive hooks fix

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: remove info about pre-receive hook from README.md for now

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: don't scan staged while using --bare option, fixes to make it work with the latest master

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: small refactor according to #1518

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

---------

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
2023-08-03 11:23:41 -05:00
ahrav 06d2eab204 include scan duration in output log (#1598)
* add scan duration to output log.

* fix linter.
2023-08-02 11:48:29 -07:00
Bill Rich 0c7ed19270 Github Oauth2 verification (#1584)
* Github Oauth2 verification

* Use prefix and include RawV2

* Make gh_oauth2 a new detector

* Remove unused struct

* Remove versioner

* Remove unused code
2023-08-02 11:16:40 -07:00
Miccah 69021f59c5 Refactor git source to allow ScanOptions and use source in engine (#1518)
* Refactor git source to allow ScanOptions and use source in engine

Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.

* Preserve ScanGit functionality of not cleaning up temporary files
2023-08-01 09:52:02 -05:00
ahrav 5043fc8756 [bug] - Fix unlocking an unlocked mutex (#1583)
* use correct mutext.

* remove unused fxn.
2023-07-31 14:06:41 -07:00
ahrav 5e7a6ca11c Concurrent detection (#1580)
* Run detection on each chunk concurrently.

* Add printer functionality.

* Add logic for dedupe.

* cleanup.

* Moddify number of notifier workers.

* Add comment.

* move consts into fxn.

* buffer resutls chan.

* fix test.

* address comments.

* return an error from Finish.

* fix test.

* fix test.

* linter.

* check err.

* address comments.
2023-07-31 11:12:08 -07:00
Dustin Decker 10b6e2898d Increase log level of engine messages (#1576) 2023-07-28 14:30:43 -07:00
Zachary Rice 1a1977f7e6 case insensitive (#1547) 2023-07-25 17:01:15 -05:00
Zachary Rice 85f363f093 init (#1538) 2023-07-24 19:09:57 -05:00
Dustin Decker fab80445d1 continue scanning on detector / decoder panic (#863) 2023-07-24 07:34:43 -07:00
Miccah 91c5472876 Implement SourceManager basics (#1515)
* Implement SourceManager basics

* Rename identifiers and add a default headlessAPI implementation

* Rewrite to use SourceInitFunc

* Update variable name to accurately reflect its value
2023-07-21 15:20:25 -05:00
Miccah a613bbb979 [chore] Remove parent manipulation in context package (#1525)
The ability to set the parent allowed creating context cycles which
shouldn't be allowed, or at the very least have unintuitive behavior.
2023-07-21 13:51:51 -05:00
Miccah e8b5e3cea3 Revert "[chore] Remove parent setting / getting in Context wrapper (#1516)" (#1519)
This reverts commit 8ec5e4916c.
This commit is somehow causing AWS verification (and possibly others) to
not work.
2023-07-20 23:31:28 -05:00
Miccah 8ec5e4916c [chore] Remove parent setting / getting in Context wrapper (#1516)
* [chore] Remove parent setting / getting in Context wrapper

* Keep the cancellable context from errgroup
2023-07-20 13:33:09 -05:00
Brandon Yan 8fad5fff79 add dockerhub scanner (#1496)
* add dockerhub scanner

* clean

* clean and fix regex logic and tests

* check length of userMatches before access

* Use camelcase.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-19 09:26:28 -07:00
Brandon Yan 17b90b32d0 add couchbase scanner to defaults (#1497)
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-16 17:45:56 -07:00
Brandon Yan 9af31f00a9 add envoy api key scanner (#1482)
* add envoy api key scanner

* Use detectors4.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-16 16:46:28 -07:00
ahrav a9213a1103 [chore] - Update loop to switch. (#1487)
* Update loop to switch.

* remove unused fxn.
2023-07-12 15:47:43 -07:00
Zachary Rice b48ac24c46 Dedupe results (#1479)
* init 4 dedupin

* use raw rather than rawv2

* rm comment

* comments

* nits

* clean up and use rawv2 too

* add decoder order test
2023-07-11 15:48:00 -05:00
Zachary Rice d4972313ff remove old detector (#1474) 2023-07-10 13:02:19 -05:00
Zachary Rice 0bdd513d88 additional similarity check for base64 and plain (#1462)
* additional similarity check for base64 and plain

* use bytes equal

* move logic into util function
2023-07-10 10:12:59 -05:00
Richard Gomez 23757dbe0a remove image4 detector (#1461) 2023-07-06 12:56:09 -07:00
Zachary Rice 18a70b64bb Introduce trufflehog:ignore tag feature (#1433)
* init ignore

* cleanup and add test

* update readme
2023-06-29 08:45:56 -05:00