Commit Graph

109 Commits

Author SHA1 Message Date
garg472 3e4496156c added new detectors and fixed mesibo detector (#1166)
* added new detectors and fixed mesibo detector

* added bscscan.com API detector

* added coinmarketcap detector

* update alchemy

* update blocknative

* update bscscan test

* update cmc test

* update tests

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-03-16 18:57:08 -07:00
Yassine Ilmi d382d5cb1c Add OpenAI API Tokens detector (#1142)
* Add OpenAI Detector

* Add OpenAI Detector tests

* Add OpenAI Detector to defaults.go

* Removing references to github detector in tests
2023-03-16 17:58:22 -07:00
ahrav c617bd7a4e Add resuming capability to GCS source (#1161)
* Add resuming capability to GCS source.

* Handle no auth scans.

* complete resume logic

* Use custom function type.

* remove functions.

* linter.

* fix test.

* fix test.

* Handle concurrent map writes.

* use string as CLI flag for include/exclude.

* handle emtpy buckets.

* Handle enumeration on initial job run.

* Rename stats to attributes.

* remove redundant return.

* If test fails due to 400, that is fine, it's expected.

* Add unauth GCS source type.

* comments.

* update proto.

* Use short flag.

* address comments.
2023-03-16 17:53:42 -07:00
ahrav 6193509098 add support for json service account and service account file. (#1185) 2023-03-16 13:04:36 -07:00
Zachary Rice f0b6b5d0d9 add a break statement when iterating through keywords (#1184) 2023-03-15 16:51:03 -05:00
ahrav cbf299aa77 Add gcs scanning integration (#1153)
* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* Remove period from file extension.

* remove used.

* Add comment.

* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* remove used.

* Add file type for objects.

* Add check for file type and size.

* Add default file size.

* Add additinoal auth options and remaining CLI flags.

* Handle errors in go routines.

* Handle resuming for buckets.

* Remove redundant words in comment.

* remove ok check on bool check.

* remove extra blank line.

* Add return if handler handles chunk.

* Add comment.

* remove extra blank line.

* cleanup comment.

* Add comment.

* move up fxn.

* go mod tidy.

* Add exclusion to perf testing buckets.

* Handle blocking the channel.

* remove unused const.

* fix tests.

* fix tests.

* Handle gcs manger options better.

* update fxn name.

* Remove arg name.

* ignore buckets in gcsManager test.

* fix test.

* propulate gsManagerOpts.

* inline err check.

* Add readme.

* update readme spelling.

* fix test.
2023-03-07 17:32:04 -08:00
Zachary Rice 4777b77ec6 Keyword optimization (#1144)
* init

* ignore trufflehog binary and added comment

* remove unused keywords in chunk, better comment

* remove keywords from engine struct
2023-03-02 11:32:37 -06:00
Miccah dd39848709 Add ability to include and exclude detectors (#1106)
* Add ability to include and exclude detectors

* Trim space before checking for empty items

* Explicitly check for integer overflow

* Use strconv.ParseInt instead of strconv.Atoi

* Address comments
2023-02-27 16:46:45 -06:00
Miccah c5b4d6f28b Support file scanning in filesystem source (#1030)
* Rename directories to paths

* Generate protos

* Add file scanning support to filesystem source

* Add directories back to filesystem proto

* Generate protos

* Combine paths and directories from in source

* Add filesystem filter

* Address comments
2023-02-27 12:15:05 -06:00
SAYGIN Metin f2139a7615 Github filter support for exclude and include (#1087)
* test

* Add missing head and base hash back.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-02-14 08:40:53 -08:00
trufflesteeeve 4f13090c01 Remove duplicated detectors (#1092)
In this case just Heroku and LinearAPI. But this includes the Moonclerck
detector, which appears to be a typo that got turned into a separate
detector type.

Co-authored-by: zubairk14 <zubair.khan@trufflesec.com>
2023-02-13 11:44:19 -05:00
Dustin Decker 0c66d30c1f Revert "Make detectors configurable (#1084)" (#1097)
This reverts commit 67784f6928.
2023-02-11 08:12:13 -08:00
ahrav 67784f6928 Make detectors configurable (#1084)
* Make detectors configurable.

* remove redundant check.

* add number of detectors.

* update comment.

* remove reflect.

* inline key.

* replace name w/ type.

* remove temp var.

* fix test name.

* fix engine start.

* add filter unverified to engine.

* reorder engine args.

* Address comments.

* Add include and exclude.

* update comments.

* add comment.

* add comment.
2023-02-10 16:30:38 -08:00
ahrav c5c8d10d28 [chore] - Remove monolithic config struct (#1091)
* REmove monolithic config struct.

* fix broken test.
2023-02-10 12:43:00 -08:00
Miccah d317ddb51a [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089)
* [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources

* Address comments
2023-02-10 11:02:55 -06:00
Miccah 0ce72ccda3 [chore] Remove logrus from github source (#1086)
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
2023-02-09 18:02:04 -06:00
Miccah 58e8c1e4ac [chore] Remove logrus from engine package (#1085) 2023-02-09 16:55:19 -06:00
Alexandr Marchenko b29b78c10d filesystem support for exclude and include filters (2nd attemp) (#1033)
* fix filter issue - empty lines should be ignored

* filesystem support for filter exclude

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-01-26 09:33:45 -08:00
ahrav 319ae64a02 [chore] - Small cleanup of CircleCi source (#1028)
* Small cleanup of CircleCi source.

* address comments.

* Add context to methods as first param.
2023-01-17 09:36:18 -08:00
Bill Rich 430d5c764c Rename and export isGitSource (#1016) 2023-01-10 12:51:58 -08:00
Bill Rich 8b2e1d36cf Copy metadata for line number aware sources (#1011)
* Copy metadata for line number aware sources

* Improve style
2023-01-10 09:35:44 -08:00
Dustin Decker 5f6143f09a Add Circle CI source (#997)
* Add Circle CI source

* remove SHA1 line

* remove trim
2023-01-05 21:44:37 -08:00
ahrav 936a139596 Allow using a glob for include list. (#977)
* Allow using a glob for include list.

* Update command flag.

* Make comment more clear.

* update comment.

* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
Bill Rich 335ce85ce4 Export line number code (#962) 2022-12-06 15:31:15 -08:00
Thiago Lages de Alencar 9757c339d9 Fix error message (#933) 2022-11-20 05:31:11 -08:00
Dustin Decker ae4b387448 add LDAP detector (#896) 2022-11-18 19:45:11 -08:00
Dustin Decker b18edef01a Enable skipping of particular key IDs (#930)
* Enable skipping of particular key IDs

* update test
2022-11-18 09:09:40 -08:00
Jessica 3d501975e4 Add filter as scan option to gitlab module's git scan (#919) 2022-11-15 13:02:37 -08:00
Bill Rich d3b24fa592 Replace plain decoder with utf8 (#922) 2022-11-15 09:36:01 -08:00
kstilwell ecd25784f5 Adding Shopify detector (#875)
* Fixes/work based on testing

* Remove some commented code

* Change how verification happens and grab additional information

* Address linter warnings.

* add shopify detector to default detectors.

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2022-11-08 16:21:57 -08:00
ahrav dd141fb55f [oc-147] - Add context to all git methods (#901)
* Add context to all git methods.

* remove logrus.

* Add ctx.

* Address comments.

* Add error to clone failing.

* Return error.
2022-11-03 16:36:52 -07:00
Dustin Decker a7fc12240f Do local URI verification, while attempting to defuse SSRF (#879)
* simplify monogo pattern

* do URI verification locally, while attempting to defuse SSRF

* test SSRF defuse

* simplify err check logic per linter recommendation

* split up detectors

* address comments

* remove unused var
2022-11-01 17:27:24 -07:00
ahrav fe029b1098 [THOG-793] - Return all unverified results (#856)
* Remove the check to filter and return only a single unverified result.

* Revert "Remove the check to filter and return only a single unverified result."

This reverts commit 494e432803.

* Add new CLI flag to filter unverified results.
2022-10-31 09:36:10 -07:00
Bill Rich 034ca4fb5b Add bytes counter to scans (#876) 2022-10-27 12:54:22 -07:00
Alexandr Marchenko 60464da3ce proposal: SqlServer connection string detector (#867)
* sqlserver added to detectors.proto

* make protos

* boilerplate detector generated

* wireup

* initial
2022-10-26 07:46:13 -07:00
Bill Rich 958266ea84 Run chunker in pipeline (#859)
* Run chunker in pipeline

* Move ChunkSize and PeekSize to source package.

* Use new Chunk and Peek size location
2022-10-24 13:57:27 -07:00
Ankush Goel d29357c9d4 added npm detector (#841) 2022-10-13 06:04:02 -07:00
Dustin Decker 85467538f6 remove faulty detector (#836) 2022-10-07 09:20:44 -07:00
ahrav 128002885a Add decoder type to results. (#835) 2022-10-06 11:55:07 -07:00
Miccah 2bc4985061 Add SSH config option for the git source (#830)
* Add SSH config option for the git source

The auth message is empty since we use the git binary underneath to
handle the SSH authentication.

* Import digitaloceanv2
2022-09-28 20:40:01 +02:00
Mildred Bernardo ad4b9406a7 Added digitaloceanv2 detector (#829)
* Added digitaloceanv2 detector

* import detector

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2022-09-28 09:56:35 -07:00
ahrav db42bcf2a2 [OC-103] - Add Gemini detector (#800)
* Add Gemini detector.

* Add regex and test code for Gemini detector.

* Remove else.

* Add commentary.

* Address comments.

* Use regular else.

* Make nice and complicated.

* use regular detection pattern.

* Add detector to default detectors.
2022-09-26 11:48:48 -07:00
Bill Rich e3107ad6bb Move head and base normalization to source (#818) 2022-09-23 08:58:45 -07:00
ahrav f2f5b6dba8 Fix recover. (#817) 2022-09-22 15:47:42 -07:00
ahrav 92f40c2031 [THOG-709] - Recover from detector panics (#810) 2022-09-22 07:01:10 -07:00
Dustin Decker e8f1bb9127 Import NGC detector 2022-09-21 16:26:58 -07:00
Bill Rich 509cf8b6fa Use headref and check empty commits for base (#815) 2022-09-21 16:04:01 -07:00
Dustin Decker fcd580406e A few improvements (#809)
* Run integration tests

* Update examples

* Import mongodb
2022-09-19 13:23:25 -07:00
Dustin Decker fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
Dustin Decker 2452e93a80 Import 27 new detectors (#737) 2022-08-26 12:35:06 -07:00