garg472
3e4496156c
added new detectors and fixed mesibo detector ( #1166 )
...
* added new detectors and fixed mesibo detector
* added bscscan.com API detector
* added coinmarketcap detector
* update alchemy
* update blocknative
* update bscscan test
* update cmc test
* update tests
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-03-16 18:57:08 -07:00
Yassine Ilmi
d382d5cb1c
Add OpenAI API Tokens detector ( #1142 )
...
* Add OpenAI Detector
* Add OpenAI Detector tests
* Add OpenAI Detector to defaults.go
* Removing references to github detector in tests
2023-03-16 17:58:22 -07:00
ahrav
c617bd7a4e
Add resuming capability to GCS source ( #1161 )
...
* Add resuming capability to GCS source.
* Handle no auth scans.
* complete resume logic
* Use custom function type.
* remove functions.
* linter.
* fix test.
* fix test.
* Handle concurrent map writes.
* use string as CLI flag for include/exclude.
* handle emtpy buckets.
* Handle enumeration on initial job run.
* Rename stats to attributes.
* remove redundant return.
* If test fails due to 400, that is fine, it's expected.
* Add unauth GCS source type.
* comments.
* update proto.
* Use short flag.
* address comments.
2023-03-16 17:53:42 -07:00
ahrav
6193509098
add support for json service account and service account file. ( #1185 )
2023-03-16 13:04:36 -07:00
Zachary Rice
f0b6b5d0d9
add a break statement when iterating through keywords ( #1184 )
2023-03-15 16:51:03 -05:00
ahrav
cbf299aa77
Add gcs scanning integration ( #1153 )
...
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* Remove period from file extension.
* remove used.
* Add comment.
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* remove used.
* Add file type for objects.
* Add check for file type and size.
* Add default file size.
* Add additinoal auth options and remaining CLI flags.
* Handle errors in go routines.
* Handle resuming for buckets.
* Remove redundant words in comment.
* remove ok check on bool check.
* remove extra blank line.
* Add return if handler handles chunk.
* Add comment.
* remove extra blank line.
* cleanup comment.
* Add comment.
* move up fxn.
* go mod tidy.
* Add exclusion to perf testing buckets.
* Handle blocking the channel.
* remove unused const.
* fix tests.
* fix tests.
* Handle gcs manger options better.
* update fxn name.
* Remove arg name.
* ignore buckets in gcsManager test.
* fix test.
* propulate gsManagerOpts.
* inline err check.
* Add readme.
* update readme spelling.
* fix test.
2023-03-07 17:32:04 -08:00
Zachary Rice
4777b77ec6
Keyword optimization ( #1144 )
...
* init
* ignore trufflehog binary and added comment
* remove unused keywords in chunk, better comment
* remove keywords from engine struct
2023-03-02 11:32:37 -06:00
Miccah
dd39848709
Add ability to include and exclude detectors ( #1106 )
...
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
2023-02-27 16:46:45 -06:00
Miccah
c5b4d6f28b
Support file scanning in filesystem source ( #1030 )
...
* Rename directories to paths
* Generate protos
* Add file scanning support to filesystem source
* Add directories back to filesystem proto
* Generate protos
* Combine paths and directories from in source
* Add filesystem filter
* Address comments
2023-02-27 12:15:05 -06:00
SAYGIN Metin
f2139a7615
Github filter support for exclude and include ( #1087 )
...
* test
* Add missing head and base hash back.
---------
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com >
2023-02-14 08:40:53 -08:00
trufflesteeeve
4f13090c01
Remove duplicated detectors ( #1092 )
...
In this case just Heroku and LinearAPI. But this includes the Moonclerck
detector, which appears to be a typo that got turned into a separate
detector type.
Co-authored-by: zubairk14 <zubair.khan@trufflesec.com >
2023-02-13 11:44:19 -05:00
Dustin Decker
0c66d30c1f
Revert "Make detectors configurable ( #1084 )" ( #1097 )
...
This reverts commit 67784f6928 .
2023-02-11 08:12:13 -08:00
ahrav
67784f6928
Make detectors configurable ( #1084 )
...
* Make detectors configurable.
* remove redundant check.
* add number of detectors.
* update comment.
* remove reflect.
* inline key.
* replace name w/ type.
* remove temp var.
* fix test name.
* fix engine start.
* add filter unverified to engine.
* reorder engine args.
* Address comments.
* Add include and exclude.
* update comments.
* add comment.
* add comment.
2023-02-10 16:30:38 -08:00
ahrav
c5c8d10d28
[chore] - Remove monolithic config struct ( #1091 )
...
* REmove monolithic config struct.
* fix broken test.
2023-02-10 12:43:00 -08:00
Miccah
d317ddb51a
[chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources ( #1089 )
...
* [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources
* Address comments
2023-02-10 11:02:55 -06:00
Miccah
0ce72ccda3
[chore] Remove logrus from github source ( #1086 )
...
* [chore] Remove logrus from github source
* Fix handleRateLimit test
* Fix tests
2023-02-09 18:02:04 -06:00
Miccah
58e8c1e4ac
[chore] Remove logrus from engine package ( #1085 )
2023-02-09 16:55:19 -06:00
Alexandr Marchenko
b29b78c10d
filesystem support for exclude and include filters (2nd attemp) ( #1033 )
...
* fix filter issue - empty lines should be ignored
* filesystem support for filter exclude
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-01-26 09:33:45 -08:00
ahrav
319ae64a02
[chore] - Small cleanup of CircleCi source ( #1028 )
...
* Small cleanup of CircleCi source.
* address comments.
* Add context to methods as first param.
2023-01-17 09:36:18 -08:00
Bill Rich
430d5c764c
Rename and export isGitSource ( #1016 )
2023-01-10 12:51:58 -08:00
Bill Rich
8b2e1d36cf
Copy metadata for line number aware sources ( #1011 )
...
* Copy metadata for line number aware sources
* Improve style
2023-01-10 09:35:44 -08:00
Dustin Decker
5f6143f09a
Add Circle CI source ( #997 )
...
* Add Circle CI source
* remove SHA1 line
* remove trim
2023-01-05 21:44:37 -08:00
ahrav
936a139596
Allow using a glob for include list. ( #977 )
...
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
Bill Rich
335ce85ce4
Export line number code ( #962 )
2022-12-06 15:31:15 -08:00
Thiago Lages de Alencar
9757c339d9
Fix error message ( #933 )
2022-11-20 05:31:11 -08:00
Dustin Decker
ae4b387448
add LDAP detector ( #896 )
2022-11-18 19:45:11 -08:00
Dustin Decker
b18edef01a
Enable skipping of particular key IDs ( #930 )
...
* Enable skipping of particular key IDs
* update test
2022-11-18 09:09:40 -08:00
Jessica
3d501975e4
Add filter as scan option to gitlab module's git scan ( #919 )
2022-11-15 13:02:37 -08:00
Bill Rich
d3b24fa592
Replace plain decoder with utf8 ( #922 )
2022-11-15 09:36:01 -08:00
kstilwell
ecd25784f5
Adding Shopify detector ( #875 )
...
* Fixes/work based on testing
* Remove some commented code
* Change how verification happens and grab additional information
* Address linter warnings.
* add shopify detector to default detectors.
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com >
2022-11-08 16:21:57 -08:00
ahrav
dd141fb55f
[oc-147] - Add context to all git methods ( #901 )
...
* Add context to all git methods.
* remove logrus.
* Add ctx.
* Address comments.
* Add error to clone failing.
* Return error.
2022-11-03 16:36:52 -07:00
Dustin Decker
a7fc12240f
Do local URI verification, while attempting to defuse SSRF ( #879 )
...
* simplify monogo pattern
* do URI verification locally, while attempting to defuse SSRF
* test SSRF defuse
* simplify err check logic per linter recommendation
* split up detectors
* address comments
* remove unused var
2022-11-01 17:27:24 -07:00
ahrav
fe029b1098
[THOG-793] - Return all unverified results ( #856 )
...
* Remove the check to filter and return only a single unverified result.
* Revert "Remove the check to filter and return only a single unverified result."
This reverts commit 494e432803 .
* Add new CLI flag to filter unverified results.
2022-10-31 09:36:10 -07:00
Bill Rich
034ca4fb5b
Add bytes counter to scans ( #876 )
2022-10-27 12:54:22 -07:00
Alexandr Marchenko
60464da3ce
proposal: SqlServer connection string detector ( #867 )
...
* sqlserver added to detectors.proto
* make protos
* boilerplate detector generated
* wireup
* initial
2022-10-26 07:46:13 -07:00
Bill Rich
958266ea84
Run chunker in pipeline ( #859 )
...
* Run chunker in pipeline
* Move ChunkSize and PeekSize to source package.
* Use new Chunk and Peek size location
2022-10-24 13:57:27 -07:00
Ankush Goel
d29357c9d4
added npm detector ( #841 )
2022-10-13 06:04:02 -07:00
Dustin Decker
85467538f6
remove faulty detector ( #836 )
2022-10-07 09:20:44 -07:00
ahrav
128002885a
Add decoder type to results. ( #835 )
2022-10-06 11:55:07 -07:00
Miccah
2bc4985061
Add SSH config option for the git source ( #830 )
...
* Add SSH config option for the git source
The auth message is empty since we use the git binary underneath to
handle the SSH authentication.
* Import digitaloceanv2
2022-09-28 20:40:01 +02:00
Mildred Bernardo
ad4b9406a7
Added digitaloceanv2 detector ( #829 )
...
* Added digitaloceanv2 detector
* import detector
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2022-09-28 09:56:35 -07:00
ahrav
db42bcf2a2
[OC-103] - Add Gemini detector ( #800 )
...
* Add Gemini detector.
* Add regex and test code for Gemini detector.
* Remove else.
* Add commentary.
* Address comments.
* Use regular else.
* Make nice and complicated.
* use regular detection pattern.
* Add detector to default detectors.
2022-09-26 11:48:48 -07:00
Bill Rich
e3107ad6bb
Move head and base normalization to source ( #818 )
2022-09-23 08:58:45 -07:00
ahrav
f2f5b6dba8
Fix recover. ( #817 )
2022-09-22 15:47:42 -07:00
ahrav
92f40c2031
[THOG-709] - Recover from detector panics ( #810 )
2022-09-22 07:01:10 -07:00
Dustin Decker
e8f1bb9127
Import NGC detector
2022-09-21 16:26:58 -07:00
Bill Rich
509cf8b6fa
Use headref and check empty commits for base ( #815 )
2022-09-21 16:04:01 -07:00
Dustin Decker
fcd580406e
A few improvements ( #809 )
...
* Run integration tests
* Update examples
* Import mongodb
2022-09-19 13:23:25 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
Dustin Decker
2452e93a80
Import 27 new detectors ( #737 )
2022-08-26 12:35:06 -07:00