a4b1fb7752
create hidden debug flag to disable overseer ( #1582 )
...
* add in new debug flag
* keep localdev local
2023-07-31 22:03:59 -04:00
5e7a6ca11c
Concurrent detection ( #1580 )
...
* Run detection on each chunk concurrently.
* Add printer functionality.
* Add logic for dedupe.
* cleanup.
* Moddify number of notifier workers.
* Add comment.
* move consts into fxn.
* buffer resutls chan.
* fix test.
* address comments.
* return an error from Finish.
* fix test.
* fix test.
* linter.
* check err.
* address comments.
2023-07-31 11:12:08 -07:00
69515bb7ac
Correctly route pprof endpoint ( #1527 )
2023-07-21 22:45:27 -05:00
da5301ea1e
Exit with non-zero exit code on chunk source error ( #1286 )
...
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
Co-authored-by: ahrav <ahravdutta02@gmail.com >
2023-06-26 11:39:57 -05:00
e856a6890d
🎉 Add Docker image scanning 🎉 ( #1412 )
...
* Add Docker source
* Add metrics
* Add test
* Add debugging, address PR comments, fix path output
* review suggestions
2023-06-22 08:02:25 -07:00
648ef3b52c
fix spelling errors ( #1413 )
2023-06-21 07:15:28 -07:00
fd4b5d1d14
remove gorilla mux ( #1411 )
2023-06-20 17:07:03 -07:00
b1675194ca
Implement EndpointCustomizer ( #1291 )
...
* Implement EndpointCustomizer
Add the EndpointCustomizer interface and EndpointSetter convenience struct,
implement EndpointCustomizer for github and gitlab detectors, and add
parsing, verification, and applying user-supplied configuration.
* Check error from SetEndpoints
* Rename variable for clarity
2023-04-27 12:23:50 -05:00
10902f802a
Add max object size flag for s3 bucket scanning ( #1294 )
...
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-26 15:39:43 -07:00
5a86c18302
Fix include and exclude detector logic ( #1267 )
...
* Fix include and exclude detector logic
* Fix test
* Add more clarifying comments
2023-04-26 10:49:54 -05:00
0507f0eb87
Only add detectors once ( #1265 )
2023-04-17 14:10:13 -07:00
b3d917f9c7
Resolve #1167 by adding support for the AWS_SESSION_TOKEN ( #1170 )
...
* Resolve #1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-03 14:56:43 -07:00
0052f60090
Allow for custom verifier ( #1070 )
...
* allow for custom verifier.
* Update engine.
* use custom detectors.
* set cap.
* Update verifiers.
* Remove nil check.
* resolved nit
* handle uppercase values
* updating missing url logs
* adding more descriptive variable names
* updating logs to use correct variables
* Removing toLower for urls
* if else nits
* Adding versioning for github and gitlab
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com >
Co-authored-by: ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com >
2023-03-29 12:26:39 -07:00
cb454bfc05
Add GitHub Actions output ( #1201 )
...
* Add GitHub Actions output
Co-authored-by: Mike Vanbuskirk <mike.vanbuskirk@trufflesec.com >
2023-03-28 09:07:26 -07:00
fb9ae75661
Support for exclude globs at the git log level ( #1202 )
...
* init
* seems to be working
* better comment
* rm conditional
* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
c617bd7a4e
Add resuming capability to GCS source ( #1161 )
...
* Add resuming capability to GCS source.
* Handle no auth scans.
* complete resume logic
* Use custom function type.
* remove functions.
* linter.
* fix test.
* fix test.
* Handle concurrent map writes.
* use string as CLI flag for include/exclude.
* handle emtpy buckets.
* Handle enumeration on initial job run.
* Rename stats to attributes.
* remove redundant return.
* If test fails due to 400, that is fine, it's expected.
* Add unauth GCS source type.
* comments.
* update proto.
* Use short flag.
* address comments.
2023-03-16 17:53:42 -07:00
cbf299aa77
Add gcs scanning integration ( #1153 )
...
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* Remove period from file extension.
* remove used.
* Add comment.
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* remove used.
* Add file type for objects.
* Add check for file type and size.
* Add default file size.
* Add additinoal auth options and remaining CLI flags.
* Handle errors in go routines.
* Handle resuming for buckets.
* Remove redundant words in comment.
* remove ok check on bool check.
* remove extra blank line.
* Add return if handler handles chunk.
* Add comment.
* remove extra blank line.
* cleanup comment.
* Add comment.
* move up fxn.
* go mod tidy.
* Add exclusion to perf testing buckets.
* Handle blocking the channel.
* remove unused const.
* fix tests.
* fix tests.
* Handle gcs manger options better.
* update fxn name.
* Remove arg name.
* ignore buckets in gcsManager test.
* fix test.
* propulate gsManagerOpts.
* inline err check.
* Add readme.
* update readme spelling.
* fix test.
2023-03-07 17:32:04 -08:00
e6846ede54
Support filtering detectors by version ( #1150 )
...
* Adjust types to use DetectorID struct
* Parse versions with detector include and exclude input
* Update detectors filter to use version
Co-authored-by: steeeve <steve@trufflesec.com >
* Implement Versioner for github, gitlab, and npm detectors
Co-authored-by: steeeve <steve@trufflesec.com >
---------
Co-authored-by: steeeve <steve@trufflesec.com >
2023-03-02 16:33:56 -06:00
0cf9139df6
Disable profiler in debug mode and add profile switch ( #1136 )
2023-02-28 12:49:54 -08:00
dd39848709
Add ability to include and exclude detectors ( #1106 )
...
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
2023-02-27 16:46:45 -06:00
c5b4d6f28b
Support file scanning in filesystem source ( #1030 )
...
* Rename directories to paths
* Generate protos
* Add file scanning support to filesystem source
* Add directories back to filesystem proto
* Generate protos
* Combine paths and directories from in source
* Add filesystem filter
* Address comments
2023-02-27 12:15:05 -06:00
012fdfe3a2
Update helper text for max-archive-size. ( #1114 )
2023-02-16 13:56:55 -08:00
ea71756e20
[chore] - archive size helper text ( #1110 )
2023-02-15 10:08:26 -08:00
ea40c0f306
Add the unit for max archive size. ( #1108 )
2023-02-15 09:45:27 -08:00
161e499142
[chore] Remove logrus from trufflehog ( #1095 )
...
* [chore] Remove logrus from trufflehog
* Minor fixes
* Fix logFatal call
* Fix logrus call
2023-02-14 17:00:07 -06:00
f2139a7615
Github filter support for exclude and include ( #1087 )
...
* test
* Add missing head and base hash back.
---------
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com >
2023-02-14 08:40:53 -08:00
0c66d30c1f
Revert "Make detectors configurable ( #1084 )" ( #1097 )
...
This reverts commit 67784f6928
2023-02-11 08:12:13 -08:00
67784f6928
Make detectors configurable ( #1084 )
...
* Make detectors configurable.
* remove redundant check.
* add number of detectors.
* update comment.
* remove reflect.
* inline key.
* replace name w/ type.
* remove temp var.
* fix test name.
* fix engine start.
* add filter unverified to engine.
* reorder engine args.
* Address comments.
* Add include and exclude.
* update comments.
* add comment.
* add comment.
2023-02-10 16:30:38 -08:00
c5c8d10d28
[chore] - Remove monolithic config struct ( #1091 )
...
* REmove monolithic config struct.
* fix broken test.
2023-02-10 12:43:00 -08:00
58e8c1e4ac
[chore] Remove logrus from engine package ( #1085 )
2023-02-09 16:55:19 -06:00
7dd2b74f1f
Make archive handler configurable ( #1077 )
...
* Make archive handler configurable.
* Use common.IsDone()
2023-02-07 15:25:14 -08:00
b29b78c10d
filesystem support for exclude and include filters (2nd attemp) ( #1033 )
...
* fix filter issue - empty lines should be ignored
* filesystem support for filter exclude
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-01-26 09:33:45 -08:00
5f6143f09a
Add Circle CI source ( #997 )
...
* Add Circle CI source
* remove SHA1 line
* remove trim
2023-01-05 21:44:37 -08:00
330a6f7cdc
Removing Debug version Println to logrus debug ( #993 )
2023-01-03 10:36:27 -06:00
8859771a2a
Remove custom log leveler ( #985 )
...
Instead of manually using a log leveler, rely on the global one defined
in the `log` package.
2022-12-20 19:03:53 -06:00
f5b83ee2a5
Add configuration parsing and custom detectors to engine ( #968 )
...
* Add configuration parsing for custom detectors
* Error on empty filename
2022-12-20 10:14:49 -06:00
936a139596
Allow using a glob for include list. ( #977 )
...
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
36ca2601e0
Add s3 object count to trace logs ( #975 )
...
* Add s3 object count to trace logs
* fix debug level
2022-12-13 16:46:09 -08:00
3b055ce3f9
Add logger to context ( #947 )
...
* Add logger to context
* Fatal on no org
2022-11-30 11:10:05 -08:00
3d501975e4
Add filter as scan option to gitlab module's git scan ( #919 )
2022-11-15 13:02:37 -08:00
dd141fb55f
[oc-147] - Add context to all git methods ( #901 )
...
* Add context to all git methods.
* remove logrus.
* Add ctx.
* Address comments.
* Add error to clone failing.
* Return error.
2022-11-03 16:36:52 -07:00
fe029b1098
[THOG-793] - Return all unverified results ( #856 )
...
* Remove the check to filter and return only a single unverified result.
* Revert "Remove the check to filter and return only a single unverified result."
This reverts commit 494e432803
2022-10-31 09:36:10 -07:00
034ca4fb5b
Add bytes counter to scans ( #876 )
2022-10-27 12:54:22 -07:00
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
c8ac7c392e
Allow secrets to be passed as env vars ( #736 )
2022-08-26 09:37:16 -07:00
0d8a154330
Add ssh:/ URI to docs ( #732 )
2022-08-23 23:26:09 -07:00
5ad3bbde37
Use pointer to config ( #715 )
2022-08-16 09:15:25 -07:00
73f9d3f0a0
[chore] - Use config struct instead of pointer for engine scans. ( #709 )
...
* Use a config struct instead of pointer when scanning engine sources.
* use config.
2022-08-12 09:56:24 -07:00
dcc102a81c
[Thog-371] Utilize config struct for engine scans ( #700 )
...
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
2022-08-10 10:11:13 -07:00
fc18a5ae0c
Bug fix and add authentication in shallow clone ( #595 )
2022-05-31 20:45:28 -05:00
Zubair Khan