9e0a2e9ddd
[chore] - Remove password info from log ( #1528 )
...
* Remove password info from log.
* update.
* one more.
2023-07-22 20:25:45 -07:00
91c5472876
Implement SourceManager basics ( #1515 )
...
* Implement SourceManager basics
* Rename identifiers and add a default headlessAPI implementation
* Rewrite to use SourceInitFunc
* Update variable name to accurately reflect its value
2023-07-21 15:20:25 -05:00
4e774d1f01
Define SourceUnit chunking interface ( #1484 )
...
* Define SourceUnit chunking interface
* Refactor to use a ChunkReporter interface
* Rename shadowed err to scanErr
2023-07-13 14:11:43 -05:00
4b7f94dea1
Rewrite SourceUnitEnumerator to use UnitReporter instead of a channel ( #1485 )
2023-07-13 13:48:33 -05:00
1594fddf05
feat(git): include line in github & gitlab links ( #1466 )
2023-07-11 20:02:27 -07:00
4334af4d34
scan GitHub PR and issue comments ( #1435 )
...
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
5c0ffda618
Define SourceUnit enumeration interface ( #1428 )
...
* Add CancellableWrite helper function
* Create SourceUnitEnumerator interface and EnumerationResult struct
* Implement SourceUnitEnumerator for the filesystem Source
* Omit explicit zero values
2023-07-10 15:05:40 -05:00
452734adc8
remove head from git diff command, rename unstaged to staged ( #1439 )
2023-06-29 15:33:30 -05:00
4a77688097
use stringer again for now ( #1430 )
2023-06-26 14:33:54 -05:00
11bff81def
Use url redaction in git ( #1399 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com >
2023-06-26 13:56:08 -05:00
da5301ea1e
Exit with non-zero exit code on chunk source error ( #1286 )
...
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
Co-authored-by: ahrav <ahravdutta02@gmail.com >
2023-06-26 11:39:57 -05:00
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
e856a6890d
🎉 Add Docker image scanning 🎉 ( #1412 )
...
* Add Docker source
* Add metrics
* Add test
* Add debugging, address PR comments, fix path output
* review suggestions
2023-06-22 08:02:25 -07:00
648ef3b52c
fix spelling errors ( #1413 )
2023-06-21 07:15:28 -07:00
e12f0f84a1
Setup SourceUnit interface ( #1393 )
...
* Test: Asymmetrical unmarshal API
* Test: Symmetric marshal API
* Revert "Test: Symmetric marshal API"
This reverts commit f51c64a797
2023-06-16 10:38:28 -05:00
401688d0a8
Add Validator interface and example ( #1397 )
...
* Add Validator interface and example
* Close sockets and improve error messages
* Remove duplicate error
* Use var declaration so err slice can be nil
2023-06-15 08:24:32 -07:00
c2e3e7d53a
Split files instead of using ReadAll ( #1387 )
...
* Split files instead of using ReadAll
* Remove dup chunk
* Actually break out of loop
2023-06-12 14:09:05 -07:00
572cb0e5dc
Loosen up version check for git
2023-06-01 12:17:48 -07:00
183037ab34
Check that git meets version requirements ( #1373 )
2023-06-01 09:41:06 -07:00
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
6db770fbe5
use md5 hash for checking if key exists. ( #1257 )
2023-05-15 10:04:14 -07:00
948828ba8c
[chore] - move objectManager interface ( #1332 )
...
* Relocate the objectManager interface to the consumer package as per Go
best practices.
* address comment.
2023-05-15 09:30:26 -07:00
fad34d4dc6
git worktree scanning fix for #827 ( #1315 )
...
* Fix worktree scan by setting EnableDotGitCommonDir
* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
In every current usage of this function, it is on an already-cloned
repository, so it should always be valid to have this set. By doing
so, it should fix some issues with worktrees.
* Remove unused go.mod replace directives
* Remove replace directives for libraries that are not in use.
2023-05-09 08:00:47 -07:00
f2924f3061
Make sure context lines are properly handled ( #1331 )
...
* Make sure context lines are properly handled
* Fix git test to account for context change
2023-05-05 12:51:27 -07:00
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
be4147a24e
Output git timestamps as UTC times ( #1323 )
2023-05-03 11:47:00 -05:00
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
67972683ea
[chore] - format log msg ( #1299 )
...
* format log msg.
* snake.
* lowercase repo.
2023-04-27 17:14:00 -07:00
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
10902f802a
Add max object size flag for s3 bucket scanning ( #1294 )
...
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-26 15:39:43 -07:00
97ce27153a
[]bytes were being logged as b64ed string ( #1255 )
2023-04-14 06:43:26 -07:00
461f1a631e
[chore] - use hex encode vs base64 ( #1256 )
...
* use hex encode vs base64.
* fix tests.
2023-04-13 19:16:06 -07:00
2fbf86a6ab
Use md5 hash for resuming key ( #1203 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Include md5 hash to the object struct.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* update comment.
add comment for shouldCache.
remove redundant return.
* use md5 instead of name.
* update tests.
* Include md5 hash to the object struct.
* use md5 instead of name.
* update tests.
* Use a persistable cache.
* fix merge.
* fix merge.
* Include md5 hash to the object struct.
* use md5 instead of name.
* update tests.
* use md5 instead of name.
* update progress tests.
* use name for log message.
* remove slice operation.
2023-04-13 18:26:45 -07:00
1db22599af
update circle test because workflows expire and need re-running ( #1251 )
2023-04-10 16:21:19 -07:00
c451f9daf8
Use persistable cache for GCS progress tracking ( #1204 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* Use a persistable cache.
* fix merge.
* fix merge.
* Add progress as part of the cache given it will be the persistence layer.
* Add test for making sure the cache doesn't persist when the increment value is not met.
* fix tests.
2023-04-10 07:55:00 -07:00
b3d917f9c7
Resolve #1167 by adding support for the AWS_SESSION_TOKEN ( #1170 )
...
* Resolve #1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-03 14:56:43 -07:00
2cf6f831d4
Use OAuth2 http client with GCS ( #1220 )
...
* Use OAuth2 http client with GCS.
* rename variable.
2023-03-29 19:40:27 -07:00
fb9ae75661
Support for exclude globs at the git log level ( #1202 )
...
* init
* seems to be working
* better comment
* rm conditional
* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
ac19de75bf
Delete progress tracking from GCS source ( #1190 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* update comment.
add comment for shouldCache.
remove redundant return.
* delete old code.
* delete more code.
* update comment.
2023-03-27 10:39:16 -07:00
03a534d59f
Use correct date format for Date posted. ( #1211 )
2023-03-27 10:27:28 -07:00
ffbd9c1ead
[chore] - log enumeration duration ( #1187 )
...
* log enumeration duration.
* use defer to print enumeration duration stat.
* remove temp var.
2023-03-21 09:14:58 -07:00
c617bd7a4e
Add resuming capability to GCS source ( #1161 )
...
* Add resuming capability to GCS source.
* Handle no auth scans.
* complete resume logic
* Use custom function type.
* remove functions.
* linter.
* fix test.
* fix test.
* Handle concurrent map writes.
* use string as CLI flag for include/exclude.
* handle emtpy buckets.
* Handle enumeration on initial job run.
* Rename stats to attributes.
* remove redundant return.
* If test fails due to 400, that is fine, it's expected.
* Add unauth GCS source type.
* comments.
* update proto.
* Use short flag.
* address comments.
2023-03-16 17:53:42 -07:00
6193509098
add support for json service account and service account file. ( #1185 )
2023-03-16 13:04:36 -07:00
ef9488c77d
[chore] Log git output on error ( #1180 )
2023-03-15 15:32:29 -05:00
a7abd6231d
Fix git commit date string formatting ( #1181 )
2023-03-14 22:39:12 -05:00
585bd82d47
update integration test excludes ( #1169 )
2023-03-10 14:41:29 -08:00
cbf299aa77
Add gcs scanning integration ( #1153 )
...
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* Remove period from file extension.
* remove used.
* Add comment.
* Setup for GCS scanning.
* Update GCS engine w/ projectID req.
* Add concurrency field to gcsManager.
* add errgroup to gcsManager.
* Update gcs manager.
* Use defautl ADC.
* use ADC.'
* Add TOOD.
* add log to iterator completion.
* use a BinaryReader instead of concrete object for channel type.
* initial test for Chunks.
* Add tests for chunking objects.
* Add concurrency.
* update metadata to include content type and acls.
* Add object reading code.
* Add integration test.
* Add entrypoint.
* Add removed wg.Wait().
* remove dead code.
* remove build.
* remove used.
* Add file type for objects.
* Add check for file type and size.
* Add default file size.
* Add additinoal auth options and remaining CLI flags.
* Handle errors in go routines.
* Handle resuming for buckets.
* Remove redundant words in comment.
* remove ok check on bool check.
* remove extra blank line.
* Add return if handler handles chunk.
* Add comment.
* remove extra blank line.
* cleanup comment.
* Add comment.
* move up fxn.
* go mod tidy.
* Add exclusion to perf testing buckets.
* Handle blocking the channel.
* remove unused const.
* fix tests.
* fix tests.
* Handle gcs manger options better.
* update fxn name.
* Remove arg name.
* ignore buckets in gcsManager test.
* fix test.
* propulate gsManagerOpts.
* inline err check.
* Add readme.
* update readme spelling.
* fix test.
2023-03-07 17:32:04 -08:00
ahrav