e391e89f3e
Initial implementation of JobReport with SourceManager usage ( #1557 )
...
* Initial implementation of JobReport with SourceManager usage
* Limit concurrent units
* Only save the last JobReport per handle
2023-07-27 10:49:56 -05:00
46823f77c9
feat(github): clarify comment log statement ( #1553 )
2023-07-26 09:40:30 -05:00
10f0963bc9
Add SourceManager tests for Run and Wait methods ( #1530 )
...
* Miscellaneous SourceManager updates
* Own the chunks channel instead of accepting it as an input
* Add Chunks and Wait methods
* Fix bug in Enroll so it actually returns the handle
* Add context.Context parameter to the SourceInitFunc type
* Add SourceManager tests for Run and Wait methods
* Rename man variables to mgr
2023-07-26 00:48:28 -05:00
2290954b02
fix(github): use apiEndpoint for basic or no auth ( #1454 )
2023-07-25 20:03:08 -07:00
f39303495a
Add commitsScanned metrics ( #1533 )
...
* Add commitsScanned metrics
* Just keep commit count
2023-07-25 11:31:01 -07:00
b5b01d3eba
[chore] - optimize chunker ( #1535 )
...
* Use chunkbytes that includes the size of peek.
* linter.
* continue.
* add TotalChunkSize const.
2023-07-24 19:30:29 -07:00
9e0a2e9ddd
[chore] - Remove password info from log ( #1528 )
...
* Remove password info from log.
* update.
* one more.
2023-07-22 20:25:45 -07:00
91c5472876
Implement SourceManager basics ( #1515 )
...
* Implement SourceManager basics
* Rename identifiers and add a default headlessAPI implementation
* Rewrite to use SourceInitFunc
* Update variable name to accurately reflect its value
2023-07-21 15:20:25 -05:00
4e774d1f01
Define SourceUnit chunking interface ( #1484 )
...
* Define SourceUnit chunking interface
* Refactor to use a ChunkReporter interface
* Rename shadowed err to scanErr
2023-07-13 14:11:43 -05:00
4b7f94dea1
Rewrite SourceUnitEnumerator to use UnitReporter instead of a channel ( #1485 )
2023-07-13 13:48:33 -05:00
1594fddf05
feat(git): include line in github & gitlab links ( #1466 )
2023-07-11 20:02:27 -07:00
4334af4d34
scan GitHub PR and issue comments ( #1435 )
...
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
5c0ffda618
Define SourceUnit enumeration interface ( #1428 )
...
* Add CancellableWrite helper function
* Create SourceUnitEnumerator interface and EnumerationResult struct
* Implement SourceUnitEnumerator for the filesystem Source
* Omit explicit zero values
2023-07-10 15:05:40 -05:00
452734adc8
remove head from git diff command, rename unstaged to staged ( #1439 )
2023-06-29 15:33:30 -05:00
4a77688097
use stringer again for now ( #1430 )
2023-06-26 14:33:54 -05:00
11bff81def
Use url redaction in git ( #1399 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com >
2023-06-26 13:56:08 -05:00
da5301ea1e
Exit with non-zero exit code on chunk source error ( #1286 )
...
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
Co-authored-by: ahrav <ahravdutta02@gmail.com >
2023-06-26 11:39:57 -05:00
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
e856a6890d
🎉 Add Docker image scanning 🎉 ( #1412 )
...
* Add Docker source
* Add metrics
* Add test
* Add debugging, address PR comments, fix path output
* review suggestions
2023-06-22 08:02:25 -07:00
648ef3b52c
fix spelling errors ( #1413 )
2023-06-21 07:15:28 -07:00
e12f0f84a1
Setup SourceUnit interface ( #1393 )
...
* Test: Asymmetrical unmarshal API
* Test: Symmetric marshal API
* Revert "Test: Symmetric marshal API"
This reverts commit f51c64a797
2023-06-16 10:38:28 -05:00
401688d0a8
Add Validator interface and example ( #1397 )
...
* Add Validator interface and example
* Close sockets and improve error messages
* Remove duplicate error
* Use var declaration so err slice can be nil
2023-06-15 08:24:32 -07:00
c2e3e7d53a
Split files instead of using ReadAll ( #1387 )
...
* Split files instead of using ReadAll
* Remove dup chunk
* Actually break out of loop
2023-06-12 14:09:05 -07:00
572cb0e5dc
Loosen up version check for git
2023-06-01 12:17:48 -07:00
183037ab34
Check that git meets version requirements ( #1373 )
2023-06-01 09:41:06 -07:00
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
6db770fbe5
use md5 hash for checking if key exists. ( #1257 )
2023-05-15 10:04:14 -07:00
948828ba8c
[chore] - move objectManager interface ( #1332 )
...
* Relocate the objectManager interface to the consumer package as per Go
best practices.
* address comment.
2023-05-15 09:30:26 -07:00
fad34d4dc6
git worktree scanning fix for #827 ( #1315 )
...
* Fix worktree scan by setting EnableDotGitCommonDir
* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
In every current usage of this function, it is on an already-cloned
repository, so it should always be valid to have this set. By doing
so, it should fix some issues with worktrees.
* Remove unused go.mod replace directives
* Remove replace directives for libraries that are not in use.
2023-05-09 08:00:47 -07:00
f2924f3061
Make sure context lines are properly handled ( #1331 )
...
* Make sure context lines are properly handled
* Fix git test to account for context change
2023-05-05 12:51:27 -07:00
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
be4147a24e
Output git timestamps as UTC times ( #1323 )
2023-05-03 11:47:00 -05:00
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
67972683ea
[chore] - format log msg ( #1299 )
...
* format log msg.
* snake.
* lowercase repo.
2023-04-27 17:14:00 -07:00
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
10902f802a
Add max object size flag for s3 bucket scanning ( #1294 )
...
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-26 15:39:43 -07:00
97ce27153a
[]bytes were being logged as b64ed string ( #1255 )
2023-04-14 06:43:26 -07:00
461f1a631e
[chore] - use hex encode vs base64 ( #1256 )
...
* use hex encode vs base64.
* fix tests.
2023-04-13 19:16:06 -07:00
2fbf86a6ab
Use md5 hash for resuming key ( #1203 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Include md5 hash to the object struct.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* update comment.
add comment for shouldCache.
remove redundant return.
* use md5 instead of name.
* update tests.
* Include md5 hash to the object struct.
* use md5 instead of name.
* update tests.
* Use a persistable cache.
* fix merge.
* fix merge.
* Include md5 hash to the object struct.
* use md5 instead of name.
* update tests.
* use md5 instead of name.
* update progress tests.
* use name for log message.
* remove slice operation.
2023-04-13 18:26:45 -07:00
1db22599af
update circle test because workflows expire and need re-running ( #1251 )
2023-04-10 16:21:19 -07:00
c451f9daf8
Use persistable cache for GCS progress tracking ( #1204 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* Use a persistable cache.
* fix merge.
* fix merge.
* Add progress as part of the cache given it will be the persistence layer.
* Add test for making sure the cache doesn't persist when the increment value is not met.
* fix tests.
2023-04-10 07:55:00 -07:00
b3d917f9c7
Resolve #1167 by adding support for the AWS_SESSION_TOKEN ( #1170 )
...
* Resolve #1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
2023-04-03 14:56:43 -07:00
2cf6f831d4
Use OAuth2 http client with GCS ( #1220 )
...
* Use OAuth2 http client with GCS.
* rename variable.
2023-03-29 19:40:27 -07:00
fb9ae75661
Support for exclude globs at the git log level ( #1202 )
...
* init
* seems to be working
* better comment
* rm conditional
* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
ac19de75bf
Delete progress tracking from GCS source ( #1190 )
...
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* update comment.
add comment for shouldCache.
remove redundant return.
* delete old code.
* delete more code.
* update comment.
2023-03-27 10:39:16 -07:00
03a534d59f
Use correct date format for Date posted. ( #1211 )
2023-03-27 10:27:28 -07:00
ffbd9c1ead
[chore] - log enumeration duration ( #1187 )
...
* log enumeration duration.
* use defer to print enumeration duration stat.
* remove temp var.
2023-03-21 09:14:58 -07:00
Miccah