1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
0ce72ccda3
[chore] Remove logrus from github source ( #1086 )
...
* [chore] Remove logrus from github source
* Fix handleRateLimit test
* Fix tests
2023-02-09 18:02:04 -06:00
4ef546a06b
fix github integration tests ( #1042 )
2023-01-25 08:57:39 -08:00
936a139596
Allow using a glob for include list. ( #977 )
...
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
7de9bdd12d
Support globbing with ignore repos ( #967 )
2022-12-09 12:10:42 -08:00
a72b9feb35
Only scan org with --org flag. ( #931 )
2022-12-06 16:18:48 -08:00
46bc010165
Add tests for including github repos. ( #854 )
2022-10-21 07:56:36 -07:00
2d6aadcb46
[THOG-774] - GitHub ignore repo full name ( #848 )
...
* Use github repo full name.
* fix tests.
2022-10-14 09:20:49 -07:00
04c9bb535e
[THOG-768] - Add ability to skip scanning Github repos ( #846 )
...
* Add ability to skip scanning Github repos.
* remove old change.
* rename method.
2022-10-12 16:28:24 -07:00
593f1e6754
Include apiClient in Github source ( #804 )
2022-09-19 14:31:48 -07:00
ecfdb0105b
Provide correct username for app cloning and add integration test ( #786 )
2022-09-08 17:41:53 -07:00
7ba583ca40
[THOG-681] - Handle errors sources ( #783 )
...
* Handle errors w/ github source.
* Fix loop var captured by func literal.
* Fix loop var captured by func literal.
* Set completed progress if the scan completes with no errors.
* Set progress to 100% if the scope and iteration are both 0.
* Fix commentary.
* Fix test.
* Return after the defer to os.RemoveAll.
* Fix unauth scan.
* Inline range loop.
* update tests for partial scan completion with errors. Ensure correct progress is set.
* Update progress for all sources.
* Update github test.
* Address comments.
2022-09-07 19:40:37 -07:00
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
30ebe84e3e
[THOG-608] - Fix linter errors. ( #701 )
...
* Fix linter errors.
* Fix gist adding test.
* Update test string for mock JSON reply.
* Remove if.
2022-08-09 19:20:02 -07:00
e793f4a5e6
Properly count the number of repos after a github scan resume ( #625 )
2022-06-17 16:21:22 -04:00
10f4d02c31
Allow gitlab to resume from encoded resume info ( #611 )
2022-06-17 11:45:17 -04:00
e3bbf293e2
Fix NPD on mutex ( #609 )
...
* Fix NPD on mutex
* fix test
2022-06-06 17:20:27 -07:00
9074006695
Fix bug in GitHub unit test mocking ( #608 )
2022-06-06 16:58:34 -07:00
fd79a367f1
Allow github to resume from encoded resume info ( #601 )
2022-06-06 12:08:57 -04:00
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. ( #539 )
...
* Remove TokenSource interface from the init method of Source.
* Remove proto message.
* Remove proto message.
* Fix tests.
* Fix filesystem test.
2022-05-13 14:35:06 -07:00
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. ( #529 )
...
* Add new parameter to the Init method for the source interface.
* Add Oauth Token service.
* remove .test file.
* remove .test file.
* Fix param spelling.
* fix tests with new param in init
* Add missing gock lib.
2022-05-10 11:11:43 -07:00
edaf1e1fd3
Move GitHub integration tests behind a build flag and add unit tests ( #527 )
...
* Add unit tests and refactor some logic
* Move integration tests to a separate file behind a build flag
* Fix bugs in normalizeRepos
* Address lint errors
* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
212aa9ba1e
Disable tests that take too long ( #524 )
2022-05-04 16:37:37 -07:00
b25295580a
Actually save the normalized repos ( #50 )
2022-03-04 08:39:22 -08:00
77418fb3f8
module v3
2022-02-15 18:54:47 -08:00
2d8756938d
Fast git scanning ( #40 )
...
* Fast git scanning
* Use original tests
* Use committer time
Co-authored-by: Bill Rich <bill.rich@trufflesec.com >
2022-02-15 18:54:47 -08:00
c131a6e4ae
add debug pprof server and metrics server
2022-02-15 18:54:47 -08:00
1fb767247f
Add missing pagination on github calls ( #30 )
...
* Add missing pagination on github calls
Includes some refactoring to improve readability and code reuse.
* Close response body and handle rate limit
* Re-include support for including users as repos to github scans
* Fix gist test to match new func signature
* Add current test name to logging
* Support username as org use case
* Also include no-auth user as org
Co-authored-by: Bill Rich <bill.rich@trufflesec.com >
2022-02-15 18:54:47 -08:00
2cc34f4633
Make tests more resilliant and more coverage
2022-02-15 18:54:47 -08:00
7e38e699f6
GitHub concurrency ( #25 )
...
* GitHub scan concurrency
* Add raw result to plain output
* Fix flakey test (still flakey)
* Fix race
2022-02-15 18:54:47 -08:00
03ead2f7ed
Integrate GitHub source
2022-02-15 18:54:47 -08:00
4218c39d99
Initial CLI w/ partially implemented Git source and demo detector ( #1 )
2022-01-13 12:02:24 -08:00
ahrav