Ahrav Dutta
d6d6940cf9
Include a total counter. This will be used to distinguish total scans regardless of success or not. this is needed in order to display initial progress when starting scans.
2022-09-17 18:22:37 -07:00
Ahrav Dutta
aa94718ba4
Add logic to only increment progress counter if a scan succeeds.
2022-09-17 17:30:17 -07:00
Ahrav Dutta
9e8b24ae6b
Add extra method to handle scan progress completion for a source.
2022-09-17 16:18:01 -07:00
Ahrav Dutta
d9f296e7d3
Initial implementation to make progress increment monotically.
2022-09-17 15:59:04 -07:00
trufflesteeeve
945de06858
Fix include-members not working on github ( #773 )
2022-09-12 13:26:38 -04:00
Dustin Decker
ecfdb0105b
Provide correct username for app cloning and add integration test ( #786 )
2022-09-08 17:41:53 -07:00
Dustin Decker
80b247286b
Improve GitHub debug logging ( #784 )
...
* close bodies early
* add more debug logging to github
* fix nil check
* Add nil checks for response
2022-09-08 12:23:40 -07:00
ahrav
7ba583ca40
[THOG-681] - Handle errors sources ( #783 )
...
* Handle errors w/ github source.
* Fix loop var captured by func literal.
* Fix loop var captured by func literal.
* Set completed progress if the scan completes with no errors.
* Set progress to 100% if the scope and iteration are both 0.
* Fix commentary.
* Fix test.
* Return after the defer to os.RemoveAll.
* Fix unauth scan.
* Inline range loop.
* update tests for partial scan completion with errors. Ensure correct progress is set.
* Update progress for all sources.
* Update github test.
* Address comments.
2022-09-07 19:40:37 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
ahrav
30ebe84e3e
[THOG-608] - Fix linter errors. ( #701 )
...
* Fix linter errors.
* Fix gist adding test.
* Update test string for mock JSON reply.
* Remove if.
2022-08-09 19:20:02 -07:00
trufflesteeeve
e793f4a5e6
Properly count the number of repos after a github scan resume ( #625 )
2022-06-17 16:21:22 -04:00
trufflesteeeve
10f4d02c31
Allow gitlab to resume from encoded resume info ( #611 )
2022-06-17 11:45:17 -04:00
Dustin Decker
9bcddbc45a
Change GHE org enum to use since ID instead of pages ( #618 )
...
* Change GHE org enum to use since ID instead of pages
* fix logging
2022-06-09 15:09:13 -07:00
Dustin Decker
8051b03bbf
improve debug logging for GHE enum ( #615 )
2022-06-08 13:56:07 -07:00
Dustin Decker
1a12a25f4d
Enumerate all visible orgs in GHE ( #612 )
2022-06-07 09:24:31 -07:00
Dustin Decker
e3bbf293e2
Fix NPD on mutex ( #609 )
...
* Fix NPD on mutex
* fix test
2022-06-06 17:20:27 -07:00
Miccah
9074006695
Fix bug in GitHub unit test mocking ( #608 )
2022-06-06 16:58:34 -07:00
trufflesteeeve
fd79a367f1
Allow github to resume from encoded resume info ( #601 )
2022-06-06 12:08:57 -04:00
ahrav
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. ( #539 )
...
* Remove TokenSource interface from the init method of Source.
* Remove proto message.
* Remove proto message.
* Fix tests.
* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. ( #529 )
...
* Add new parameter to the Init method for the source interface.
* Add Oauth Token service.
* remove .test file.
* remove .test file.
* Fix param spelling.
* fix tests with new param in init
* Add missing gock lib.
2022-05-10 11:11:43 -07:00
Miccah
edaf1e1fd3
Move GitHub integration tests behind a build flag and add unit tests ( #527 )
...
* Add unit tests and refactor some logic
* Move integration tests to a separate file behind a build flag
* Fix bugs in normalizeRepos
* Address lint errors
* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
Miccah
85208606bb
Reorganize GitHub source ( #517 )
...
* Reorganize GitHub source
This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.
* Check errors and revert bug fix
2022-05-06 05:00:46 -07:00
Miccah
71442320ec
Chunk orgs the same when authenticated as unauthenticated ( #501 )
...
Also debug log the amount of forks we find in addReposByOrg.
2022-05-02 17:26:01 -07:00
dcRUSTy
a74bea0e8e
fix: regex for hostnames ( #494 )
...
* fix: regex for ends with github.com
* fix: regex for hooks.zapier.com
* fix: regex for hooks.slack.com
* fix: regex for signalwire.com
* fix: regex for kanbantool.com
* fix: regex for invoiceoceam.com
* fix: regex for invoiceocean.com
* fix: regex for freshdesk.com
* fix: regex for discord.com
* fix: regex for deputy.com
2022-05-02 09:44:37 -07:00
Bill Rich
bf5c757cd3
Add head and base support to github ( #346 )
2022-04-08 14:32:54 -07:00
Dustin Decker
d41d18af3a
fix cli parsing and improve github user scanning error handling
2022-04-03 13:42:23 -07:00
steeeve
a770f643df
Add placeholder for encoded resume info in SetProgressComplete
2022-03-24 12:43:36 -04:00
Bill Rich
c2b4f0bc39
Include line numbers in git metadata ( #97 )
...
* Include line numbers in git metadata
* Update tests for fragments and line numbers
2022-03-22 09:27:15 -07:00
Bill Rich
0ee34a5be6
Use gitleaks git cmd stream ( #75 )
...
* Switch to git cmd stream
* Fix rebase issues
2022-03-14 17:12:58 -07:00
Dustin Decker
794a082b6c
Fix Dockerfiles, readme example, and github rate limit handling
2022-03-12 07:36:42 -08:00
Dustin Decker
8d754a88d4
update dependency
2022-03-09 15:49:57 -08:00
Bill Rich
55b839fc5a
Only scan files that changed from base to head ( #68 )
...
* Only scan files that changed from base to head
* Acknowledge ignored errors
2022-03-08 08:48:00 -08:00
Bill Rich
2a0ece9eef
Wait until rate limit reset ( #69 )
2022-03-08 08:47:37 -08:00
Bill Rich
665b0bf928
Add timestamp ( #61 )
...
* Add timestamp to sources
* Include timestamp in sources
2022-03-04 08:39:23 -08:00
Bill Rich
b25295580a
Actually save the normalized repos ( #50 )
2022-03-04 08:39:22 -08:00
Bill Rich
c742f6a816
Do not continue if semaphore can't be acquired ( #49 )
2022-03-04 08:39:22 -08:00
Dustin Decker
c20e9f4732
improvements
2022-03-04 08:39:17 -08:00
Dustin Decker
77418fb3f8
module v3
2022-02-15 18:54:47 -08:00
Dustin Decker
152ef6d4e1
add include forks option ( #37 )
2022-02-15 18:54:47 -08:00
Dustin Decker
c131a6e4ae
add debug pprof server and metrics server
2022-02-15 18:54:47 -08:00
Bill Rich
1fb767247f
Add missing pagination on github calls ( #30 )
...
* Add missing pagination on github calls
Includes some refactoring to improve readability and code reuse.
* Close response body and handle rate limit
* Re-include support for including users as repos to github scans
* Fix gist test to match new func signature
* Add current test name to logging
* Support username as org use case
* Also include no-auth user as org
Co-authored-by: Bill Rich <bill.rich@trufflesec.com >
2022-02-15 18:54:47 -08:00
Dustin Decker
e15fa3a5be
helpful logging
2022-02-15 18:54:47 -08:00
Dustin Decker
7e38e699f6
GitHub concurrency ( #25 )
...
* GitHub scan concurrency
* Add raw result to plain output
* Fix flakey test (still flakey)
* Fix race
2022-02-15 18:54:47 -08:00
Bill Rich
28ed0c3b7c
Complete support for existing git scan flags ( #13 )
...
* Add `since_commit` to git scan
* Support `max_depth` option for git scan
* Use new options in github and gitlab sources
* Address review feedback
Co-authored-by: Bill Rich <bill.rich@trufflesec.com >
2022-02-15 18:54:47 -08:00
Dustin Decker
03ead2f7ed
Integrate GitHub source
2022-02-15 18:54:47 -08:00
Dustin Decker
d6ffadb1ee
Initial docs and release automation ( #5 )
2022-01-18 16:59:18 -08:00
Dustin Decker
4218c39d99
Initial CLI w/ partially implemented Git source and demo detector ( #1 )
2022-01-13 12:02:24 -08:00