Commit Graph

40 Commits

Author SHA1 Message Date
ahrav 7ba583ca40 [THOG-681] - Handle errors sources (#783)
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
2022-09-07 19:40:37 -07:00
Dustin Decker fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
ahrav 30ebe84e3e [THOG-608] - Fix linter errors. (#701)
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
2022-08-09 19:20:02 -07:00
trufflesteeeve e793f4a5e6 Properly count the number of repos after a github scan resume (#625) 2022-06-17 16:21:22 -04:00
trufflesteeeve 10f4d02c31 Allow gitlab to resume from encoded resume info (#611) 2022-06-17 11:45:17 -04:00
Dustin Decker 9bcddbc45a Change GHE org enum to use since ID instead of pages (#618)
* Change GHE org enum to use since ID instead of pages

* fix logging
2022-06-09 15:09:13 -07:00
Dustin Decker 8051b03bbf improve debug logging for GHE enum (#615) 2022-06-08 13:56:07 -07:00
Dustin Decker 1a12a25f4d Enumerate all visible orgs in GHE (#612) 2022-06-07 09:24:31 -07:00
Dustin Decker e3bbf293e2 Fix NPD on mutex (#609)
* Fix NPD on mutex

* fix test
2022-06-06 17:20:27 -07:00
Miccah 9074006695 Fix bug in GitHub unit test mocking (#608) 2022-06-06 16:58:34 -07:00
trufflesteeeve fd79a367f1 Allow github to resume from encoded resume info (#601) 2022-06-06 12:08:57 -04:00
ahrav d2605354fe [THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav b0d79180f6 [THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
Miccah edaf1e1fd3 Move GitHub integration tests behind a build flag and add unit tests (#527)
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
Miccah 85208606bb Reorganize GitHub source (#517)
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
2022-05-06 05:00:46 -07:00
Miccah 71442320ec Chunk orgs the same when authenticated as unauthenticated (#501)
Also debug log the amount of forks we find in addReposByOrg.
2022-05-02 17:26:01 -07:00
dcRUSTy a74bea0e8e fix: regex for hostnames (#494)
* fix: regex for ends with github.com

* fix: regex for hooks.zapier.com

* fix: regex for hooks.slack.com

* fix: regex for signalwire.com

* fix: regex for kanbantool.com

* fix: regex for invoiceoceam.com

* fix: regex for invoiceocean.com

* fix: regex for freshdesk.com

* fix: regex for discord.com

* fix: regex for deputy.com
2022-05-02 09:44:37 -07:00
Bill Rich bf5c757cd3 Add head and base support to github (#346) 2022-04-08 14:32:54 -07:00
Dustin Decker d41d18af3a fix cli parsing and improve github user scanning error handling 2022-04-03 13:42:23 -07:00
steeeve a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich c2b4f0bc39 Include line numbers in git metadata (#97)
* Include line numbers in git metadata

* Update tests for fragments and line numbers
2022-03-22 09:27:15 -07:00
Bill Rich 0ee34a5be6 Use gitleaks git cmd stream (#75)
* Switch to git cmd stream

* Fix rebase issues
2022-03-14 17:12:58 -07:00
Dustin Decker 794a082b6c Fix Dockerfiles, readme example, and github rate limit handling 2022-03-12 07:36:42 -08:00
Dustin Decker 8d754a88d4 update dependency 2022-03-09 15:49:57 -08:00
Bill Rich 55b839fc5a Only scan files that changed from base to head (#68)
* Only scan files that changed from base to head

* Acknowledge ignored errors
2022-03-08 08:48:00 -08:00
Bill Rich 2a0ece9eef Wait until rate limit reset (#69) 2022-03-08 08:47:37 -08:00
Bill Rich 665b0bf928 Add timestamp (#61)
* Add timestamp to sources

* Include timestamp in sources
2022-03-04 08:39:23 -08:00
Bill Rich b25295580a Actually save the normalized repos (#50) 2022-03-04 08:39:22 -08:00
Bill Rich c742f6a816 Do not continue if semaphore can't be acquired (#49) 2022-03-04 08:39:22 -08:00
Dustin Decker c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker 77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Dustin Decker 152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker c131a6e4ae add debug pprof server and metrics server 2022-02-15 18:54:47 -08:00
Bill Rich 1fb767247f Add missing pagination on github calls (#30)
* Add missing pagination on github calls

Includes some refactoring to improve readability and code reuse.

* Close response body and handle rate limit

* Re-include support for including users as repos to github scans

* Fix gist test to match new func signature

* Add current test name to logging

* Support username as org use case

* Also include no-auth user as org

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Dustin Decker e15fa3a5be helpful logging 2022-02-15 18:54:47 -08:00
Dustin Decker 7e38e699f6 GitHub concurrency (#25)
* GitHub scan concurrency

* Add raw result to plain output

* Fix flakey test (still flakey)

* Fix race
2022-02-15 18:54:47 -08:00
Bill Rich 28ed0c3b7c Complete support for existing git scan flags (#13)
* Add `since_commit` to git scan

* Support `max_depth` option for git scan

* Use new options in github and gitlab sources

* Address review feedback

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Dustin Decker 03ead2f7ed Integrate GitHub source 2022-02-15 18:54:47 -08:00
Dustin Decker d6ffadb1ee Initial docs and release automation (#5) 2022-01-18 16:59:18 -08:00
Dustin Decker 4218c39d99 Initial CLI w/ partially implemented Git source and demo detector (#1) 2022-01-13 12:02:24 -08:00