Commit Graph

104 Commits

Author SHA1 Message Date
ahrav 04c9bb535e [THOG-768] - Add ability to skip scanning Github repos (#846)
* Add ability to skip scanning Github repos.

* remove old change.

* rename method.
2022-10-12 16:28:24 -07:00
Miccah 891996f546 Do not fail scanning if we cannot enumerate gists (#826) 2022-09-27 20:59:10 +02:00
Bill Rich 1c00014051 Include public/private in github metadata (#812)
* Include public/private in github metadata

* CR feedback

* Fix typos and naming
2022-09-26 14:55:46 -07:00
Dustin Decker 97a73710de 403 on listing user gist should not fail org scan (#822) 2022-09-26 14:37:25 -07:00
trufflesteeeve 63fcf33ce6 Fix improper github org member pagination (#814)
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
2022-09-21 16:22:42 -07:00
Dustin Decker 335e676caa Provide user when during private clones with token and fix integration tests (#811) 2022-09-19 15:53:21 -07:00
Bill Rich 593f1e6754 Include apiClient in Github source (#804) 2022-09-19 14:31:48 -07:00
trufflesteeeve 945de06858 Fix include-members not working on github (#773) 2022-09-12 13:26:38 -04:00
Dustin Decker ecfdb0105b Provide correct username for app cloning and add integration test (#786) 2022-09-08 17:41:53 -07:00
Dustin Decker 80b247286b Improve GitHub debug logging (#784)
* close bodies early

* add more debug logging to github

* fix nil check

* Add nil checks for response
2022-09-08 12:23:40 -07:00
ahrav 7ba583ca40 [THOG-681] - Handle errors sources (#783)
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
2022-09-07 19:40:37 -07:00
Dustin Decker fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
ahrav dcc102a81c [Thog-371] Utilize config struct for engine scans (#700)
* Use a config struct when scanning and engine source.

* fix tests.

* Move test_helpers to the sources pkg.

* Handle ScanGit error in tests.

* adderss comments.

* Use functional options.

* Remove temp var.

* Add better var names for the setup functions for each config.

* Remove unused var.

* fix error logs.

* fix error logs.

* single line.

* remove blank lines.
2022-08-10 10:11:13 -07:00
ahrav 30ebe84e3e [THOG-608] - Fix linter errors. (#701)
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
2022-08-09 19:20:02 -07:00
trufflesteeeve e793f4a5e6 Properly count the number of repos after a github scan resume (#625) 2022-06-17 16:21:22 -04:00
trufflesteeeve 10f4d02c31 Allow gitlab to resume from encoded resume info (#611) 2022-06-17 11:45:17 -04:00
Dustin Decker 9bcddbc45a Change GHE org enum to use since ID instead of pages (#618)
* Change GHE org enum to use since ID instead of pages

* fix logging
2022-06-09 15:09:13 -07:00
Dustin Decker 8051b03bbf improve debug logging for GHE enum (#615) 2022-06-08 13:56:07 -07:00
Dustin Decker 1a12a25f4d Enumerate all visible orgs in GHE (#612) 2022-06-07 09:24:31 -07:00
Dustin Decker e3bbf293e2 Fix NPD on mutex (#609)
* Fix NPD on mutex

* fix test
2022-06-06 17:20:27 -07:00
Miccah 9074006695 Fix bug in GitHub unit test mocking (#608) 2022-06-06 16:58:34 -07:00
trufflesteeeve fd79a367f1 Allow github to resume from encoded resume info (#601) 2022-06-06 12:08:57 -04:00
ahrav d2605354fe [THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav b0d79180f6 [THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
Miccah edaf1e1fd3 Move GitHub integration tests behind a build flag and add unit tests (#527)
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
Miccah 85208606bb Reorganize GitHub source (#517)
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
2022-05-06 05:00:46 -07:00
Bill Rich 212aa9ba1e Disable tests that take too long (#524) 2022-05-04 16:37:37 -07:00
Miccah 71442320ec Chunk orgs the same when authenticated as unauthenticated (#501)
Also debug log the amount of forks we find in addReposByOrg.
2022-05-02 17:26:01 -07:00
dcRUSTy a74bea0e8e fix: regex for hostnames (#494)
* fix: regex for ends with github.com

* fix: regex for hooks.zapier.com

* fix: regex for hooks.slack.com

* fix: regex for signalwire.com

* fix: regex for kanbantool.com

* fix: regex for invoiceoceam.com

* fix: regex for invoiceocean.com

* fix: regex for freshdesk.com

* fix: regex for discord.com

* fix: regex for deputy.com
2022-05-02 09:44:37 -07:00
Bill Rich bf5c757cd3 Add head and base support to github (#346) 2022-04-08 14:32:54 -07:00
Dustin Decker d41d18af3a fix cli parsing and improve github user scanning error handling 2022-04-03 13:42:23 -07:00
steeeve a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich c2b4f0bc39 Include line numbers in git metadata (#97)
* Include line numbers in git metadata

* Update tests for fragments and line numbers
2022-03-22 09:27:15 -07:00
Bill Rich 0ee34a5be6 Use gitleaks git cmd stream (#75)
* Switch to git cmd stream

* Fix rebase issues
2022-03-14 17:12:58 -07:00
Dustin Decker 794a082b6c Fix Dockerfiles, readme example, and github rate limit handling 2022-03-12 07:36:42 -08:00
Dustin Decker 8d754a88d4 update dependency 2022-03-09 15:49:57 -08:00
Bill Rich 55b839fc5a Only scan files that changed from base to head (#68)
* Only scan files that changed from base to head

* Acknowledge ignored errors
2022-03-08 08:48:00 -08:00
Bill Rich 2a0ece9eef Wait until rate limit reset (#69) 2022-03-08 08:47:37 -08:00
Bill Rich 665b0bf928 Add timestamp (#61)
* Add timestamp to sources

* Include timestamp in sources
2022-03-04 08:39:23 -08:00
Bill Rich b25295580a Actually save the normalized repos (#50) 2022-03-04 08:39:22 -08:00
Bill Rich c742f6a816 Do not continue if semaphore can't be acquired (#49) 2022-03-04 08:39:22 -08:00
Dustin Decker c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker 77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Bill Rich 2d8756938d Fast git scanning (#40)
* Fast git scanning

* Use original tests

* Use committer time

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Dustin Decker 152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker c131a6e4ae add debug pprof server and metrics server 2022-02-15 18:54:47 -08:00
Bill Rich 1fb767247f Add missing pagination on github calls (#30)
* Add missing pagination on github calls

Includes some refactoring to improve readability and code reuse.

* Close response body and handle rate limit

* Re-include support for including users as repos to github scans

* Fix gist test to match new func signature

* Add current test name to logging

* Support username as org use case

* Also include no-auth user as org

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Bill Rich 2cc34f4633 Make tests more resilliant and more coverage 2022-02-15 18:54:47 -08:00
Dustin Decker e15fa3a5be helpful logging 2022-02-15 18:54:47 -08:00
Dustin Decker 7e38e699f6 GitHub concurrency (#25)
* GitHub scan concurrency

* Add raw result to plain output

* Fix flakey test (still flakey)

* Fix race
2022-02-15 18:54:47 -08:00