Commit Graph

82 Commits

Author SHA1 Message Date
ahrav d2605354fe [THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav b0d79180f6 [THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
Miccah edaf1e1fd3 Move GitHub integration tests behind a build flag and add unit tests (#527)
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
Miccah 85208606bb Reorganize GitHub source (#517)
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
2022-05-06 05:00:46 -07:00
Bill Rich 212aa9ba1e Disable tests that take too long (#524) 2022-05-04 16:37:37 -07:00
Miccah 71442320ec Chunk orgs the same when authenticated as unauthenticated (#501)
Also debug log the amount of forks we find in addReposByOrg.
2022-05-02 17:26:01 -07:00
dcRUSTy a74bea0e8e fix: regex for hostnames (#494)
* fix: regex for ends with github.com

* fix: regex for hooks.zapier.com

* fix: regex for hooks.slack.com

* fix: regex for signalwire.com

* fix: regex for kanbantool.com

* fix: regex for invoiceoceam.com

* fix: regex for invoiceocean.com

* fix: regex for freshdesk.com

* fix: regex for discord.com

* fix: regex for deputy.com
2022-05-02 09:44:37 -07:00
Bill Rich bf5c757cd3 Add head and base support to github (#346) 2022-04-08 14:32:54 -07:00
Dustin Decker d41d18af3a fix cli parsing and improve github user scanning error handling 2022-04-03 13:42:23 -07:00
steeeve a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich c2b4f0bc39 Include line numbers in git metadata (#97)
* Include line numbers in git metadata

* Update tests for fragments and line numbers
2022-03-22 09:27:15 -07:00
Bill Rich 0ee34a5be6 Use gitleaks git cmd stream (#75)
* Switch to git cmd stream

* Fix rebase issues
2022-03-14 17:12:58 -07:00
Dustin Decker 794a082b6c Fix Dockerfiles, readme example, and github rate limit handling 2022-03-12 07:36:42 -08:00
Dustin Decker 8d754a88d4 update dependency 2022-03-09 15:49:57 -08:00
Bill Rich 55b839fc5a Only scan files that changed from base to head (#68)
* Only scan files that changed from base to head

* Acknowledge ignored errors
2022-03-08 08:48:00 -08:00
Bill Rich 2a0ece9eef Wait until rate limit reset (#69) 2022-03-08 08:47:37 -08:00
Bill Rich 665b0bf928 Add timestamp (#61)
* Add timestamp to sources

* Include timestamp in sources
2022-03-04 08:39:23 -08:00
Bill Rich b25295580a Actually save the normalized repos (#50) 2022-03-04 08:39:22 -08:00
Bill Rich c742f6a816 Do not continue if semaphore can't be acquired (#49) 2022-03-04 08:39:22 -08:00
Dustin Decker c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker 77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Bill Rich 2d8756938d Fast git scanning (#40)
* Fast git scanning

* Use original tests

* Use committer time

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Dustin Decker 152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker c131a6e4ae add debug pprof server and metrics server 2022-02-15 18:54:47 -08:00
Bill Rich 1fb767247f Add missing pagination on github calls (#30)
* Add missing pagination on github calls

Includes some refactoring to improve readability and code reuse.

* Close response body and handle rate limit

* Re-include support for including users as repos to github scans

* Fix gist test to match new func signature

* Add current test name to logging

* Support username as org use case

* Also include no-auth user as org

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Bill Rich 2cc34f4633 Make tests more resilliant and more coverage 2022-02-15 18:54:47 -08:00
Dustin Decker e15fa3a5be helpful logging 2022-02-15 18:54:47 -08:00
Dustin Decker 7e38e699f6 GitHub concurrency (#25)
* GitHub scan concurrency

* Add raw result to plain output

* Fix flakey test (still flakey)

* Fix race
2022-02-15 18:54:47 -08:00
Bill Rich 28ed0c3b7c Complete support for existing git scan flags (#13)
* Add `since_commit` to git scan

* Support `max_depth` option for git scan

* Use new options in github and gitlab sources

* Address review feedback

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-02-15 18:54:47 -08:00
Dustin Decker 03ead2f7ed Integrate GitHub source 2022-02-15 18:54:47 -08:00
Dustin Decker d6ffadb1ee Initial docs and release automation (#5) 2022-01-18 16:59:18 -08:00
Dustin Decker 4218c39d99 Initial CLI w/ partially implemented Git source and demo detector (#1) 2022-01-13 12:02:24 -08:00